Text processing method, first model training method and related device

Abstract

The invention provides a text processing method, a first model training method and a related device. The method comprises the steps of performing P times of first prediction based on a first text unit sequence to obtain P first text units; splicing the P first text units to a first text unit sequence to obtain a second text unit sequence; the first text unit sequence comprises N text units, P and N are both greater than 0, and P is less than N; performing M times of second prediction based on the second text unit sequence to obtain M second text units; m is greater than 0; and determining a text processing result of the first text unit sequence based on the P first text units and the M second text units. Through the text processing result generation method and device, the accuracy of text processing result generation can be guaranteed, and meanwhile the prediction efficiency of prediction is improved by controlling the sequence length during prediction.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a text processing method, a first model training method, and related devices. Background Art

[0002] The continuous advancement of large-scale model technology has provided a bright prospect for the widespread use of artificial intelligence products in various industries. Among related technologies, large-scale model products represented by the GPT series of models have been gradually applied to various fields. For example, the GPT series of large models are not only used in question-answering scenarios, that is, text-to-text applications, but are also gradually being applied to cross-modal applications such as text-to-audio, text-to-image, and text-to-video. The GPT series of large models is a type of autoregressive large model. Its inference output is output one text unit (Token) at a time. The prediction result of each output will also be used as part of the input for the next step of inference. In other words, the GPT series of large models can gradually build a sequence, considering the currently generated text unit to determine the next text unit, thereby generating coherent and realistic data. Summary of the Invention

[0003] The embodiments of the present application provide a text processing method, a first model training method, and related devices, which can improve the prediction efficiency of text processing by controlling the sequence length during prediction while ensuring the accuracy of the generated text processing results.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] An embodiment of the present application provides a text processing method, the method comprising: performing P first predictions based on a first text unit sequence to obtain P first text units; and splicing the P first text units to the first text unit sequence to obtain a second text unit sequence; the first text unit sequence includes N text units, P and N are both greater than 0, and P is less than N; performing M second predictions based on the second text unit sequence to obtain M second text units; wherein the (i+1)th second prediction comprises: masking part of the text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence; splicing i second text units obtained from the previous i second predictions to the third text unit sequence to obtain a fourth text unit sequence; performing the (i+1)th second prediction based on the fourth text unit sequence to obtain an (i+1)th second text unit; M is greater than 0, and i is an integer greater than 0 and less than M; and determining a text processing result of the first text unit sequence based on the P first text units and the M second text units.

[0006] An embodiment of the present application provides a first model training method, which includes: obtaining a sample data set; the sample data in the sample data set includes a sample text unit sequence and a label text unit sequence; performing Q first predictions based on the sample text unit sequence to obtain Q first sample units; and splicing the Q first sample units to the sample text unit sequence to obtain a sixth text unit sequence; Q is greater than 0; performing K second predictions based on the sixth text unit sequence to obtain K second sample units; K is greater than 0; determining a text prediction result of the sample text unit sequence based on the Q first sample units and the K second sample units; performing loss calculation based on the first sample unit, the second sample unit, the text prediction result and the label text unit sequence to obtain a loss result; and updating the model parameters of the first model based on the loss result to obtain a trained first model.

[0007] An embodiment of the present application provides a text processing device, comprising: a first text unit generation module, configured to perform P first predictions based on a first text unit sequence to obtain P first text units; and splicing the P first text units to the first text unit sequence to obtain a second text unit sequence; the first text unit sequence includes N text units, P and N are both greater than 0, and P is less than N; a second text unit generation module, configured to perform M second predictions based on the second text unit sequence to obtain M second text units; wherein the (i+1)th second prediction comprises: masking some text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence; splicing i second text units obtained from the previous i second predictions to the third text unit sequence to obtain a fourth text unit sequence; performing an (i+1)th second prediction based on the fourth text unit sequence to obtain an (i+1)th second text unit; M is greater than 0, and i is an integer greater than 0 and less than M; and a first determination module, configured to determine a text processing result of the first text unit sequence based on the P first text units and the M second text units.

[0008] An embodiment of the present application provides a first model training device, comprising: an acquisition module for acquiring a sample data set; the sample data in the sample data set includes a sample text unit sequence and a label text unit sequence; a first prediction module for performing Q first predictions based on the sample text unit sequence to obtain Q first sample units; and splicing the Q first sample units to the sample text unit sequence to obtain a sixth text unit sequence; Q is greater than 0; a second prediction module for performing K second predictions based on the sixth text unit sequence to obtain K second sample units; K is greater than 0; a second determination module for determining a text prediction result of the sample text unit sequence based on the Q first sample units and the K second sample units; a loss calculation module for performing loss calculation based on the first sample unit, the second sample unit, the text prediction result and the label text unit sequence to obtain a loss result; an update module for updating the model parameters of the first model based on the loss result to obtain a trained first model.

[0009] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions; and a processor for implementing the text processing method provided in the embodiment of the present application, or implementing the first model training method provided in the embodiment of the present application, when executing the computer-executable instructions stored in the memory.

[0010] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the text processing method provided in the embodiment of the present application when executed by a processor, or implementing the first model training method provided in the embodiment of the present application.

[0011] An embodiment of the present application provides a computer program product, which includes executable instructions, and the executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the executable instructions from the computer-readable storage medium and executes the executable instructions, the text processing method provided in the embodiment of the present application is implemented, or the first model training method provided in the embodiment of the present application is implemented.

[0012] The embodiments of the present application have the following beneficial effects:

[0013] First, the prediction process for the first text unit sequence is divided into two stages: the first prediction and the second prediction. During the first prediction, P first predictions are made based on the first text unit sequence, resulting in P corresponding first text units. These P first text units are then concatenated to the first text unit sequence to obtain a second text unit sequence. In other words, each first prediction within the first prediction stage is based on all text units in the first text unit sequence. This ensures that information from the entire first text unit sequence is considered when generating the first text unit within the first prediction stage, making the predicted first text unit more accurate and comprehensive.

[0014] Then, during the second prediction, M second predictions are performed based on the second text unit sequence to obtain M second text units; and during the i+1th second prediction, it is first necessary to mask the portion of the text units in the second text unit sequence that belong to the first text unit sequence, and then splice the i second text units obtained from the previous i second predictions to the masked third text unit sequence, and then perform the i+1th second prediction on the spliced ​​fourth text unit sequence. That is, during the second prediction, at least the following three factors are considered: first, the fourth text unit sequence of each second prediction includes all second text units generated by the previous prediction; second, because the first prediction of the first prediction stage for the first text unit sequence has been completed before the second prediction in the second prediction stage, the fourth text unit sequence already includes information that can represent the dependency relationship between all text units in the first text unit sequence; third, during the second prediction, the portion of the text units in the second text unit sequence that belong to the first text unit sequence is masked, that is, the fourth text unit sequence of each second prediction only includes the portion of the text units of the first text unit sequence, thereby reducing the number of text units in the input fourth text unit sequence. In this way, in the second prediction stage, while ensuring the generation quality of the second text unit, the sequence length of the fourth text unit sequence of each second prediction can be effectively controlled to improve the prediction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 This is a schematic diagram of the model training process in the related technology provided by the embodiment of the present application;

[0016] Figure 2 This is a schematic diagram of the model reasoning process in the related technology provided by the embodiments of the present application;

[0017] Figure 3 is a structural diagram of a text processing system provided in an embodiment of the present application;

[0018] Figure 4This is an optional flowchart of the text processing method provided in the embodiment of the present application;

[0019] Figure 5 This is another optional flowchart of the text processing method provided in the embodiment of the present application;

[0020] Figure 6 This is a schematic diagram of an implementation of determining a third text unit sequence provided by an embodiment of the present application;

[0021] Figure 7 This is an optional implementation diagram of determining the offset of the second prediction provided in an embodiment of the present application;

[0022] Figure 8 This is another optional implementation diagram of determining the offset of the second prediction provided in an embodiment of the present application;

[0023] Figure 9 This is an optional flowchart of the first model training method provided in an embodiment of the present application;

[0024] Figure 10 This is a schematic diagram of an implementation of determining loss results provided by an embodiment of the present application;

[0025] Figure 11 This is a schematic diagram of the relationship between the generation condition sequence and the target generation sequence provided in the embodiment of the present application;

[0026] Figure 12 This is a schematic diagram of the implementation of random masking processing during model training provided in an embodiment of the present application;

[0027] Figure 13 1 is a schematic diagram of changes in the input sequence before and after masking processing in the first iterative training of the second training stage provided by an embodiment of the present application;

[0028] Figure 14 is a structural diagram of a text processing device provided in an embodiment of the present application;

[0029] Figure 15 Schematic diagram of the structure of the first model training device provided in an embodiment of the present application;

[0030] Figure 16 It is a schematic diagram of the composition structure of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0032] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0034] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, and the informed consent or separate consent of the personal information subject should be obtained. Subsequent data use and processing should be carried out within the scope of authorization of laws and regulations and the personal information subject.

[0035] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0036] 1) Large models: These are machine learning models with large parameters and complex computational structures. They are typically built from deep neural networks and may contain billions or even hundreds of billions of parameters. Large models are designed to improve their expressiveness and predictive performance, enabling them to handle more complex tasks and data. Large models are widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems. Large models learn complex patterns and features by training on massive amounts of data, resulting in stronger generalization capabilities and the ability to make accurate predictions on unseen data.

[0037] 2) Autoregressive generative model: This is a type of generative neural network model that can generate a series of data, such as text, audio, or images. The characteristic of the autoregressive generative model is that it uses iterative step-by-step reasoning when generating task reasoning. The result of each step of reasoning is not only part of the final reasoning result, but also the input of the next step of reasoning. In the autoregressive generative model, the data is divided into multiple text units (tokens). These tokens can be characters in text, sound wave samples in audio, or pixels in an image. During the model learning process, it learns how to predict the next token based on the previous tokens. That is, the generation of each token is based on the previously generated token until a complete sentence is generated or a certain termination condition is reached.

[0038] In related technologies, large autoregressive models are subject to sequence length limitations during model training. Generally, machine training requires a specific number of samples, forming a training batch. However, GPU memory capacity is limited, and if the sequence length is too long, it will exceed the GPU's available capacity, causing the training process to crash. There are also corresponding sequence length limitations during model inference. Assuming that during model training, the total sequence length of the sample data is distributed between 100 and 800, and during model inference, the attempt to infer a generated result with a total sequence length distributed between 1500 and 2500 will result in a significant decrease in the quality of the generated result and a significant increase in inference time, among other unfavorable consequences.

[0039] Taking TorToiseTTS in related technologies as an example, first, a pre-trained Vector Quantized-Variational AutoEncoder (VQ-VAE) module is used to convert the speech audio (Mel spectrum) into an audio token (an integer value encoding. In natural language processing, a token refers to the smallest semantic unit in a text, i.e., a text unit). The corresponding text of the speech is then tokenized to obtain an integer sequence of text tokens. The text tokens and audio tokens are used as the upper input and lower output of the model, respectively, and are concatenated into a sample data. The model training is based on this sample data. The model training of TorToiseTTS uses the large model architecture of GPT2. The learning task of this architecture is to predict the next token value.

[0040] like Figure 1As shown in the figure, during model training, the input text is converted into a text token sequence using a text tokenizer. A special "text start" token is added to the beginning of the text token sequence, and a special "text end" token is added to the end. First, the speech data corresponding to the input text is converted into a mel-spectrogram, which is then passed through a pre-trained VQVAE encoder to obtain a mel-spectrogram sequence. A special "mel-start" token is added to the beginning of the mel-spectrogram sequence, and a special "mel-end" token is added to the end. The text token sequence and the mel-spectrogram sequence are concatenated, and a timbre condition is added at the beginning of the entire sequence. This timbre condition is the reference timbre encoding obtained by running the timbre encoder on the reference audio. The model training loss is the cross-entropy loss, Loss = CrossEntropy(X, Y), where X is the predicted output sequence obtained after model training, taking the token values ​​at all positions between the second position (inclusive) and the penultimate position (inclusive) of the generated result; Y is the model label, specifically the token values ​​at all positions in the sample data from the "text start" position (excluding) to the "mel-end" position (inclusive). CrossEntropy represents the cross entropy loss function, and Loss represents the cross entropy loss of the model.

[0041] like Figure 2 As shown in the figure, during the model inference process, the initial input is the timbre condition, which is then concatenated with a text token sequence (including the two special tokens "text start" and "text end"), followed by a special token "Mel start". Assuming the length of this input sequence is N, the length of the output sequence after model inference is also N, and the first predicted Mel token is at the last position of the output sequence. The next iteration is to concatenate the first predicted Mel token from the previous round to the end of the input sequence. The second predicted Mel token is then at the last position of the output sequence. This continues until the Mel token resulting from a certain round of inference is the "Mel end" token. The iteration ends, and the resulting sequence of all Mel tokens from the first Mel token to (excluding) the "Mel end" token is the speech token sequence synthesized by TorToiseTTS.

[0042] It can be seen from this that in the related technology, since the generation condition sequence of TortoiseTTS input is a text token sequence, and the generation result is a voice token sequence, both of which are long sequences, the sequence length of the input sequence will become longer and longer during the entire model inference process, resulting in two adverse consequences: first, an overly long input sequence will lead to generation failure and the synthesis of mixed and invalid speech; second, as the sequence length of the input sequence becomes longer, the time consumed by each subsequent iterative inference also increases, making the efficiency of the entire model inference slower.

[0043] Based on at least one problem with the related art, this embodiment enhances the "semi-conditional" reasoning capability of the first model by introducing a second training phase during the model training process. This makes the current prediction iteration of the first model relatively more dependent on the generated results of the previous prediction iteration, rather than solely on the generated conditional sequence. Furthermore, during the model reasoning process, step-by-step evolutionary reasoning is employed, gradually deleting some tokens from the generated conditional sequence during the prediction iteration to control the length of the input sequence during the model reasoning process. This effectively addresses the issues of low model prediction accuracy and slow model reasoning efficiency caused by overly long input sequences.

[0044] See also Figure 3 , Figure 3 It is a structural diagram of the text processing system 100 architecture provided in an embodiment of the present application. In order to support a text processing application, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0045] The terminal 400 is used to send a text processing request to the server 200, and the server 200 is used to respond to the text processing request. First, based on the first text unit sequence, P first predictions are performed to obtain P first text units; and the P first text units are spliced ​​to the first text unit sequence to obtain a second text unit sequence; the first text unit sequence includes N text units, P and N are both greater than 0, and P is less than N; then, based on the second text unit sequence, M second predictions are performed to obtain M second text units; wherein the i+1 second prediction includes: masking part of the text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence; splicing i second text units obtained from the previous i second predictions to the third text unit sequence to obtain a fourth text unit sequence; performing the i+1 second prediction based on the fourth text unit sequence to obtain the i+1 second text unit; M is greater than 0, and i is an integer greater than 0 and less than M; finally, determining the text processing result of the first text unit sequence based on the P first text units and the M second text units. After obtaining the text processing result, the server 200 may also send the text processing result to the terminal 400 so that the text processing result can be outputted on the terminal 400 .

[0046] In some embodiments, the server 200 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, car terminal, etc., but is not limited to this. The terminal 400 and the server 200 can be directly or indirectly connected by wired or wireless communication, which is not limited in the embodiments of the present application.

[0047] The text processing methods provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the text processing methods in each embodiment of the present application can be executed by a server, or by a terminal, or by interaction between a server and a terminal.

[0048] See also Figure 4 , Figure 4 This is an optional flow chart of the text processing method provided in the embodiment of the present application, which will be combined with Figure 4The steps shown are described below, and the text processing method is described by taking the execution subject as a server as an example. The method includes the following steps S101 to S103:

[0049] Step S101 : performing P first predictions based on a first text unit sequence to obtain P first text units; and concatenating the P first text units to the first text unit sequence to obtain a second text unit sequence.

[0050] In an embodiment of the present application, the first text unit sequence refers to a sequence composed of a series of text units in a specific arrangement order, that is, the first text unit sequence includes N text units, and N is an integer greater than 0. The text unit can be a word, a subword, a character, or a token composed of characters. The first text unit sequence can be used as an input sequence for the first model, so that the first model predicts the next text unit in the sequence by analyzing the statistical characteristics and contextual relationship of the input sequence. The first model is a deep learning model for processing the mapping problem between the input sequence and the output sequence. For example, the first model can be a deep learning model based on a network structure such as a long short-term memory network, a gated recurrent unit, or a Transformer. There is no specific limitation on the model structure of the first model.

[0051] For example, in a text-based application scenario, the text units in the first text unit sequence may be character-level text units. At the character level, the text units are the most basic characters, such as letters, numbers, or punctuation marks. For example, for the sentence "The cat sat on the mat.", the first text unit sequence at the character level can be "'T', 'h', 'e', ​​",'c', 'a', 't', ",'s', 'a', 't', ",'o', 'n', ",'t', 'h', 'e', ​​",'m', 'a', 't', '.". The text units in the first text unit sequence can also be word-level text units. At the word level, the text unit is the entire word. For the sentence "The cat sat on the mat.", the first text unit sequence at the word level can be "'The', 'cat', 'sat', 'on', 'the', 'mat', '.". The text units in the first text unit sequence can also be sentence fragment-level text units. In some application scenarios, the text unit may be a longer fragment, such as a phrase or part of a sentence.

[0052] In some embodiments, the process of generating the first text unit sequence also involves preprocessing the original text, which requires segmenting the original text into word, subword, or character-level text units, and performing necessary cleaning and formatting. The multiple text units obtained after preprocessing are arranged and combined in a certain order to obtain the first text unit sequence.

[0053] Here, in the first model, the model starts the generation process based on the first text unit sequence obtained. For example, when the first model is applied to a text translation scenario, assuming that the first text unit sequence corresponding to the source language text is "'The', 'cat', 'sat', 'on', 'the', 'mat', '.'", a start symbol is added to the beginning of the target language sequence to indicate the start of text translation. The first model will first generate the first text unit "this" of the target language sequence based on the context vector corresponding to the first text unit sequence and the start symbol. It will then continue to generate text units such as "only", "cat", "in", and so on, until a complete sentence is generated or a certain end condition is reached.

[0054] In the embodiment of the present application, the prediction process for the first text unit sequence is divided into two stages: the first prediction stage is the first stage in the prediction process for the first text unit sequence, and the second prediction stage is the second stage in the prediction process for the first text unit sequence. Two different prediction methods are used in the first prediction stage and the second prediction stage respectively.

[0055] The first prediction phase is the process of performing a first prediction based on all text units in the first text unit sequence. First prediction refers to the process by which the first model generates the next text unit based on the input first text unit sequence. Specifically, during the first prediction phase, P first predictions are performed on the first text unit sequence, and the first text unit generated by each first prediction is appended to the first text unit sequence to form a second text unit sequence. P predictions represent the preset prediction period within the first prediction phase, where P is an integer greater than 0 and less than N. The value of P can be adaptively adjusted based on the actual length of the first text unit sequence or the difficulty of the actual prediction task, and is not specifically limited here. The first text unit is the output result of each first prediction during the first prediction phase. P first predictions correspond to P first text units, and the number of first text units generated remains the same as the number of first predictions during the first prediction phase. Specifically, for each first prediction during the first prediction phase, the first text unit generated by the previous first prediction is appended to the first text unit sequence to obtain the input sequence for the current first prediction. The first model then performs a first prediction on this input sequence, obtaining the output result (i.e., the first text unit) of the current first prediction.

[0056] For example, the input sequence A during the first first prediction is the first text unit sequence; the input sequence B during the second first prediction is the text unit sequence after the first first prediction, where the first text unit a obtained by the first first prediction is spliced ​​to the text unit sequence after the first text unit sequence A, that is, the input sequence B during the second first prediction can be expressed as: "A", "a"; the input sequence C during the third first prediction is the text unit sequence after the second first prediction, where the first text unit b obtained by the second first prediction is spliced ​​to the text unit sequence after the input sequence B during the second first prediction, that is, the input sequence C during the third first prediction can be expressed as: "A", "a", "b", and so on.

[0057] Here, through the P first predictions in the first prediction stage, the first model can combine all the contextual information in the first text unit sequence to accurately understand all the semantics and structures in the first text unit sequence, thereby improving the accuracy of the prediction result of the first text unit.

[0058] Step S102 : performing M second predictions based on the second text unit sequence to obtain M second text units.

[0059] In an embodiment of the present application, the second prediction stage is a process of performing a second prediction based on a portion of the text units in the first text unit sequence. That is, during the second prediction stage, the portion of the text units in the second text unit sequence that belong to the first text unit sequence is masked, so that the input sequence for each second prediction only includes the portion of the text units in the first text unit sequence. That is, during the second prediction stage, the (i+1)th second prediction includes: masking the portion of the text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence; concatenating the i second text units obtained from the previous i second predictions to the third text unit sequence to obtain a fourth text unit sequence; and performing the (i+1)th second prediction based on the fourth text unit sequence to obtain the (i+1)th second text unit. M is greater than 0, and i is an integer greater than 0 and less than M. M second predictions are a preset prediction period within the second prediction stage. The fourth text unit sequence is the input sequence for each second prediction during the second prediction stage, and the second text unit is the output result of each second prediction during the second prediction stage, i.e., the text unit obtained by performing the second prediction on the fourth text unit sequence. Each second prediction for the second text unit sequence generates a second text unit, and the number of second text units generated is consistent with the number of second predictions in the second prediction stage. That is, for each second prediction in the second prediction stage, the fourth text unit sequence of each second prediction only includes some of the text units in the first text unit sequence, and the number of text units from the first text unit sequence included in the fourth text unit sequence in the previous second prediction is greater than or equal to the number of text units from the first text unit sequence included in the fourth text unit sequence in the next second prediction.

[0060] For example, assume that the input sequence D of the first second prediction is a text unit sequence composed of the first text units "a" and "b" obtained by P first predictions in the first prediction stage, and some text units "A'", "B'", "C'" in the first text unit sequence, that is, the input sequence D of the first second prediction can be expressed as: "A'", "B'", "C'", "a", "b"; the input sequence E of the second second prediction can be the text unit sequence after the first second prediction, after the second text unit c obtained by the first second prediction is spliced ​​to the input sequence D, that is, the input sequence E of the second second prediction can be expressed as: "A'", "a", "b", "c"; the input sequence E of the second second prediction can also be the text unit sequence after the first second prediction, after the second text unit c obtained by the first second prediction is spliced ​​to the masked input sequence D', and the masked input sequence D' contains some text units "B'", "C'" in the first text unit sequence, that is, the input sequence E of the second second prediction can also be expressed as: "B'", "C'", "a", "b", "c", and so on.

[0061] Here, since the first model has been able to accurately understand all the semantics and structures in the first text unit sequence after P first predictions in the first prediction stage, in the second prediction stage, in order to consider the sequence length restriction of the first model for the input sequence, that is, the impact of the sequence length of the input sequence of the first model on the model performance, in an embodiment of the present application, the sequence length of the input sequence of the first model is reduced by deleting some text units of the first text unit sequence in the input sequence, so that the first model can ensure the accuracy of the prediction result of the second text unit based on the remaining text units in the first text unit sequence and the accurate first text unit generated by the first prediction stage.

[0062] Step S103 : determining a text processing result of the first text unit sequence based on the P first text units and the M second text units.

[0063] In an embodiment of the present application, the text processing result may be the prediction result output by the first model after the first prediction stage and the second prediction stage, and the text processing result corresponds to the first text unit sequence. For example, in a Vincent text application scenario, the text processing result may be the predicted text sequence; in a Vincent image application scenario, the text processing result may be the predicted pixel sequence; in a Vincent audio application scenario, the text processing result may be the predicted audio sequence.

[0064] The text processing result may also be the result of post-processing the prediction result output by the first model after the first prediction stage and the second prediction stage. For example, in the application scenario of Vincent text, post-processing may include correcting grammatical errors in the prediction result, removing unnecessary punctuation, etc., in which case the text processing result may be the final text after post-processing; in the application scenario of Vincent image, post-processing may include correcting the image generated by the prediction result (for example, adjusting contrast, brightness, saturation, etc.), using image processing technology to remove image noise, etc., in which case the text processing result may be the final image after post-processing; in the application scenario of Vincent audio, post-processing may include editing the audio generated by the prediction result (for example, removing unnecessary blanks or repeated parts, etc.), removing background noise in the audio, etc., in which case the text processing result may be the final audio after post-processing.

[0065] After obtaining P first text units generated in the first prediction stage and M second text units generated in the second prediction stage, the M second text units are spliced ​​to the P first text units, and the spliced ​​text unit sequence is post-processed to retain the valid text units in the spliced ​​text unit sequence, and finally a text processing result corresponding to the first text unit sequence is obtained.

[0066] The present application embodiment provides a text processing method,

[0067] First, the prediction process for the first text unit sequence is divided into two stages: the first prediction and the second prediction. During the first prediction, P first predictions are made based on the first text unit sequence, resulting in P corresponding first text units. These P first text units are then concatenated to the first text unit sequence to obtain a second text unit sequence. In other words, each first prediction within the first prediction stage is based on all text units in the first text unit sequence. This ensures that information from the entire first text unit sequence is considered when generating the first text unit within the first prediction stage, making the predicted first text unit more accurate and comprehensive.

[0068] Then, during the second prediction, M second predictions are performed based on the second text unit sequence to obtain M second text units; and during the i+1th second prediction, it is first necessary to mask the portion of the text units in the second text unit sequence that belong to the first text unit sequence, and then splice the i second text units obtained from the previous i second predictions to the masked third text unit sequence, and then perform the i+1th second prediction on the spliced ​​fourth text unit sequence. That is, during the second prediction, at least the following three factors are considered: first, the fourth text unit sequence of each second prediction includes all second text units generated by the previous prediction; second, because the first prediction of the first prediction stage for the first text unit sequence has been completed before the second prediction in the second prediction stage, the fourth text unit sequence already includes information that can represent the dependency relationship between all text units in the first text unit sequence; third, during the second prediction, the portion of the text units in the second text unit sequence that belong to the first text unit sequence is masked, that is, the fourth text unit sequence of each second prediction only includes the portion of the text units of the first text unit sequence, thereby reducing the number of text units in the input fourth text unit sequence. In this way, in the second prediction stage, while ensuring the generation quality of the second text unit, the sequence length of the fourth text unit sequence of each second prediction can be effectively controlled to improve the prediction efficiency.

[0069] The text processing method in the embodiment of the present application will be described below in conjunction with the interaction process between the server and the terminal in the text processing system. It should be noted that the text processing method here is a text processing method implemented by the interaction between the server and the terminal, which is essentially the same as the text processing method executed by the server in the above embodiment. The only difference is that the embodiment of the present application also describes the actions performed by the terminal during the execution of the text processing method, and some steps can be executed by both the terminal and the server. Therefore, for the steps in this embodiment that are the same as those in the above embodiment but have different execution entities, this embodiment is only an illustrative description. During the implementation process, they can be executed by any execution entity, and the embodiment of the present application does not limit this.

[0070] See also Figure 5 , Figure 5 This is another optional flow chart of the text processing method provided in the embodiment of the present application, such as Figure 5 As shown, the method includes the following steps S201 to S208:

[0071] Step S201: The terminal receives a text processing operation input by a user.

[0072] In embodiments of the present application, a text processing operation may be an operation input by a user through a client of a text processing system, such as a text translation operation, a text-to-audio operation, or a text-to-image operation input by a user on a text conversion website. A user may input a text processing operation through the client of a text processing system. The text processing system may provide a text processing function, and the user may input the text processing operation through the text processing function page. In some embodiments, when inputting a text processing operation, a first text unit sequence may be input simultaneously, and the first text unit sequence and a trained first model for processing the first text unit sequence may also be input simultaneously.

[0073] Step S202: The terminal generates a text processing request in response to the text processing operation.

[0074] In an embodiment of the present application, the data input by the user can be encapsulated into a text processing request. For example, the first text unit sequence input by the user can be encapsulated into the text processing request, and the first text unit sequence input by the user and the trained first model for processing the first text unit sequence can also be encapsulated into the text processing request.

[0075] Step S203: The terminal sends a text processing request to the server.

[0076] In step S204 , the server responds to the text processing request, performs P first predictions based on the first text unit sequence to obtain P first text units, and concatenates the P first text units to the first text unit sequence to obtain a second text unit sequence.

[0077] In an embodiment of the present application, when a text processing request encapsulates a first text unit sequence, the text processing request is parsed to directly parse and obtain the first text unit sequence. When the text processing request also encapsulates a trained first model for processing the first text unit sequence, the text processing request is parsed to directly parse and obtain the trained first model. The trained first model can also be obtained by the server directly calling from the server backend in response to the text processing request. For the explanation of the specific meaning of performing P first predictions based on the first text unit sequence to obtain P first text units; and splicing the P first text units to the first text unit sequence to obtain the second text unit sequence, please refer to the description of step S101 above, which will not be repeated here.

[0078] In some embodiments, the first first prediction includes: performing a first prediction based on the first text unit sequence to obtain the first first text unit; the j+1th first prediction includes: splicing the j first text units obtained from the previous j first predictions to the first text unit sequence to obtain a fifth text unit sequence; j is an integer greater than 0 and less than P; performing the j+1th first prediction based on the fifth text unit sequence to obtain the j+1th first text unit.

[0079] That is, during the first prediction phase, the input sequence of the first model during the first prediction is the first text unit sequence. During the first prediction, the first text unit sequence is input into the trained first model, and the trained first model predicts the first text unit sequence to obtain the first text unit of the first prediction. During the j+1th prediction, the first text unit obtained from the jth prediction is first concatenated with the input sequence of the jth prediction, forming the input sequence of the j+1th prediction. Then, the input sequence of the j+1th prediction is input into the trained first model for the j+1th prediction, obtaining the prediction result of the first model for the j+1th prediction, i.e., the first text unit of the j+1th prediction. That is, during the first prediction phase, the first text unit obtained from each prediction is concatenated with the first text unit sequence to form the input sequence for the next prediction, i.e., the fifth text unit sequence.

[0080] Here, in the first prediction stage, the first text unit sequence is modeled through the trained first model to capture the contextual information between the text units in the first text unit sequence, so that the first text unit text generated by each iterative prediction is more coherent and accurate.

[0081] Step S205: The server performs M second predictions based on the second text unit sequence to obtain M second text units.

[0082] In the embodiment of the present application, for performing M second predictions based on the second text unit sequence to obtain M second text units, the specific meaning and specific implementation method involved in this step can be found in the description of step S102 above, and will not be repeated here. Among them, the i+1th second prediction includes: masking some text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence; splicing the i second text units obtained from the previous i second predictions to the third text unit sequence to obtain a fourth text unit sequence; performing the i+1th second prediction based on the fourth text unit sequence to obtain the i+1th second text unit; M is greater than 0, and i is an integer greater than 0 and less than M.

[0083] In some embodiments, see Figure 6 , Figure 6 It is shown that in step S205, part of the text units belonging to the first text unit sequence in the second text unit sequence is masked to obtain a third text unit sequence, which can be achieved by the following steps S2051 to S2053:

[0084] Step S2051: determine the offset of the second prediction.

[0085] In the embodiment of the present application, the offset is a value used to indicate the starting position and the ending position of the mask, that is, the offset is the mask length. Each second prediction corresponds to an offset, and the offset corresponding to each second prediction can be the same or different.

[0086] In some embodiments, see Figure 7 , Figure 7 It is shown that in step S2051, for the first second prediction, the offset of the second prediction is determined, which can be achieved by the following steps S701 to S705:

[0087] Step S701: Obtain a sample data set for training a first model.

[0088] In the embodiments of the present application, the sample data set refers to the data set used in the training process of the first model. The sample data in the sample data set includes a sample text unit sequence and a label text unit sequence. The sample text unit sequence is the input sequence of the first model during the training process of the first model, and the label text unit sequence is the label sequence of the first model during the training process of the first model. The sample text unit sequence corresponds to the label text unit sequence. That is, the sample text units in the sample text unit sequence correspond one-to-one to the label text units in the label text unit sequence.

[0089] Step S702: determining the mean sequence length of the sample text unit sequences in the sample data set.

[0090] In the embodiments of the present application, the mean sequence length refers to the average value of the sequence lengths. The sequence lengths of multiple sample text unit sequences in the sample dataset are determined, and the sequence lengths of the multiple sample text unit sequences are added together to obtain the sum of the sequence lengths of the sample dataset. Finally, the sum of the sequence lengths is divided by the number of sample text unit sequences in the sample dataset to obtain the mean sequence lengths of the sample text unit sequences in the sample dataset.

[0091] Step S703 : when the mean of the sequence lengths is greater than the first length threshold, determining a first ratio of each sample data.

[0092] In the embodiments of the present application, the first length threshold is a preset sequence length threshold for the sample text unit sequence. The first length threshold can be adaptively adjusted based on the actual sequence length of the sample text unit sequence and is not specifically limited herein. The first ratio is the ratio of the sequence length of the sample text unit sequence to the sequence length of the label text unit sequence.

[0093] The mean sequence length of the sample text unit sequences in the sample data set is numerically compared with a preset first length threshold. When the mean sequence length is greater than the first length threshold, a first ratio is calculated for each sample data. That is, the first ratio is obtained by dividing the sequence length of the sample text unit sequence in each sample data by the sequence length of the label text unit sequence.

[0094] Step S704 , averaging the first ratios of the sample data in the sample data set to obtain a first average ratio.

[0095] In the embodiment of the present application, the first average ratio is the average of the first ratios of the plurality of sample data in the sample data set. After obtaining the first ratios of the plurality of sample data in the sample data set, the first ratios of the plurality of sample data are added together to obtain a first sum of the ratios. Finally, the sum of the first ratios is divided by the number of sample data in the sample data set to obtain the first average ratio.

[0096] Step S705 : determining an offset between the first and second predictions based on the number of predictions P of the first prediction and the first average ratio.

[0097] In this embodiment of the present application, the offset between the first and second predictions is the product of the prediction number P of the first prediction and the first average ratio. After determining the prediction number P of the first prediction and the first average ratio, the prediction number P of the first prediction is multiplied by the first average ratio. If the product of the multiplication is an integer, the product is determined as the offset between the first and second predictions. If the product of the multiplication is not an integer, the product can be rounded to obtain the offset between the first and second predictions.

[0098] Here, when determining the offset of the first second prediction, the number of predictions P of the first prediction, the mean sequence length of the sample text unit sequence during model training, and the ratio between the sequence length of the sample text unit sequence and the sequence length of the label text unit sequence are comprehensively considered. This ensures that the offset is consistent with the training distribution of the first model, obtains the optimal offset, avoids the performance degradation of the first model due to excessively long or too short masking, and thus improves the generation quality of the text unit subsequently.

[0099] In some embodiments, when the mean sequence length is less than or equal to a first length threshold, the offset of the first second prediction is determined based on the mean sequence length.

[0100] In the embodiment of the present application, the mean sequence length of the sample text unit sequence in the sample data set is numerically compared with a preset first length threshold. When the mean sequence length is less than or equal to the first length threshold, a value less than the mean sequence length is selected and determined as the offset of the first second prediction. The offset is less than the mean sequence length.

[0101] Here, when the mean sequence length is less than or equal to the first length threshold, the offset of the first second prediction is determined according to the size of the mean sequence length, which can effectively avoid the problem of excessive masking causing the first model to be unable to effectively utilize the context information in the input sequence.

[0102] In some embodiments, see Figure 8 , Figure 8 It is shown that in step S2051, for the (i+1)th second prediction, the offset of the second prediction is determined, which can be achieved by the following steps S801 to S803:

[0103] Step S801 : determining an offset at the Lth second prediction before the (i+1)th second prediction.

[0104] In the embodiment of the present application, the offset of the Lth second prediction is greater than 0, and the offsets from the L+1th second prediction to the i-th second prediction are all equal to 0; L is a positive integer, and L is less than i. That is, the Lth second prediction masks some text units in the second text unit sequence that belong to the first text unit sequence, and the masking length is equal to the offset of the Lth second prediction, which is greater than 0. The L+1th second prediction to the i-th second prediction do not mask some text units in the second text unit sequence that belong to the first text unit sequence, and the masking length is equal to the offset, which is equal to 0.

[0105] Step S802 : When i+1-L is equal to the first number threshold, determine the offset for the (i+1)th second prediction to be any offset in a preset offset set.

[0106] In an embodiment of the present application, the first numerical threshold refers to a masking period preset for masking, and the preset offset set refers to a set of offset ranges determined based on the offsets of the first and second predictions, and the offsets in the preset offset set are values ​​close to the offsets of the first and second predictions.

[0107] When i+1-L is equal to the first numerical threshold, it means that the preset masking period is reached, and the i+1th second prediction masks some text units belonging to the first text unit sequence in the second text unit sequence. An offset can be arbitrarily selected from the preset offset set and used as the offset for the i+1th second prediction.

[0108] Step S803 : When i+1-L is less than the first number threshold, the offset of the (i+1)th second prediction is determined to be 0.

[0109] In an embodiment of the present application, when i+1-L is less than the first numerical threshold, it means that the preset masking period has not been reached, and the i+1th second prediction does not mask some text units in the second text unit sequence belonging to the first text unit sequence, and the masking length is equal to the offset equal to 0.

[0110] Here, based on the first number threshold and the offset of the first second prediction, the offset of the i+1th second prediction can be dynamically determined so that the subsequent first model can better adapt to text sequences of different lengths and complexities and improve the generation quality of text units.

[0111] Step S2052 : Starting from the starting position of the first text unit sequence in the third text unit sequence of the i-th second prediction, determine the third text unit at the position corresponding to the offset.

[0112] In this embodiment of the present application, after determining the offset for the (i+1)th second prediction, starting from the starting position of the first text unit sequence in the third text unit sequence of the i-th second prediction, the text unit at the position corresponding to the offset is determined as the third text unit in the third text unit sequence to be masked. For example, assuming the offset is 3, the third text unit is the text unit at the first three positions in the first text unit sequence in the third text unit sequence of the i-th second prediction.

[0113] Step S2053: mask the third text unit to obtain the (i+1)th second predicted third text unit sequence.

[0114] In an embodiment of the present application, the third text unit belonging to the first text unit sequence in the third text unit sequence of the i-th second prediction is masked, that is, the third text unit at the offset corresponding to the i+1-th second prediction in the third text unit sequence of the i-th second prediction is deleted to obtain the third text unit sequence of the i+1-th second prediction, which is the input sequence of the first model in the i+1-th second prediction.

[0115] Here, through the masking process of the second prediction in the above-mentioned second prediction stage, the text units belonging to the first text unit sequence at the start position of the second text unit sequence are gradually masked, so that in the second prediction stage, the text units belonging to the first text unit sequence in the input sequence input into the first model during the second prediction are gradually reduced, so that after the input sequence during the i+1th second prediction is input into the first model, the generated second text unit does not mainly depend on the first text unit sequence, but depends on the first text unit and the second text unit generated by the first model. This can effectively control the sequence length of the input sequence of the first model and ensure the generation quality and prediction efficiency of the model prediction.

[0116] Step S206: The server determines a text processing result of the first text unit sequence based on the P first text units and the M second text units.

[0117] In this embodiment of the present application, after splicing the M second text units to the P first text units, a text processing result for the first text unit sequence is obtained. Regarding the specific meaning and implementation of determining the text processing result for the first text unit sequence based on the P first text units and the M second text units, please refer to the description of step S103 above and will not be repeated here.

[0118] Step S207: The server sends the text processing result to the terminal.

[0119] Step S208: The terminal outputs the text processing result.

[0120] In the embodiment of the present application, the prediction process for the first text unit sequence is divided into two prediction stages, namely the first prediction stage and the second prediction stage. In the first prediction stage, P first predictions are performed for the first text unit sequence, and the first text unit obtained from each first prediction is spliced ​​to the first text unit sequence to form a second text unit sequence. That is, each first prediction in the first prediction stage is based on all text units in the first text unit sequence, ensuring that the generated first text unit contains information about the entire first text unit sequence, making the predicted first text unit more accurate and comprehensive.

[0121] Then, during the second prediction, M second predictions are performed based on the second text unit sequence to obtain M second text units; and during the i+1th second prediction, it is first necessary to mask the portion of the text units in the second text unit sequence that belong to the first text unit sequence, and then splice the i second text units obtained from the previous i second predictions to the masked third text unit sequence, and then perform the i+1th second prediction on the spliced ​​fourth text unit sequence. That is, during the second prediction, at least the following three factors are considered: first, the fourth text unit sequence of each second prediction includes all second text units generated by the previous prediction; second, because the first prediction of the first prediction stage for the first text unit sequence has been completed before the second prediction in the second prediction stage, the fourth text unit sequence already includes information that can represent the dependency relationship between all text units in the first text unit sequence; third, during the second prediction, the portion of the text units in the second text unit sequence that belong to the first text unit sequence is masked, that is, the fourth text unit sequence of each second prediction only includes the portion of the text units of the first text unit sequence, thereby reducing the number of text units in the input fourth text unit sequence.

[0122] Finally, based on the first text unit generated in the first prediction phase and the second text unit generated in the second prediction phase, the text processing result of the first text unit sequence is determined. In this way, by iteratively predicting the first and second prediction phases for the first text unit, the accuracy of the text processing result can be guaranteed while improving the prediction efficiency of the iterative prediction by controlling the sequence length during prediction.

[0123] The first model training method provided in each embodiment of the present application can also be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the first model training method in each embodiment of the present application can be executed by a server, or by a terminal, or can be executed through interaction between a server and a terminal.

[0124] See also Figure 9 , Figure 9 This is an optional flow chart of the first model training method provided in the embodiment of the present application, which will be combined with Figure 9 The steps shown are described below, and the execution subject of the first model training method is a server as an example. The method includes the following steps S901 to S906:

[0125] Step S901: Obtain a sample data set.

[0126] In the embodiment of the present application, for the specific meaning and implementation of the step of obtaining the sample data set, please refer to the description of step S701 above, and will not be repeated here.

[0127] Step S902 : performing Q first predictions based on the sample text unit sequence to obtain Q first sample units; and concatenating the Q first sample units to the sample text unit sequence to obtain a sixth text unit sequence.

[0128] In the embodiment of the present application, Q is greater than 0. The iterative training process for the first model to be trained is divided into two training stages, namely the first training stage and the second training stage, and the training methods adopted in the first training stage and the second training stage are different. In the first training stage, the sample text unit sequence is input into the first model to be trained, and the first sample unit of each first prediction is generated through each first prediction of the first model to be trained, until the iterative training threshold of the first training stage is reached or the model parameters tend to converge. And after splicing the Q first sample units to the sample text unit sequence, a sixth sample unit sequence is obtained. The Q-time first prediction process is similar to the prediction process of the P-time first prediction in step S204 above. Therefore, for performing Q-time first predictions based on the sample text unit sequence, Q first sample units are obtained; and the Q first sample units are spliced ​​to the sample text unit sequence to obtain the sixth text unit sequence. The specific meaning and specific implementation method involved in this step can be found in the description of step S204 above, and will not be repeated here.

[0129] Step S903 : Perform K second predictions based on the sixth text unit sequence to obtain K second sample units.

[0130] In the embodiment of the present application, K is greater than 0. The K-times second prediction process is similar to the M-times second prediction process in step S205 above. Therefore, for performing K-times second prediction based on the sixth text unit sequence to obtain K second sample units, the specific meaning and specific implementation method involved in this step can be found in the description of step S205 above and will not be repeated here.

[0131] Step S904 : determining a text prediction result of the sample text unit sequence based on the Q first sample units and the K second sample units.

[0132] In the embodiment of the present application, the text prediction result of the sample text unit sequence is similar to the process of determining the text processing result of the first text unit sequence in step S103 above. Therefore, for determining the text prediction result of the sample text unit sequence based on Q first sample units and K second sample units, the specific meaning and specific implementation method involved in this step can be found in the description of step S103 above, and will not be repeated here.

[0133] Step S905 , performing loss calculation based on the first sample unit, the second sample unit, the text prediction result, and the label text unit sequence to obtain a loss result.

[0134] In some embodiments, see Figure 10 , Figure 10 It is shown that in step S905, loss calculation is performed based on the first sample unit, the second sample unit, the text prediction result and the label text unit sequence to obtain a loss result, which can be achieved by the following steps S9051 to S9054:

[0135] Step S9051: Perform a first loss calculation based on the first sample unit and label text unit sequence obtained from each first prediction to obtain a first loss value.

[0136] In the embodiment of the present application, the first loss calculation refers to measuring the difference between the first sample unit obtained by each first prediction and the label text unit at the corresponding position in the label text unit sequence through a specific loss function. The value of the loss function (i.e., the first loss value) reflects the performance of the model under the current parameters. The smaller the first loss value, the closer the first sample unit obtained by the model for each first prediction is to the label text unit at the corresponding position in the label text unit sequence; the larger the first loss value, the greater the gap between the first sample unit obtained by the model for each first prediction and the label text unit at the corresponding position in the label text unit sequence.

[0137] Step S9052: Perform a second loss calculation based on the second sample unit and label text unit sequence obtained from each second prediction to obtain a second loss value.

[0138] In an embodiment of the present application, at each second prediction, the offset of the second prediction is determined, and the sample unit to be masked for the second prediction is determined based on the offset, the sample unit is masked, and the input sequence of the second prediction is obtained. At the same time, the label text unit corresponding to the sample unit in the label text unit sequence is masked to obtain the masked label text unit sequence. A second prediction is made on the input sequence to obtain the second sample unit of the second prediction. The second loss calculation refers to measuring the difference between the second sample unit obtained in each second prediction and the label text unit at the corresponding position in the masked label text unit sequence through a specific loss function. The value of the loss function (i.e., the second loss value) reflects the performance of the model under the current parameters. The smaller the second loss value, the closer the second sample unit obtained by the model for each second prediction is to the label text unit at the corresponding position in the masked label text unit sequence; the larger the second loss value, the greater the gap between the second sample unit obtained by the model for each second prediction and the label text unit at the corresponding position in the masked label text unit sequence.

[0139] Step S9053: Perform a third loss calculation based on the text prediction result and the label text unit sequence to obtain a third loss value.

[0140] In the embodiments of the present application, the third loss calculation refers to measuring the difference between the model's text prediction results and the label text unit sequence through a specific loss function. The value of the loss function (i.e., the third loss value) reflects the performance of the model under the current parameters. The smaller the third loss value, the closer the model's text prediction results are to the label text unit sequence; the larger the third loss value, the greater the gap between the model's text prediction results and the label text unit sequence.

[0141] Step S9054: fuse the first loss value, the second loss value, and the third loss value to obtain a loss result.

[0142] In the embodiment of the present application, fusion refers to the process of weighted summing the first loss value, the second loss value, and the third loss value according to the preset loss weights. After determining the first loss value, the second loss value, and the third loss value, the first loss value, the second loss value, and the third loss value are weighted summed according to the preset loss weights to obtain the loss result of the first model.

[0143] Here, the difference between the model prediction and the true value is quantified through the loss calculation process, that is, the difference between the first sample unit obtained by each first prediction and the label text unit at the corresponding position in the label text unit sequence, the difference between the second sample unit obtained by each second prediction and the label text unit at the corresponding position in the masked label text unit sequence, and the difference between the text prediction result and the label text unit sequence are comprehensively considered. This can effectively improve the accuracy of the loss result, so that the model optimization can be correctly guided based on the loss result in the future.

[0144] Step S906: Update the model parameters of the first model based on the loss result to obtain the trained first model.

[0145] In an embodiment of the present application, based on the loss result, the model parameters in the first model are iteratively updated according to the preset iteration conditions to obtain the trained first model. The loss result is used to comprehensively measure the degree of inconsistency between the first sample unit obtained by each first prediction of the model and the label text unit sequence, the degree of inconsistency between the second sample unit obtained by each second prediction and the label text unit sequence, and the degree of inconsistency between the text prediction result and the label text unit sequence, that is, to calculate the gap between the forward calculation result of each iteration of the first model and the label (that is, the label text unit sequence), thereby guiding the next step of the first model training to proceed in the right direction.

[0146] The preset iteration conditions may be a loss threshold of the loss result, a maximum iteration threshold, and a cutoff iteration time, etc. Based on the derivative of the loss function corresponding to the loss result of the first model to be trained, the loss result is transmitted back along the direction of minimum gradient to update the model parameters in the first model to be trained, such as the various weight values ​​in the first model to be trained. A loss threshold is set in advance. When the loss result is less than the pre-set loss threshold, the iterative training is stopped, that is, the model parameter update is stopped. A maximum iteration threshold may also be set in advance. When the number of iterations exceeds the maximum iteration threshold, the model parameter update is stopped. A cutoff iteration time may also be set in advance. When the iteration time reaches the cutoff iteration time, the model parameter update is stopped, and the trained first model is finally obtained.

[0147] In an embodiment of the present application, the training process of the first model is divided into a first training phase and a second training phase. In the first training phase, the first model can predict the first sample unit generated by the previous first prediction based on all sample units in the sample text unit sequence. By modeling the information of the entire sample text unit sequence, the dependencies within the sample text unit sequence can be better captured, thereby improving the generation quality of the first model. In the second training phase, the pre-trained first model obtained in the first training phase is directly used. During each second prediction, the input sequence of the pre-trained first model is masked, that is, the sequence start position of the sample text unit sequence in the input sequence is masked. The masked input sequence is then input into the pre-trained first model, and the pre-trained first model is iteratively trained. In this way, through two-stage iterative training, the first model can maintain high accuracy and consistency when generating results under incomplete sample text unit sequences. The generation results of the first model are not primarily dependent on the first sample unit sequence, but rather on the generated portion that has passed the first model. This allows the trained first model to maintain generation quality and efficiency even when the model input sequence is longer.

[0148] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0149] The embodiment of the present application provides a large model training and reasoning method based on autoregression, which enhances the training of the fragment conditional reasoning ability during model training. Specifically: First, before model training, a corresponding relationship between the generation condition sequence (i.e., the first sample unit sequence) and the target generation sequence (i.e., the first processing unit sequence) is established. Figure 11 As shown, assuming that the total length of the generation condition sequence input by the model (i.e., the first model mentioned above) is N, and the total length of the corresponding target generation sequence is M, the relationship between the generation condition token and the target generation token can be determined through a specific tool or model. That is, the token sequence corresponding to the generation condition sequence is divided into [n1, n2, ... n x ], x≤N, split the Token sequence corresponding to the target generation sequence into [m1,m2,..m y ],y≤M, so that And x=y, that is, the lengths of the two segments are equal. For example, in Figure 11 In the example, the generation condition sequence is represented as “I”, “Love”, “Bear”, and “Cat”, with corresponding lengths of n1=1, n2=1, and n3=2 respectively; the target generation sequence is represented as “I”, “Love”, and “Panda”, with corresponding lengths of m1=1, m2=1, and m3=1 respectively.

[0150] Then, when the model is trained, it is divided into two training phases. In the first training phase, the model training uses the default training method until the model converges. When the model is trained in the second training phase, the beginning of the sequence of the generated conditional sequence is randomly masked and used as the input for model training. When calculating the loss of the model, only the remaining prediction result tokens in the model's prediction results, excluding the prediction result tokens corresponding to the beginning of the sequence of the randomly masked generated conditional sequence, and the loss between the remaining target generation sequence tokens in the target generation sequence, excluding the target generation sequence tokens corresponding to the beginning of the sequence of the randomly masked generated conditional sequence, are calculated. This allows the model to strengthen the target generation capability under "semi-conditional input (ie, partial conditional input)". The "semi-conditional input" reasoning capability refers to the large model's iterative prediction of the next Token, which relies more on the prediction result token that has been generated before, and less on the input generation conditional sequence. As Figure 12 As shown, delimiter tokens are special tokens such as the end of a text and the beginning of a mel. When truncating the conditional token length n1, the target generated token length m1 corresponding to the condition needs to be truncated. The loss during model training can be the cross-entropy loss, which is calculated over the sequence range after truncating the target generated token of length m1.

[0151] During model inference, the complete generated condition sequence is used as the model input sequence for the first P iterations of prediction. After reaching P iterations, S generation condition tokens are removed from the beginning of the generated condition sequence every Q iterations. This continues until the end token of the prediction result is predicted, at which point inference ends.

[0152] Below, the specific implementation process of the scheme proposed in the embodiment of the present application is described by taking the GPT large model in TorToiseTTS as an example. In TorToiseTTS, the generation condition sequence is a text token sequence, and the target generation sequence is a mel token sequence. The sequence organization structure of the model training data is as follows: the first position is the timbre condition (speaker condition), the second position is the text begin (text begin) Token indicating the end of the input text condition, followed by the generation condition sequence (textTokens) and the text end (text end) Token indicating the end of the generation condition sequence, the mel begin (mel begin) Token indicating the start of the target generation sequence, the VQ-VAE encoding sequence (melcode Tokens) of the audio mel spectrum corresponding to the generation condition sequence, and the mel end (mel end) Token indicating the end of the mel token sequence, which is also the end of the entire sample sequence. That is, the model training data can be expressed as: [speaker condition, text begin, textTokens…, text end, mel begin, mel code Tokens…, mel end]. In essence, the input generation condition sequence is text Tokens, and the target generation sequence is mel code Tokens. Mel code Tokens is the integer value obtained by VQ-VAE encoding of the audio mel spectrum. Every 4 mel spectrum frames correspond to a mel code. In the embodiment of the present application, the mel spectrum frame length is 12.5 milliseconds, that is, 80 frames of mel spectrum per second, so 1 second of audio can be converted into a sequence of 20 mel codes. Text Tokens is essentially the encoding of text phonemes, and each phoneme has a fixed code value. For example, in the Chinese pinyin phoneme sequence, there is a token for each initial consonant and a token for each final vowel. Therefore, assuming that the above audio corresponds to a short sentence of 5 Chinese characters, the length of the corresponding text Tokens is 10.

[0153] First, establish the correspondence between the generated conditional sequence and the target generated sequence. That is, a single-phoneme feature extraction tool based on Kaldi can be used to establish the correspondence between each phoneme and the audio frame. The single-phoneme feature extraction tool can mark the pronunciation frame length corresponding to each text Token. Assume that the Chinese character in the training sample is "你好吗" and the converted audio sequence is "n i3 hao3 ma5", and the corresponding Token sequence is [201, 60, 59, 103, 34, 12]. Assume that according to the running result of the single-phoneme feature extraction tool, the Mel-spectrum frame lengths corresponding to these 6 phonemes are 2 frames, 10 frames, 4 frames, 8 frames, 4 frames, and 4 frames respectively. Then the input conditions of the two are respectively segmented as follows: the phoneme sequence (i.e., the generated conditional sequence) [1, 1, 1, 1, 1, 1], that is, each phoneme Token is a unit. The Mel-encoded sequence (i.e., the target generated sequence) is segmented into [1, 3, 1, 2, 1, 1], and every four frames of the Mel-spectrum is a Mel-encoding. That is to say, for the first phoneme n, the number of corresponding Mel-encodings is 1; for the second phoneme i3, the number of Mel-encodings corresponding to the pronunciation is 3; and so on. In this way, the correspondence between each Token in the generated conditional sequence and each Token in the target generated sequence can be obtained.

[0154] In the embodiment of this application, the GPT large model training of TorToiseTTS is divided into two stages: In the first training stage, which is consistent with the general TortoiseTTS training process, after the model is trained until it converges or basically converges, the second training stage is then carried out. The feature of the second-stage training is that a part of the generated conditional sequence is randomly masked, and when calculating the loss of GPT, other parts except the target Mel-encoding segment are masked, and only the loss calculation for the next Token prediction is performed on the target Mel-encoding segment part. This masking processing strategy increases gradually with the training process.

[0155] For example, in the first iteration training of the second training stage, for the above training sample "你好吗", the first phoneme "n" is masked in the input of the masked training. The input sequences before and after masking are as Figure 13As shown. At this time, the organizational structure of the input sequence for model training becomes a sequence part that does not include the first phoneme "n". Then, when calculating the loss and returning the loss, according to the mapping relationship between the previous phoneme sequence and the mel coding sequence, the prediction loss of the mel coding positions other than the first mel coding position in the mel coding sequence is calculated (i.e., the second loss mentioned above). Because according to the above mapping relationship, the mel coding corresponding to the first phoneme "n" is the first mel coding position of the target generated sequence. If the masked phoneme parts are the first two phonemes "n" and "i3", then according to the mapping relationship, they correspond to the first four mel coding positions in the mel coding sequence. Therefore, when calculating the loss function, the prediction loss of the other mel coding positions except these four mel coding positions is calculated.

[0156] Note that the masking strategy for phonemes starts from the beginning of the phoneme sequence, not from the middle or the second half of the text token sequence, and the masking position for each iterative training is also the beginning of the sequence. As the number of iterative training increases, the masking length gradually increases. For example, in the first iterative training, the random masking length is equal to 1. In this case, the first phoneme "n" can be masked; in the second iterative training, the random masking length is equal to 3. In this case, the first three phonemes "n", "i3" and "h" can be masked, and so on.

[0157] During the second training phase, multiple iterations of training can be performed using the large model. The number of iterations in the second training phase can be adjusted based on the specific model and data distribution. The key to the second training phase is to determine the true correspondence between the masked mel encoding range used in the loss calculation and the range of the masked phoneme token sequence in the input sequence. After the second training phase is complete, the trained model is saved as the target TorToiseTTS large model.

[0158] During model inference, based on the trained TortoiseTTS large-scale model of Wensheng speech, the input sequence is a phoneme sequence converted from the text sequence to be synthesized. This phoneme sequence is tokenized to obtain a phoneme token sequence consisting of L = 100 phoneme tokens. Assuming the number of iterative predictions P in the first prediction phase is 40, i.e., 2 seconds of audio is generated first. The initial inference input length is 1 (corresponding to the timbre condition token) + 1 (corresponding to the text start token) + 100 (corresponding to the phoneme token sequence) + 1 (corresponding to the text end token) + 1 (corresponding to the mel start token) = 104. First, P = 40 iterative predictions are performed. The resulting token from each prediction (i.e., the first text unit) is concatenated to the end of the input token sequence (i.e., the second text unit sequence) during each prediction iteration to form the input for the next prediction iteration. After 40 prediction iterations, 40 mel tokens are obtained. At this point, the total input sequence length for the next prediction iteration is 104 + 40 = 144 tokens.

[0159] Then, starting from the beginning of the phoneme token sequence, S = 1 phoneme token is deleted after each iterative prediction. Assuming that the final number of generated Mel-scale tokens is 200, the total length of the input token sequence corresponding to the generation of the 200th Mel-scale token is 144–80+160=224, effectively controlling the growth of the input sequence length. Because if this method is not used, the total input sequence length at the last iterative prediction will reach 100+4+199=303, it can be seen that this method reduces the input sequence length by approximately 30%. Note that deleting S = 1 phoneme token after each iterative prediction must ensure that the next token is a phoneme token. If the next token is found to be the end-of-text token, that is, the phoneme token sequence has been deleted, it cannot be deleted in the subsequent inference process.

[0160] The model training and reasoning method in the embodiment of the present application, during the large model training stage, trains the "semi-conditional" synthesis capability of the large model through the fine-grained correspondence between the large model's generation condition sequence and the target generation sequence, so that the large model can adapt to the synthesis under the partial generation condition sequence during the generation iteration. At this time, the large model synthesis mainly refers to the target sequence part that has been generated, rather than just the previous generation condition Token sequence. Then, during model reasoning, the above-mentioned adaptability of the large model is used to design a step-by-step iterative reasoning method in the reasoning process. This method continuously generates new target sequence Tokens while continuously deleting generation condition Tokens at the beginning during the large model iterative reasoning process, thereby suppressing the growth rate of the total length of the input sequence of the iterative reasoning, effectively controlling the problem of the total input length being too long, and ensuring the synthesis stability and reasoning efficiency. In particular, the reasoning efficiency of the middle and latter half of the target generation sequence.

[0161] The following continues to describe the exemplary structure of the text processing device 455 provided in the embodiment of the present application as a software module. In some embodiments, such as Figure 14 As shown, the software modules in the text processing device 455 may include: a first text unit generation module 4551, configured to perform P first predictions based on a first text unit sequence to obtain P first text units; and splice the P first text units to the first text unit sequence to obtain a second text unit sequence; the first text unit sequence includes N text units, P and N are both greater than 0, and P is less than N; a second text unit generation module 4552, configured to perform M second predictions based on the second text unit sequence to obtain M second text units; wherein the (i+1)th second prediction includes: Masking is performed on some text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence; i second text units obtained by the previous i second predictions are spliced ​​into the third text unit sequence to obtain a fourth text unit sequence; the i+1th second prediction is performed based on the fourth text unit sequence to obtain the i+1th second text unit; M is greater than 0, and i is an integer greater than 0 and less than M; a first determination module 4553 is used to determine the text processing result of the first text unit sequence based on the P first text units and the M second text units.

[0162] In some embodiments, the first text unit generation module 4551 is also used for: the first first prediction, including: performing a first prediction based on the first text unit sequence to obtain the first first text unit; the j+1th first prediction, including: splicing the j first text units obtained from the previous j first predictions to the first text unit sequence to obtain a fifth text unit sequence; j is an integer greater than 0 and less than P; performing the j+1th first prediction based on the fifth text unit sequence to obtain the j+1th first text unit.

[0163] In some embodiments, the second text unit generation module 4552 is also used to: determine the offset of the second prediction; starting from the starting position of the third text unit sequence belonging to the first text unit sequence in the i-th second prediction, determine the third text unit at the position corresponding to the offset; mask the third text unit to obtain the third text unit sequence of the i+1-th second prediction.

[0164] In some embodiments, the text processing method is implemented by a first model; for the first second prediction, the second text unit generation module 4552 is also used to: obtain a sample data set for training the first model; the sample data in the sample data set includes a sample text unit sequence and a label text unit sequence; determine the mean sequence length of the sample text unit sequence in the sample data set; when the mean sequence length is greater than a first length threshold, determine a first ratio of each sample data; the first ratio is the ratio between the sequence length of the sample text unit sequence and the sequence length of the label text unit sequence; average the first ratios of the sample data in the sample data set to obtain a first average ratio; based on the number of predictions P of the first prediction and the first average ratio, determine the offset of the first second prediction.

[0165] In some embodiments, the device 455 also includes an offset determination module, which is used to determine the offset of the first second prediction based on the mean sequence length when the mean sequence length is less than or equal to the first length threshold; wherein the offset is less than the mean sequence length.

[0166] In some embodiments, the second text unit generation module 4552 is further used to: determine the offset at the Lth second prediction before the i+1th second prediction; wherein, the offset of the Lth second prediction is greater than 0, and the offsets from the L+1th second prediction to the i-th second prediction are all equal to 0; L is a positive integer, and L is less than i; when i+1-L is equal to the first number threshold, determine that the offset at the i+1th second prediction is any offset in the preset offset set; when i+1-L is less than the first number threshold, determine that the offset at the i+1th second prediction is 0.

[0167] The first module training device 456 provided in the embodiment of the present application is implemented as an exemplary structure of a software module. In some embodiments, such as Figure 15 As shown, the software modules in the first module training device 456 may include: an acquisition module 4561, used to acquire a sample data set; the sample data in the sample data set includes a sample text unit sequence and a label text unit sequence; a first prediction module 4562, used to perform Q first predictions based on the sample text unit sequence to obtain Q first sample units; and splicing the Q first sample units to the sample text unit sequence to obtain a sixth text unit sequence; Q is greater than 0; a second prediction module 4563, used to perform K second predictions based on the sixth text unit sequence to obtain K second sample units; K is greater than 0; a second determination module 4564, used to determine the text prediction result of the sample text unit sequence based on the Q first sample units and the K second sample units; a loss calculation module 4565, used to perform loss calculation based on the first sample unit, the second sample unit, the text prediction result and the label text unit sequence to obtain a loss result; an update module 4566, used to update the model parameters of the first model based on the loss result to obtain the trained first model.

[0168] In some embodiments, the loss calculation module 4565 is also used to perform a first loss calculation based on the first sample unit and the label text unit sequence obtained from each first prediction to obtain a first loss value; perform a second loss calculation based on the second sample unit and the label text unit sequence obtained from each second prediction to obtain a second loss value; perform a third loss calculation based on the text prediction result and the label text unit sequence to obtain a third loss value; and fuse the first loss value, the second loss value and the third loss value to obtain the loss result.

[0169] It should be noted that the description of the device embodiment of the present application is similar to the description of the method embodiment described above, and has similar beneficial effects as the method embodiment, so it will not be repeated. For technical details not disclosed in the device embodiment, please refer to the description of the method embodiment of the present application for understanding.

[0170] Correspondingly, an embodiment of the present application provides an electronic device, Figure 16 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application, such as Figure 16 As shown, the electronic device 1200 includes at least: a processor 1201 , a communication interface 1202 and a storage medium 1203 configured to store executable instructions, wherein: the processor 1201 generally controls the overall operation of the electronic device 1200 .

[0171] The communication interface 1202 enables the electronic device to communicate with other terminals or servers through a network.

[0172] The storage medium 1203 is configured to store instructions and applications executable by the processor 1201, and can also cache data to be processed or processed by the processor 1201 and each module in the electronic device 1200, which can be implemented through flash memory (FLASH) or random access memory (RAM).

[0173] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the text processing method provided by the embodiment of the present application, for example, Figure 4 The text processing method shown, or the first model training method provided in the embodiment of the present application is executed.

[0174] An embodiment of the present application provides a computer program product, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the text processing method provided in the embodiment of the present application, or to perform the first model training method provided in the embodiment of the present application.

[0175] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0176] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0177] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0178] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0179] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A text processing method, characterized in that: The method comprises: Performing P first predictions based on the first text unit sequence to obtain P first text units; and concatenating the P first text units to the first text unit sequence to obtain a second text unit sequence; the first text unit sequence includes N text units, P and N are both greater than 0, and P is less than N; Performing M second predictions based on the second text unit sequence to obtain M second text units; wherein the (i+1) second prediction includes: masking some text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence; splicing i second text units obtained from the previous i second predictions to the third text unit sequence to obtain a fourth text unit sequence; performing the (i+1) second prediction based on the fourth text unit sequence to obtain the (i+1) second text unit; M is greater than 0, and i is an integer greater than 0 and less than M. Based on the P first text units and the M second text units, a text processing result of the first text unit sequence is determined.

2. The method according to claim 1, characterized in that The first first prediction includes: performing a first prediction based on the first text unit sequence to obtain a first first text unit; The j+1th first prediction includes: splicing j first text units obtained from the previous j first predictions to the first text unit sequence to obtain a fifth text unit sequence; A j+1th first prediction is performed based on the fifth text unit sequence to obtain the j+1th first text unit.

3. The method according to claim 2, characterized in that The step of masking some text units in the second text unit sequence that belong to the first text unit sequence to obtain a third text unit sequence includes: determining an offset of the second prediction; Starting from the starting position of the third text unit sequence in the i-th second prediction that belongs to the first text unit sequence, determining the third text unit at the position corresponding to the offset; The third text unit is masked to obtain a third text unit sequence of the (i+1)th second prediction.

4. The method according to claim 3, characterized in that The text processing method is implemented by a first model; for a first second prediction, determining an offset of the second prediction includes: Acquire a sample data set for training the first model; the sample data in the sample data set includes a sample text unit sequence and a label text unit sequence; Determining a mean sequence length of a sequence of sample text units in the sample data set; When the mean of the sequence lengths is greater than a first length threshold, determining a first ratio of each sample data; the first ratio is a ratio between the sequence length of the sample text unit sequence and the sequence length of the label text unit sequence; averaging the first ratios of the sample data in the sample data set to obtain a first average ratio; The offset of the first second prediction is determined based on the prediction number P of the first prediction and the first average ratio.

5. The method according to claim 4, characterized in that The method further comprises: When the mean value of the sequence length is less than or equal to the first length threshold, the offset of the first second prediction is determined based on the mean value of the sequence length; wherein the offset is less than the mean value of the sequence length.

6. The method according to claim 5, characterized in that Determining the offset of the second prediction includes: Determine an offset of the Lth second prediction before the (i+1)th second prediction; wherein the offset of the Lth second prediction is greater than 0, and the offsets from the (L+1)th second prediction to the i-th second prediction are all equal to 0; L is a positive integer, and L is less than i; When i+1-L is equal to the first number threshold, determining the offset of the i+1th second prediction to be any offset in the preset offset set; When i+1-L is less than the first threshold, the offset of the (i+1)th second prediction is determined to be 0.

7. A first model training method, characterized in that: The method comprises: Acquire a sample data set; the sample data in the sample data set includes a sample text unit sequence and a label text unit sequence; Performing Q first predictions based on the sample text unit sequence to obtain Q first sample units; and splicing the Q first sample units to the sample text unit sequence to obtain a sixth text unit sequence; Q is greater than 0; Perform K second predictions based on the sixth text unit sequence to obtain K second sample units; K is greater than 0; Determining a text prediction result of the sample text unit sequence based on the Q first sample units and the K second sample units; Performing loss calculation based on the first sample unit, the second sample unit, the text prediction result, and the label text unit sequence to obtain a loss result; The model parameters of the first model are updated based on the loss result to obtain a trained first model.

8. The method according to claim 7, characterized in that The performing loss calculation based on the first sample unit, the second sample unit, the text prediction result, and the label text unit sequence to obtain a loss result includes: Performing a first loss calculation based on the first sample unit and the label text unit sequence obtained from each first prediction to obtain a first loss value; Performing a second loss calculation based on the second sample unit and the label text unit sequence obtained from each second prediction to obtain a second loss value; Performing a third loss calculation based on the text prediction result and the label text unit sequence to obtain a third loss value; The first loss value, the second loss value, and the third loss value are fused to obtain the loss result.

9. An electronic device, characterized in that: include: a memory for storing computer-executable instructions; The processor is used to implement the text processing method described in any one of claims 1 to 6 when executing the computer-executable instructions stored in the memory, or to implement the first model training method described in claim 7 or 8.

10. A computer program product, characterized in that The computer program product includes computer-executable instructions stored in a computer-readable storage medium; In which, when the processor of the electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, it implements the text processing method described in any one of claims 1 to 6, or implements the first model training method described in claim 7 or 8.

Images