Pre-training model product and processing method, electronic device and computer storage medium

Abstract

Embodiments of the present application provide a pre-training model product and processing method, an electronic device and a computer storage medium, wherein a pre-training model processing method comprises: obtaining a rewritten text generated by a generator of a pre-training model based on a masked training text; predicting and generating a corresponding word quality ranking level for each word in the rewritten text through a hierarchical classifier of the pre-training model, wherein the word quality ranking level is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality ranking level comprises at least three levels; and training the pre-training model comprising the generator and the hierarchical classifier according to the word quality ranking level. Through the embodiments of the present application, the training accuracy and efficiency of the pre-training model can be improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a pre-training model processing method, a pre-training model product, an electronic device and a computer storage medium. BACKGROUND

[0002] With the development of NLP (Natural Language Processing) technology, language model pre-training is becoming an important part of NLP. Generally, the language model in the pre-training stage is also called a pre-training language model or a pre-training model. A well-pre-trained pre-training model can be effectively applied to downstream tasks after fine-tuning based on downstream task sample data, greatly improving the training efficiency of the model used for downstream tasks and reducing the training cost.

[0003] However, in order to further improve the pre-training efficiency of the pre-training model, in a current manner, a generator combined with a discriminator is used to implement the form of the pre-training model. In the pre-training stage, the generator part performs text prediction based on the result of mask processing on the input training sample, and generates the corresponding predicted text. Then, the discriminator performs training on the change identification of the predicted text with respect to the training sample based on the predicted text generated by the generator. Subsequently, the trained discriminator will be used for downstream task migration to implement different downstream NLP tasks after migration.

[0004] However, the current pre-training model and its pre-training method still have deficiencies in training accuracy and efficiency. How to further improve the training efficiency of the pre-training model and improve the accuracy of the training result has become a problem to be solved. SUMMARY

[0005] In view of this, embodiments of the present application provide a pre-training model processing and pre-training model product scheme to at least partially solve the above problems.

[0006] According to a first aspect of embodiments of the present application, a pre-training model processing method is provided, including: obtaining a rewritten text generated by a generator of a pre-training model based on a masked training text; predicting and generating a corresponding word quality arrangement level for each word in the rewritten text by a hierarchical classifier of the pre-training model, wherein the word quality arrangement level is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality arrangement level includes at least three levels; and training the pre-training model including the generator and the hierarchical classifier according to the word quality arrangement level.

[0007] According to a second aspect of the embodiments of the present application, a pre-trained model processing method is provided, including: receiving a user inputted to-be-processed text through an interactive interface, wherein the to-be-processed text at least includes a first text part and a second text part; obtaining a logical inference relationship between the first text part and the second text part determined by a preset natural language processing model after logical inference processing of the first text part and the second text part; wherein the natural language processing model is obtained based on task migration of a pre-trained model with completed training, and the pre-trained model includes a hierarchical partitioner used for predicting and generating logical inference relationships between multiple texts.

[0008] According to a third aspect of the embodiments of the present application, a pre-trained model processing method is provided, including: receiving a user inputted to-be-processed text through an interactive interface, wherein the to-be-processed text at least includes a first text part and a second text part; obtaining a semantic similarity between the first text part and the second text part determined by a preset natural language processing model after similarity judgment of the first text part and the second text part; wherein the natural language processing model is obtained based on task migration of a pre-trained model with completed training, and the pre-trained model includes a hierarchical partitioner used for predicting and generating semantic similarity information between multiple texts.

[0009] According to a fourth aspect of the embodiments of the present application, a pre-trained model processing method is provided, including: receiving a user inputted to-be-processed text through an interactive interface; obtaining a corresponding sentiment analysis result obtained by a preset natural language processing model after sentiment analysis of the to-be-processed text; wherein the natural language processing model is obtained based on task migration of a pre-trained model with completed training, and the pre-trained model includes a hierarchical partitioner used for predicting and generating sentiment information based on a text.

[0010] According to a fifth aspect of the embodiments of the present application, a pre-trained model processing method is provided, including: receiving a user inputted to-be-processed text through an interactive interface; obtaining a grammatical correctness result of the to-be-processed text determined by a preset natural language processing model after syntax analysis of the to-be-processed text; wherein the natural language processing model is obtained based on task migration of a pre-trained model with completed training, and the pre-trained model includes a hierarchical partitioner used for predicting and generating a grammatical correctness result based on a text.

[0011] According to a sixth aspect of the embodiments of the present application, a pre-training model product is provided, comprising: a generator unit, a level label determination unit, and a hierarchical partitioner unit; wherein: the generator unit is configured to generate corresponding rewritten text based on the masked training text; the level label determination unit is configured to determine a corresponding word quality ranking level label for each of the plurality of first words in the rewritten text according to a first generation probability corresponding to the plurality of first words in the rewritten text and a second generation probability corresponding to the plurality of second words in the training text; the hierarchical partitioner unit is configured to predict a corresponding word quality ranking level for each of the plurality of first words, wherein the word quality ranking level is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality ranking level comprises at least three levels; and the pre-training model is trained according to the difference between the predicted word quality ranking level corresponding to the plurality of first words in the rewritten text and the word quality ranking level label corresponding to the plurality of first words in the rewritten text.

[0012] According to a seventh aspect of the embodiments of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, the processor, the memory, and the communication interface are in communication with each other through the communication bus; the memory is configured to store at least one executable instruction, and the executable instruction causes the processor to perform operations corresponding to the method of any one of the first aspect to the sixth aspect.

[0013] According to an eighth aspect of the embodiments of the present application, a computer storage medium is provided, and the computer storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of the first aspect to the sixth aspect.

[0014] According to the scheme provided by the embodiments of the present application, in the pre-training model, a hierarchical partitioner is arranged, which can sort the quality levels of the words in the rewritten text generated by the generator of the pre-training model, to explicitly represent the quality level of a predicted word. Moreover, the word quality ranking level representing the quality level is divided into at least three levels, so that the quality of the rewritten text can be evaluated from a finer granularity. Since the hierarchical partitioner fully utilizes the information of the generator and learns the corresponding word quality features from the granularity, the pre-training model is adjusted and trained in reverse, so that the training precision and training efficiency of the pre-training model can be improved. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0016] Figure 1 A schematic diagram of an exemplary system suitable for use in embodiments of the present application;

[0017] Figure 2A A flowchart of steps of a pre-trained model processing method according to Embodiment One of the present application;

[0018] Figure 2B A structural block diagram of a pre-trained model according to the embodiment shown in Figure 2A

[0019] A training schematic diagram of a pre-trained model according to the embodiment shown in Figure 2C Figure 2A A first scene example diagram after downstream task migration of a pre-trained model trained according to the embodiment shown in

[0020] Figure 2D Figure 2A A second scene example diagram after downstream task migration of a pre-trained model trained according to the embodiment shown in

[0021] Figure 2E A third scene example diagram after downstream task migration of a pre-trained model trained according to the embodiment shown in Figure 2A

[0022] A fourth scene example diagram after downstream task migration of a pre-trained model trained according to the embodiment shown in Figure 2F Figure 2A A structural block diagram of a pre-trained model product according to Embodiment Two of the present application;

[0023] Figure 2G Figure 2A A structural schematic diagram of an electronic device according to Embodiment Three of the present application.

[0024] Figure 3 A structural block diagram of a pre-trained model product according to Embodiment Two of the present application;

[0025] Figure 4 A structural schematic diagram of an electronic device according to Embodiment Three of the present application. DETAILED DESCRIPTION

[0026] ​​​​To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0027] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0028] Figure 1 An exemplary system applicable to embodiments of this application is shown. For example... Figure 1 As shown, the system 100 may include a cloud server 102, a communication network 104, and / or one or more downstream devices 106. Figure 1 The example in the text shows multiple downstream devices. Among them, downstream device 106 can be a downstream server device or a high-performance downstream terminal device.

[0029] The cloud server 102 can be any suitable device for storing information, data, programs, and / or any other suitable type of content, including but not limited to distributed storage system devices, server clusters, computing cloud server clusters, etc. In some embodiments, the cloud server 102 can perform any suitable function. For example, in some embodiments, the cloud server 102 can be used to train a pre-trained model. As an optional example, in some embodiments, the cloud server 102 can be used to train a pre-trained model including a generator and a hierarchical partitioner. The generator of the pre-trained model is used to generate rewritten text based on masked training text; the hierarchical partitioner of the pre-trained model can predict the quality of the rewritten text generated by the generator, generating at least three levels of word quality ranking. Furthermore, based on the word quality ranking, the generator and the hierarchical partitioner are jointly trained to train the pre-trained model. Therefore, the pre-trained model in this embodiment is a pre-trained model in which the generator and the hierarchical partitioner cooperate, and through their cooperation and training, the pre-trained model has higher training accuracy and efficiency. In some embodiments, after training is completed, the cloud server 102 can send the pre-trained model to the downstream device 106 so that the downstream device 106 can perform downstream task transfer based on the pre-trained model, so as to apply it more effectively to downstream natural language processing tasks.

[0030] In some embodiments, the communication network 104 can be any suitable combination of one or more wired and / or wireless networks. For example, the communication network 104 can include any one or more of the Internet, an intranet, a Wide Area Network (WAN), a Local Area Network (LAN), a wireless network, a Digital Subscriber Line (DSL) network, a frame relay network, an Asynchronous Transfer Mode (ATM) network, a Virtual Private Network (VPN), and / or any other suitable communication network. The downstream device 106 can be connected to the communication network 104 through one or more communication links (e.g., communication links 112), which can be linked to the cloud server 102 via one or more communication links (e.g., communication links 114). The communication links can be any communication links suitable for communicating data among the downstream device 106 and the cloud server 102, such as network links, dial-up links, wireless links, hard-wired links, any other suitable communication links, or any suitable combination of such links.

[0031] In some embodiments, after obtaining the trained pre-training model from the cloud server 102, the downstream device 106 can select appropriate training samples based on the local task to fine-tune the pre-training model, so as to obtain a natural language processing model that can meet the requirements of the downstream task. As mentioned above, the downstream device 106 can be a downstream server device or a high-performance downstream terminal device.

[0032] Based on the above system, the embodiments of the present application provide a pre-training model processing method, which is described below through multiple embodiments.

[0033] Embodiment One

[0034] Reference Figure 2A Fig. 1 shows a step flowchart of a pre-training model processing method according to Embodiment One of the present application.

[0035] For ease of illustration, the structure of the pre-training model of the embodiments of the present application is first described below, as shown in Fig. 2. As shown in Fig. 2, the pre-training model can include a pre-training model 201 and a downstream task model 202. Figure 2B Figure 2B ​It can be seen that the pre-training model comprises a generator, a hierarchical partitioner, and a level label determination part located between the two. In one possible implementation, the generator and the hierarchical partitioner can be implemented by an encoder, for example, a Transformer structure-based encoder. The Transformer is an encoder-decoder model structure based on an attention mechanism. By using the attention mechanism, the Transformer can more efficiently extract the features of interest. In addition, in the embodiment of the present application, the size of the generator (including the number of hidden layers, etc.) is smaller than that of the hierarchical partitioner, so as to ensure the model training effect and reduce the implementation cost of the model.

[0036] In the pre-training stage, the generator and the hierarchical partitioner are jointly trained. After the training is completed, the pre-training model will be migrated to adapt to different downstream natural language processing tasks. After successful migration, the generated natural language model will mainly play a role in the hierarchical partitioner in the inference stage. Based on this, in the pre-training stage, a certain proportion (such as 15%) of the original training text is masked to generate a masked text, i.e., a masked training text, which is then input into the generator. The generator performs an MLM (Masked Language Model) task based on the input training text to predict a new word for each masked position to replace the original word, and keeps the original word unchanged for each non-masked position, thereby generating a rewritten text. Further, the level label determination part will determine a word quality ranking level label corresponding to each word of the rewritten text, which represents the prediction quality. This part will be described in detail below. On this basis, the hierarchical partitioner will predict the word quality ranking level based on the rewritten text with the label as supervision, and train the entire pre-training model according to the difference between the prediction result and the label.

[0037] Based on the pre-training model structure, the pre-training model processing method of the embodiment comprises the following steps:

[0038] Step S202: Based on the masked training text, the rewritten text generated by the generator of the pre-training model is obtained.

[0039] As described above, the generator can perform feature encoding on the input masked training text, and predict the corresponding rewritten text by performing an MLM task. Generally, the training text comprises a plurality of words, which are not limited to specific languages and forms, such as English, Chinese, numbers, symbols, etc. For the rewritten text predicted by the generator, some of the words included therein are the same as the original input training text, and some are different.

[0040] The generator, when predicting the output rewritten text, calculates the probability of each word in the preset word table based on the preset word table, and then outputs the corresponding word according to the probability. For example, assuming that there are 100,000 words in the preset word table, and taking the word "good" in the training text "Today's weather is really good" as an example, the generator generates a probability corresponding to the 100,000 words in the preset word table for this word (and the sum of the probabilities of the 100,000 words is 1). Then, according to the probability distribution, the generator outputs the corresponding word by sampling, such as "good" or "beautiful". The generator predicts and outputs new words for each word in the mask position of the training text by sampling, and keeps the original word unchanged at each non-mask position, thereby generating the rewritten text. However, there are probabilities corresponding to 100,000 words for each word position of the training text.

[0041] It should be noted that in the embodiments of the present application, unless otherwise specified, the number related to "multiple" such as "multiple", "multiple" and the like refers to two or more.

[0042] Step S204: predicting and generating corresponding word quality ranking for each word in the rewritten text through the hierarchical partitioner of the pre-trained model.

[0043] The word quality ranking is used to indicate the semantic difference level of the rewritten text and the training text, and the word quality ranking includes at least three levels. As described above, taking the training text "Today's weather is really good" as an example, if the corresponding rewritten text is "Today's weather is really beautiful", because "good" and "beautiful" have similar semantics in the sentence, the word quality ranking corresponding to the word can be a higher level; on the contrary, if the corresponding rewritten text is "Today's weather is really eat", because "good" and "eat" have a large semantic difference in the sentence, the word quality ranking corresponding to the word can be a lower level. Considering that the word corresponding to the mask part in the training text may be the same as the original word or different after being processed by the generator, in order to facilitate the fine-grained differentiation of the difference, and considering the same and the fine-grained differentiation, the word quality ranking is set to at least three levels in the present application.

[0044] In the embodiments of the present application, the hierarchical partitioner is used to generate the word quality ranking corresponding to each word in the rewritten text output by the generator as input, thereby learning the relationship between the multiple words.

[0045] In the case of supervised learning, the input level classifier further includes word quality ranking level labels corresponding to each word of the rewritten text. In this embodiment, the labels are automatically determined by the level label determination part according to the probabilities of each word in the preset word table corresponding to the word positions output by the generator, without manual annotation, greatly improving the annotation efficiency and reducing the annotation cost. Moreover, in this way, the generator and the level classifier are effectively combined into a mutually cooperative whole.

[0046] Based on this, in a feasible manner, before predicting and generating the corresponding word quality ranking level for each word in the rewritten text by the level classifier, the corresponding word quality ranking level label is further determined for each second word in the rewritten text according to the first generation probability corresponding to the first word in the rewritten text and the second generation probability corresponding to the second word in the training text. In this way, the corresponding word quality ranking level can be determined relatively objectively and accurately for the word. Moreover, compared with the way of evaluating the word quality by only distinguishing whether the word in the rewritten text is the same as the original word in the training text, the evaluation is more accurate, and more detailed and rich information is provided for subsequent model training, improving the model training effect and efficiency.

[0047] Still taking the word "good" in the training text "Today's weather is really good" as an example, if the prediction result is also "good" by sampling, i.e., the word at the corresponding position in the rewritten text is also "good", because they are the same, the probability corresponding to the second word in the training text (i.e., the original word in the training text) (i.e., the second word probability) and the probability corresponding to the first word in the rewritten text (i.e., the second word probability) are also the same. In this case, a higher word quality ranking level label can be determined for the first word, such as the first level when arranged from high to low. If the rewritten word is "beautiful" by sampling, i.e., different from the original word, and assuming that the probability corresponding to the word "beautiful" is 0.35 and the probability corresponding to the original word "good" is 0.3, the word quality ranking level label of the word "beautiful" can be determined according to the preset level range, for example, the second level, and so on.

[0048] The rank range can be preset by a person skilled in the art, but in order to make the rank range more reasonable and accurate, in a feasible manner, the corresponding word quality ranking label of each of the plurality of first words in the rewritten text can be determined according to the first generation probability corresponding to the plurality of first words in the rewritten text and the second generation probability corresponding to the plurality of second words in the training text, which can be implemented as follows: obtaining the plurality of first generation probabilities corresponding to the plurality of first words in the rewritten text, and the plurality of second generation probabilities of the plurality of second words corresponding to the plurality of first words in the training text; determining the corresponding plurality of rank ranges according to the plurality of second generation probabilities and the preset rank parameters; and determining the corresponding word quality ranking label of each of the plurality of first words according to the relationship between the plurality of first generation probabilities and the plurality of rank ranges.

[0049] In this way, the word quality ranking label of each word in the rewritten text can be determined. However, in order to further improve the label determination efficiency, in a feasible manner, it can also be determined according to the word position whether there is a first word in the plurality of first words in the rewritten text that is the same as the second word at the corresponding position in the training text; if there is, the same first word is set to have the highest word quality ranking label; and the operation of determining the corresponding word quality ranking label of each of the plurality of first words according to the first generation probability corresponding to the plurality of first words in the rewritten text and the second generation probability corresponding to the plurality of second words in the training text is performed on the first words that are not the same. That is, if the first word is the same as the second word at the corresponding position, the highest word quality ranking label is directly determined for it; and the operation of determining the word quality ranking label according to the relationship between the generation probability and the rank range is performed on those words that are not the same. Thus, for the case where the first word is the same as the second word, there is no need to compare the rank ranges, thereby further improving the label determination efficiency.

[0050] Further optionally, for a first word in the rewritten text, if it is not the same as the second word at the corresponding position in the training text, and the ranking level of the second generation probability of the second word at the corresponding position in the training text is not within the range of the word quality ranking level, the first word is set as a preset out-of-range label. If the generation probability of the second word is much smaller than its expected probability when the generator predicts and generates the corresponding word, that is, it is out of the preset range of the corresponding word quality ranking level, it indicates that the prediction deviation of the generator is large, which may affect the learning of the subsequent hierarchical classifier, and further affect the overall training effect of the model. Therefore, the out-of-range label is also set in the embodiment, which can be a default label. When this situation occurs, the first word is directly set as the default label. Then, in the subsequent learning, the first word can no longer be used as the basis, or it can be used as a sample different from the second word for coarse-grained word quality learning.

[0051] In the following, the determination process of the word quality ranking level label is exemplarily described through a specific example, as shown in Table 1 below.

[0052] Table 1

[0053]

[0054] In Table 1, K word quality ranking levels are set. If a first word (i.e., a word in the rewritten text) corresponds to a corresponding level, it will be set as the word quality ranking level label (Label) of the level. K is set by a person skilled in the art according to actual needs, and the embodiment of the present application does not limit this. i represents the second word (i.e., the original word in the training text) at the i-th position in the original training text, represents the first word at the i-th position in the rewritten text; represents the generation probability of the generator for the second word at the i-th position (second generation probability), represents the generation probability of the first word at the i-th position in the rewritten text input to the hierarchical classifier (first generation probability); T is a hyperparameter, representing the number of probabilities ranked first in the preset word table, represents that the generation probability of the second word at the i-th position predicted by the generator is within the first T of all word probabilities in the preset word table, where the specific setting of T can be set by a person skilled in the art according to actual conditions. For example, it can be equal to the number of word quality ranking levels, but is not limited to this. The case of inequality is also applicable to the scheme of the embodiment of the present application; τ represents a probability division hyperparameter, {τ1, τ2,..., τ K-2} is a set of probability division hyperparameters, where 0 < τ1< τ2< … < τ K-2wherein the specific setting of t can be set by those skilled in the art according to actual conditions, and exemplary values from small to large are (1, 4, 16,...), etc.; [] represents a closed interval, () represents an open interval, / represents division, ∧ represents the "and" relationship (i.e., the conditions connected by the coincidence need to be met at the same time), and ∈ represents the belonging relationship.

[0055] As can be seen from Table 1 above, when That is, when the first word at the i-th position in the rewritten text is the same as the second word, the word quality ranking label of the first word at the i-th position in the rewritten text is set to the highest level, i.e., the Kth level. In addition, if That is, when the first word at the i-th position in the rewritten text is different from the second word, and the level corresponding to the generation probability of the second word has exceeded the preset level, a label of exceeding the range is set for it, and the example in Table 1 above is "-1". For example, the second word is "good", the first word is "beautiful", and then the first word is different from the second word. Assuming that the generation probability of "good" predicted by the generator is the 100th in the preset word table, and T is 3, then The condition is met. Accordingly, the word quality ranking label corresponding to the "beautiful" word can be set to -1.

[0056] For other cases, the word quality ranking label corresponding to the first word can be determined through the calculation as shown in Table 1.

[0057] Through the above process, the word quality ranking labels of different first words can be accurately divided and determined, and input into the hierarchical classifier together with the rewritten text to learn the hierarchical classifier.

[0058] But in practical application, the first generation probability of some first words is at the edge of two different levels, which may cause misclassification of word quality ranking level label. Therefore, in a feasible manner, the above determination of the corresponding multiple level ranges according to the multiple second generation probabilities and the preset level parameters can also be implemented as: determining the corresponding multiple level ranges according to the multiple second generation probabilities and the preset level parameters and the preset level buffer bandwidth. Wherein, the level buffer bandwidth is used to set a range threshold between two adjacent word quality ranking levels, so as to avoid that the words corresponding to the range are misclassified, and thus the inaccuracy of the word quality ranking level label corresponding thereto. It should be noted that for the two adjacent word quality ranking levels, although the part of the words will not be classified into the two levels, for the higher or lower level, the part of the words has a clear level. For example, it is assumed that the word quality ranking levels from high to low are: K, K-1, K-2+Δ, K-2, K-3+Δ, K-3, wherein Δ represents the level buffer bandwidth. It is assumed that when a word is determined to be K-1 or K-2, it is classified into K-2+Δ level, then it does not belong to K-1 level or K-2 level. However, when K level is classified, it can be determined that it is not K level. And when K-3 level is classified, it can be clearly determined that it is not K-3 level.

[0059] In the following, through a specific example, the determination process of the word quality ranking level label considering the level buffer bandwidth is exemplarily illustrated, as shown in the following table 2.

[0060] Table 2

[0061]

[0062] In table 2, the same symbols as in table 1 have the same meanings as in table 1. Different from table 1, in order to facilitate accurate numerical calculation of the machine, the corresponding calculation in the table adopts the logit function corresponding to the probability. Specifically in this example, The meaning of represents the meaning of in table 1. logit (i) (x i |x M ) represents the meaning of in table 1. δ represents the level buffer bandwidth, which is also a hyperparameter, and its value is set by the person skilled in the art according to the actual situation. For a certain level k, it satisfies τ k (1+δ)<τ k+1 It is enough; corresponding to δ, in the other levels except K and-1 level, a buffer interval is generated due to δ, which is represented as increasing a correction value Δ in the corresponding interval, and Δ∈(0,1). It can be known that when Δ is 0, the result calculated according to table 2 will be the same as table 1.

[0063] Thus, a more accurate word quality ranking label determination of the first word in the rewritten text is achieved.

[0064] Based on the first words in the rewritten text and their corresponding word quality ranking labels, the input layer classifier is divided, and the predicted word quality ranking is generated by the hierarchical classifier. Based on the word quality ranking labels, the difference between the two can be used to jointly train the pre-trained model including the generator and the hierarchical classifier, as described below.

[0065] Step S206: Train the pre-trained model including the generator and the hierarchical classifier according to the word quality ranking.

[0066] The word quality ranking in this step means the word quality ranking generated by the hierarchical classifier for each first word. In a supervised learning mode, this step can be implemented as follows: based on the difference between the word quality ranking corresponding to the multiple first words of the rewritten text predicted by the hierarchical classifier and the word quality ranking labels corresponding to the multiple first words of the rewritten text, the pre-trained model including the generator and the hierarchical classifier is trained to improve the overall training speed and efficiency of the model.

[0067] Generally, model training adjustment is based on the calculation results of the loss function. Therefore, in one feasible implementation of the embodiments of the present application, the pre-trained model including the generator and the hierarchical classifier can be trained according to the word quality ranking predicted by the hierarchical classifier and the preset loss function; wherein the loss function includes a generator loss function and a hierarchical classifier loss function, wherein the hierarchical classifier loss function includes a first loss calculation part based on the judgment of whether the words at the corresponding positions in the rewritten text and the training text are the same, and a second loss calculation part for calculating the word quality ranking when the words are different. The first loss calculation part can also be considered as a loss calculation part for a coarse-grained division (based on whether the first word is the same as the second word), and the second loss calculation part is a loss calculation part for a fine-grained division (when the first word is not the same as the second word, the fine-grained quality level is divided).

[0068] In one example, the loss function of the hierarchical classifier part can be represented as:

[0069]

[0070] wherein the first part of the summation is the first loss calculation part, and the part after the second summation is the second loss calculation part. I[] represents an indicator function of the word quality ranking level; i represents a word position, i = (1, …, n), and n is the number of words of the rewritten text; R i represents the word quality ranking level label of the i-th first word; the meanings of K, K-1 and K-2 are as described in Table 1 or Table 2, and the meaning of Δ is as described in Table 2; P() represents a probability function; x C represents the rewritten text; γ r represents a weight hyperparameter, and exemplarily, γ r may be 0.5, r represents a word quality ranking level, and r ∈ {1, …, K-2}.

[0071] Through the above loss function, the loss of the hierarchical classifier part can be effectively calculated, and then combined with the loss function of the generator part, the joint training of the generator and the hierarchical classifier, that is, the training of the whole pre-training model, can be realized. The loss function of the generator part can use a conventional cross-entropy loss function.

[0072] Exemplarily, the loss function of the generator part can be represented as:

[0073]

[0074] wherein x M represents the masked training text; i represents a word position; represents a set containing all mask positions; represents the generation probability of the word at the i-th position in the masked training text.

[0075] Based on this, the loss function of the pre-training model can be represented as:

[0076]

[0077] wherein λ represents the weight corresponding to the hierarchical classifier loss function, and exemplarily, λ can be 50; χ represents a training sample set.

[0078] Through the present embodiment, a hierarchical classifier is set in the pre-training model, which can sort the quality levels of the words in the rewritten text generated by the generator of the pre-training model, to clearly represent the quality good or bad degree of the predicted word. Moreover, the word quality ranking level representing the quality good or bad degree is divided into at least three levels, so that the quality of the rewritten text can be evaluated from a finer granularity. Since the hierarchical classifier makes full use of the information of the generator and learns the corresponding word quality features from the granularity, the training model is adjusted and trained in reverse, so that the training accuracy and training efficiency of the pre-training model can be improved.

[0079] The above process is exemplarily illustrated with a specific example as follows. Figure 2C

[0080] Figure 2C The pre-training model in the above formula (1) only shows the generator and the hierarchical partitioner, but as previously described, a rank label determination part is also arranged between the two. Figure 2C In the above formula (1), the training text is “the pediatrician treated the child” (a pediatrician treated a child), which includes 5 words. Among them, 3 words are masked, Figure 2C In the above formula (1), the training text is “the pediatrician treated the child” (a pediatrician treated a child), which includes 5 words. Among them, 3 words are masked, Figure 2C In the above formula (1), the training text is “the pediatrician treated the child” (a pediatrician treated a child), which includes 5 words. Among them, 3 words are masked,

[0081] ​Further, the rank label determination part (not shown in the figure) determines the word quality rank label of each word in "the pediatrician told the boy" based on the generation probability of each word in "the pediatrician told the boy" and the generation probability of each word in "the pediatrician treated the child". In this example, the word quality rank is set to three levels, from low to high, as rank1, rank2, and rank3. Suppose that through the calculation as shown in Table 1 or Table 2, it is determined that the label corresponding to "the" is rank 3, the label corresponding to "pediatrician" is also rank 3, the label corresponding to "told" is rank 1, the label corresponding to "the" is rank 3, and the label corresponding to "boy" is rank 2. Then, each word of the rewritten text and the corresponding label are input into the ranker, which also adopts the Transformer encoder structure in this example. However, the structure size of the ranker is much larger than that of the generator, and it also has more hidden layers. After "the", "pediatrician", "told", "the", and "boy" are input into the ranker, the ranker performs word quality rank prediction of these words based on the feature encoding results, and the prediction results are combined with the labels of the input words through a loss function to calculate the loss value. Further, based on the loss value, the generator and the ranker are jointly trained until the training termination condition is reached. In an ideal state, the output result of the ranker is the same as the input label, as shown in Table 3. Figure 2C

[0082] As can be seen, through this example, the training accuracy and efficiency of the pre-training model are improved without increasing the calculation cost.

[0083] After the pre-training phase is completed, the pre-training model can be migrated to a downstream task, which is mainly realized by the ranker of the pre-training model. The following examples are used to illustrate the downstream migration and downstream task processing process.

[0084] Scenario one: natural language logical reasoning

[0085] After the ranker completes the training task in the pre-training phase, it can be migrated to the downstream. In this scenario, because the downstream task is a natural language logical reasoning task, a suitable amount of training samples and corresponding logical labels for the natural language logical reasoning task can be selected to fine-tune the ranker.

[0086] ​Exemplarily, the fine-tuning training data contains premise sentence text A, hypothesis sentence text B, and the label of whether there is an inference implication relationship between A and B: {"implication", "neutral", "contradiction"} in each sample. The hierarchical classifier is fine-tuned based on the training data, so that the trained hierarchical classifier becomes a natural language processing model capable of inferring whether there is an implication relationship between text C and text D.

[0087] After obtaining the natural language processing model, the client can receive the user inputted text to be processed through the interactive interface, wherein the text to be processed at least includes a first text part and a second text part. Then, the first text part and the second text part are processed by the preset natural language processing model to determine the logical inference relationship between the first text part and the second text part. The natural language processing model is obtained by task migration based on the trained pre-training model (the hierarchical classifier part thereof), and the pre-training model includes a hierarchical classifier for predicting the logical inference relationship between multiple texts. Then, the logical inference relationship is displayed to the user on the client.

[0088] Exemplarily, as shown in Figure 2D , if the text "Xiaomei exercises for an hour every day at 5 o'clock, and now her body is great" is inputted into the natural language processing model, the conclusion that the two text parts have an implication relationship will be outputted after processing by the natural language processing model.

[0089] Scenario two: text semantic similarity determination scenario

[0090] In this scenario, because the downstream task is a text semantic similarity determination task, a suitable amount of training samples for similarity determination and their corresponding semantic similarity labels can be selected to fine-tune the hierarchical classifier that has completed the pre-training phase.

[0091] Exemplarily, each sample of the fine-tuning training data contains text A and text B, which are labeled as similar. Another sample contains text C and text D, which are labeled as dissimilar. The hierarchical classifier is trained based on the training data, so that the trained hierarchical classifier becomes capable of determining whether text E and text F have semantic similarity.

[0092] After obtaining the natural language processing (NLP) model, the client can receive user-inputted text to be processed through an interactive interface. This text includes at least a first text portion and a second text portion. The pre-defined NLP model performs a similarity assessment on the first and second text portions, determining the semantic similarity between them. This NLP model is obtained through task transfer from a pre-trained model (including its hierarchical partitioner component). As mentioned earlier, this pre-trained model includes a hierarchical partitioner used to predict semantic similarity information between multiple generated texts. The obtained semantic similarity is then displayed to the user on the client side.

[0093] For example, such as Figure 2E As shown, assuming the input texts "The scenery in spring is really beautiful" and "Spring is really beautiful" are given to this natural language processing model, the model will output the conclusion that the two texts have semantic similarity after processing, which is indicated as "semantic similarity" in the figure.

[0094] Scenario 3: Sentiment Analysis Scenario

[0095] In this scenario, since the downstream task is a sentiment analysis task for text, an appropriate amount of training samples and their corresponding sentiment labels can be selected to fine-tune the hierarchical divider of the training task that has completed the pre-training stage.

[0096] For example, each sample in the fine-tuned training data is labeled with a sentiment tag of "like," "indifferent," or "dislike." A hierarchical classifier is trained based on this training data, with the goal of developing a hierarchical classifier capable of determining the sentiment expressed in the text.

[0097] After obtaining the natural language processing (NLP) model, the client can receive user-inputted text for processing via an interactive interface. The pre-defined NLP model then performs sentiment analysis on the text, yielding the corresponding sentiment analysis results. This NLP model is obtained through task transfer from a pre-trained model (including its hierarchical partitioner component). As mentioned earlier, this pre-trained model includes a hierarchical partitioner used to predict and generate sentiment information based on the text. The obtained sentiment analysis results can then be displayed to the user on the client side.

[0098] For example, such as Figure 2F As shown, if the input text "This plot made me laugh for a long time" is given to the natural language processing model, the model will output the conclusion that the emotion corresponding to the text is "like".

[0099] Scenario 4: Syntax Analysis Scenario

[0100] In this scenario, since the downstream task is a syntax analysis task for text, a suitable amount of training samples for syntax analysis and their corresponding syntax labels can be selected to fine-tune the hierarchical partitioner that has completed the pre-training phase.

[0101] For example, each sample of the fine-tuning training data has a "yes" or "no" label indicating the syntax correctness of the sample, where "yes" indicates syntax correctness and "no" indicates syntax error. The hierarchical partitioner is trained based on the training data, so that the trained hierarchical partitioner can output the correctness of the syntax of the text.

[0102] After obtaining the natural language processing model, the client can receive the user input text to be processed through the interactive interface; and determine the syntax correctness result of the text to be processed after the natural language processing model performs syntax analysis on the text to be processed. The natural language processing model is obtained based on the task migration of the trained pre-training model (the hierarchical partitioner part), as described above, the pre-training model includes a hierarchical partitioner, which is used to predict the syntax correctness result based on the text. Further, the client can show the user the obtained syntax correctness result.

[0103] For example, as shown in Figure 2G , assuming that the text "I ate lunch" is input to the natural language processing model, the natural language processing model will output the syntax analysis result corresponding to the text, such as "no", indicating that the text has syntax error.

[0104] As can be seen from the above, the natural language processing model obtained based on the hierarchical partitioner for task migration can be effectively applied to various scenarios, and the above scenario is only an example. Those skilled in the art can perform downstream task migration and downstream task execution according to actual needs. The natural language processing model obtained by the hierarchical partitioner has better accuracy and higher efficiency.

[0105] Embodiment Two

[0106] Referring to Figure 3 , a structural block diagram of a pre-training model product according to Embodiment Two of the present application is shown.

[0107] The pre-training model product of this embodiment includes a generator unit 302, a level label determination unit 304, and a hierarchical partitioner unit 306.

[0108] The generator unit 302 is configured to generate the corresponding rewritten text based on the masked training text; the rank label determination unit 304 is configured to determine, for each of the first words in the rewritten text, a word quality rank label corresponding to the first word according to a first generation probability of the first word and a second generation probability of a second word in the training text corresponding to the first word; the hierarchical partitioner unit 306 is configured to predict, for each of the first words, a word quality rank corresponding to the first word, wherein the word quality rank is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality rank includes at least three levels; and the pre-training model is trained according to a difference between the predicted word quality rank of each of the first words and the word quality rank label corresponding to the first word in the rewritten text.

[0109] Optionally, the rank label determination unit 304 is configured to obtain a plurality of first generation probabilities respectively corresponding to the first words in the rewritten text, and a plurality of second generation probabilities respectively corresponding to a plurality of second words in the training text corresponding to the first words in the rewritten text; determine a plurality of rank ranges corresponding to the plurality of second generation probabilities according to a preset rank parameter; and determine, for each of the first words, a word quality rank label corresponding to the first word according to a relationship between the first generation probability of the first word and the rank range corresponding to the first word.

[0110] Optionally, the rank label determination unit 304 is further configured to determine, according to a word position, whether there is a first word in the first words in the rewritten text that is the same as a second word at a corresponding position in the training text; if there is, set the same first word to a highest word quality rank label; and perform the operation of determining, for each of the first words in the rewritten text, a word quality rank label corresponding to the first word according to a first generation probability of the first word and a second generation probability of a second word in the training text corresponding to the first word, on the first words that are not the same.

[0111] Optionally, the rank label determination unit 304 is further configured to, for a certain first word in the rewritten text, if the first word is not the same as a second word at a corresponding position in the training text, and a rank of the second generation probability of the second word at the corresponding position in the training text is not within a range of the word quality rank, set the first word to a preset out-of-range label.

[0112] Optionally, when the rank label determination unit 304 determines a plurality of rank ranges corresponding to the plurality of second generation probabilities according to a preset rank parameter, the rank label determination unit 304 can determine the plurality of rank ranges according to the plurality of second generation probabilities, the preset rank parameter, and a preset rank buffer bandwidth.

[0113] Optionally, the hierarchical classifier unit 306 is configured to predictively generate a corresponding word quality ranking for each of the plurality of first words in the rewritten text, wherein the word quality ranking is used to indicate a level of semantic difference between the predicted rewritten text and the training text, and the word quality ranking comprises at least three levels; and the pre-training model is trained according to the word quality ranking and a preset loss function; wherein the loss function comprises a generator loss function and a hierarchical classifier loss function, and the hierarchical classifier loss function comprises a first loss calculation part configured to calculate based on a judgment of whether the words at the corresponding positions in the rewritten text and the training text are the same (indicating whether the quality is the highest level), and a second loss calculation part configured to calculate the word quality ranking for the different words.

[0114] The pre-training model product of the present embodiment is used to implement the aforementioned pre-training model processing method and has the beneficial effects of the corresponding parts in the method embodiments, which will not be repeated here. The description of each part of the pre-training model product of the present embodiment is relatively simple, and the relevant parts are implemented with reference to the corresponding parts in Embodiment One.

[0115] Embodiment Three

[0116] Referring to Figure 4 , a structural schematic diagram of an electronic device according to Embodiment Three of the present application is shown, and the specific implementation of the electronic device is not limited in the specific embodiments of the present application.

[0117] As shown in Figure 4 , the electronic device can include a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0118] Among them:

[0119] The processor 402, the communications interface 404, and the memory 406 complete mutual communication through the communications bus 408.

[0120] The communications interface 404 is configured to communicate with other electronic devices or servers.

[0121] The processor 402 is configured to execute the program 410, and specifically can execute the related steps of any one of the above-mentioned method embodiments.

[0122] Specifically, the program 410 can include program code, and the program code includes computer operation instructions.

[0123] The processor 402 can be a CPU, or a GPU (Graphic Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0124] The memory 406 is configured to store a program 410. The memory 406 can include a high-speed RAM memory, and can further include a non-volatile memory, such as at least one disk memory.

[0125] The program 410 can include a plurality of computer instructions, and the program 410 can specifically cause the processor 402 to perform operations corresponding to the pre-trained model processing method described in the foregoing method embodiments.

[0126] The specific implementation of each step in the program 410 can refer to the corresponding description in the corresponding steps and units in the foregoing method embodiments, and has corresponding beneficial effects, which will not be described here. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the devices and modules described above can refer to the corresponding process description in the foregoing method embodiments, which will not be described here.

[0127] The embodiments of the present application further provide a computer storage medium, which stores a computer program, and the program is executed by a processor to implement the method described in any one of the foregoing method embodiments. The computer storage medium includes but is not limited to: a CD-ROM (Compact Disc Read-Only Memory), a RAM (Random Access Memory), a floppy disk, a hard disk, or a magneto-optical disk, etc.

[0128] The embodiments of the present application further provide a computer program product, which includes computer instructions, and the computer instructions instruct a computing device to perform operations corresponding to any one of the pre-trained model processing methods in the foregoing method embodiments.

[0129] In addition, it should be noted that the information related to the user (including but not limited to user equipment information, user personal information, etc.) and data (including but not limited to sample data for training the model, data for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the country and region, and provide corresponding operation portal for the user to choose authorization or refusal.

[0130] It should be noted that, according to the needs of implementation, each component / step described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or part of the operation of the components / steps can be combined into a new component / step, to achieve the purpose of the embodiments of the present application.

[0131] The above method according to the embodiments of the present application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium such as a CD-ROM, RAM, floppy disk, hard disk or magneto-optical disk, or downloaded through a network and stored in a remote recording medium or non-transitory machine readable medium and then stored in a local recording medium, so that the method described herein can be stored in such software processing of recording medium using a general computer, special processor or programmable or special hardware such as Application Specific Integrated Circuit (ASIC) or Field Programmable Gate Array (FPGA). It can be understood that the computer, processor, microprocessor controller or programmable hardware includes storage components (for example, Random Access Memory (RAM), Read-Only Memory (ROM), flash memory, etc.) that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, processor or hardware, the method described herein is implemented. In addition, when the general computer accesses the code for implementing the method shown herein, the execution of the code will convert the general computer into a special computer for executing the method shown herein.

[0132] Those skilled in the art can understand that the units and method steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for a specific application, but such implementation should not be considered as beyond the scope of the embodiments of the present application.

[0133] The above embodiments are only used to illustrate but not to limit the embodiments of the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application, and all equivalent technical solutions belong to the scope of the embodiments of the present application. The patent protection scope of the embodiments of the present application should be defined by the claims.

Claims

1. A pre-trained model processing method, characterized in that, The method comprises: obtaining a rewritten text generated by a generator of a pre-trained model based on a masked training text; determining a word quality ranking label for each of a plurality of first words in the rewritten text according to a first generation probability corresponding to the first word and a second generation probability corresponding to a second word in the training text; predicting a word quality ranking for each of a plurality of words in the rewritten text by a hierarchical classifier of the pre-trained model, wherein the word quality ranking is used to indicate a semantic difference level between the rewritten text and the training text, and the word quality ranking comprises at least three levels; training the pre-trained model comprising the generator and the hierarchical classifier according to the word quality ranking; the training of the pre-trained model comprising the generator and the hierarchical classifier according to the word quality ranking comprises training the pre-trained model according to a difference between the word quality ranking predicted by the hierarchical classifier and the word quality ranking label corresponding to the first word in the rewritten text.

2. The method of claim 1, wherein, The method comprises: obtaining a plurality of first generation probabilities corresponding to a plurality of first words in the rewritten text, and a plurality of second generation probabilities corresponding to a plurality of second words in the training text corresponding to the plurality of first words; determining a plurality of level ranges according to the plurality of second generation probabilities and a preset level parameter; determining a word quality ranking label for each of the plurality of first words according to a relationship between the plurality of first generation probabilities and the plurality of level ranges.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: determining whether there is a first word in the plurality of first words that is the same as a second word in the training text at a corresponding position according to a word position; if there is, setting the same first word to a highest word quality ranking label; and performing the operation of determining a word quality ranking label for each of the plurality of first words according to a first generation probability corresponding to the first word in the rewritten text and a second generation probability corresponding to a second word in the training text.

4. The method of claim 3, wherein, The method further comprises: for a certain first word in the rewritten text, if the first word is not the same as a second word in the training text at a corresponding position, and a ranking of the second generation probability of the second word in the training text at the corresponding position is not within the range of the word quality ranking, setting the first word to a preset out-of-range label.

5. The method of claim 2, wherein, The method further comprises: According to the second generation probability, a preset level parameter and a preset level buffer bandwidth, a corresponding multiple level range is determined.

6. The method according to any one of claims 1-2, characterized in that, The pre-training model including the generator and the hierarchical classifier is trained according to the word quality arrangement level. The pre-training model including the generator and the hierarchical classifier is trained according to the word quality arrangement level and a preset loss function. The loss function includes a generator loss function and a hierarchical classifier loss function, wherein the hierarchical classifier loss function includes a first loss calculation part based on the judgment of whether the words at the corresponding positions in the rewritten text and the training text are the same, and a second loss calculation part for calculating the word quality arrangement level of the different words.

7. A pre-trained model processing method, comprising: The method comprises: A user inputs a to-be-processed text through an interactive interface, wherein the to-be-processed text at least includes a first text part and a second text part; A preset natural language processing model is used to determine the logical inference relationship between the first text part and the second text part after logical inference processing of the first text part and the second text part. The natural language processing model is obtained by task migration based on a trained pre-training model, and the pre-training model includes a hierarchical classifier and a generator. The method further comprises: obtaining a rewritten text generated by the generator based on the masked training text; determining a corresponding word quality arrangement level label for each of the first words in the rewritten text according to the first generation probability corresponding to the first words and the second generation probability corresponding to the second words in the training text; predicting and generating a corresponding word quality arrangement level for each of the words in the rewritten text through the hierarchical classifier, wherein the word quality arrangement level is used to indicate the semantic difference level of the rewritten text and the training text, and the word quality arrangement level includes at least three levels; and training the pre-training model including the generator and the hierarchical classifier according to the difference between the word quality arrangement level of the first words in the rewritten text predicted and generated by the hierarchical classifier and the word quality arrangement level label of the first words in the rewritten text, to obtain the trained pre-training model.

8. A pre-trained model processing method, comprising: The method comprises: A user inputs a to-be-processed text through an interactive interface, wherein the to-be-processed text at least includes a first text part and a second text part; A preset natural language processing model is used to determine the semantic similarity between the first text part and the second text part after similarity judgment of the first text part and the second text part. The natural language processing model is obtained by task migration based on a trained pre-training model, and the pre-training model includes a hierarchical classifier and a generator. The method further comprises: obtaining the rewritten text generated by the generator based on the masked training text; determining a corresponding word quality ranking level label for each of the first words in the rewritten text according to a first generation probability corresponding to the first words and a second generation probability corresponding to the second words in the training text; predicting a corresponding word quality ranking level for each of the words in the rewritten text by the hierarchical classifier, wherein the word quality ranking level is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality ranking level comprises at least three levels; and training a pre-training model comprising the generator and the hierarchical classifier according to the difference between the word quality ranking level corresponding to the first words in the rewritten text predicted by the hierarchical classifier and the word quality ranking level label corresponding to the first words in the rewritten text, to obtain the trained pre-training model.

9. A pre-trained model processing method, comprising: The method comprises: receiving a to-be-processed text input by a user through an interactive interface; obtaining a corresponding sentiment analysis result obtained after a preset natural language processing model performs sentiment analysis on the to-be-processed text; The natural language processing model is obtained by task migration based on a trained pre-training model, and the pre-training model comprises a hierarchical classifier and a generator, and the hierarchical classifier is used to predict and generate sentiment information based on a text. The method further comprises: obtaining the rewritten text generated by the generator based on the masked training text; determining a corresponding word quality ranking level label for each of the first words in the rewritten text according to a first generation probability corresponding to the first words and a second generation probability corresponding to the second words in the training text; predicting a corresponding word quality ranking level for each of the words in the rewritten text by the hierarchical classifier, wherein the word quality ranking level is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality ranking level comprises at least three levels; and training a pre-training model comprising the generator and the hierarchical classifier according to the difference between the word quality ranking level corresponding to the first words in the rewritten text predicted by the hierarchical classifier and the word quality ranking level label corresponding to the first words in the rewritten text, to obtain the trained pre-training model.

10. A pre-trained model processing method, comprising: The method comprises: receiving a to-be-processed text input by a user through an interactive interface; obtaining a corresponding sentiment analysis result obtained after a preset natural language processing model performs sentiment analysis on the to-be-processed text; The natural language processing model is obtained by task migration based on a trained pre-training model, and the pre-training model comprises a hierarchical classifier and a generator, and the hierarchical classifier is used to predict and generate sentiment information based on a text. The method further comprises: obtaining the rewritten text generated by the generator based on the masked training text; determining a corresponding word quality ranking level label for each of the first words in the rewritten text according to a first generation probability corresponding to the first words and a second generation probability corresponding to the second words in the training text; predicting a corresponding word quality ranking level for each of the words in the rewritten text by the hierarchical classifier, wherein the word quality ranking level is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality ranking level comprises at least three levels; and training a pre-training model comprising the generator and the hierarchical classifier according to the difference between the word quality ranking level corresponding to the first words in the rewritten text predicted by the hierarchical classifier and the word quality ranking level label corresponding to the first words in the rewritten text, to obtain the trained pre-training model. The method further comprises: obtaining the rewritten text generated by the generator based on the masked training text; determining a word quality ranking label corresponding to each of the first words in the rewritten text according to a first generation probability corresponding to the first words and a second generation probability corresponding to the second words in the training text; predicting a word quality ranking corresponding to each of the words in the rewritten text by the hierarchical classifier, wherein the word quality ranking is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality ranking comprises at least three levels; and training the pre-training model comprising the generator and the hierarchical classifier according to the difference between the word quality ranking corresponding to each of the first words in the rewritten text predicted by the hierarchical classifier and the word quality ranking label corresponding to each of the first words in the rewritten text, to obtain the trained pre-training model.

11. A pre-trained model product, characterized in that, The method comprises: a generator unit, a level label determination unit, and a hierarchical classifier unit, and a training unit; The generator unit is configured to obtain the rewritten text generated by the generator of the pre-training model based on the masked training text. The level label determination unit is configured to determine a word quality ranking label corresponding to each of the first words in the rewritten text according to a first generation probability corresponding to the first words and a second generation probability corresponding to the second words in the training text. The hierarchical classifier unit is configured to predict a word quality ranking corresponding to each of the first words by the hierarchical classifier of the pre-training model, wherein the word quality ranking is used to indicate a semantic difference level of the rewritten text and the training text, and the word quality ranking comprises at least three levels. The training unit is configured to train the pre-training model comprising the generator and the hierarchical classifier according to the word quality ranking. The training unit is further configured to train the pre-training model according to the difference between the word quality ranking corresponding to each of the first words predicted by the hierarchical classifier and the word quality ranking label corresponding to each of the first words. The method comprises:

12. An electronic device, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface complete communication with each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform corresponding operations of the method according to any one of claims 1-10. A computer program is stored thereon, and the program is executed by a processor to implement the method according to any one of claims 1-10.

13. A computer storage medium, characterized in that, ​

Images