Model training methods, text processing methods, devices and computer equipment

Abstract

This specification discloses a model training method, a text processing method, an apparatus, and a computer device. The model training method includes: acquiring original text, positive samples, and negative samples, where the positive and negative samples are rewritten versions of the original text; determining, through an encoder, a first distribution parameter corresponding to the original text, a second distribution parameter corresponding to the positive samples, and a third distribution parameter corresponding to the negative samples, wherein the first distribution parameter is used to define a first semantic distribution, the second distribution parameter is used to define a second semantic distribution, and the third distribution parameter is used to define a third semantic distribution; and optimizing the encoder's model parameters based on the first, second, and third distribution parameters using a loss function, wherein the loss function includes a first term, which constrains the similarity between the first and second semantic distributions to be greater than the similarity between the first and third semantic distributions. This specification embodiment can improve model training performance.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a model training method, a text processing method, an apparatus, and a computer device. Background Technology

[0002] Deep learning is gradually becoming a mainstream business processing method in the industry, and can be used in scenarios such as natural language processing. With the development of internet technology, the performance requirements for machine learning models are becoming increasingly demanding. Therefore, it is necessary to provide a model training method to ensure that the trained machine learning model has high performance, such as high accuracy. Summary of the Invention

[0003] This specification provides a model training method, a text processing method, an apparatus, and a computer device. These embodiments can train a model. Furthermore, they can also rewrite original text.

[0004] A first aspect of the embodiments of this specification provides a model training method, including:

[0005] Obtain the original text, positive samples, and negative samples, wherein the positive samples and negative samples are rewritten versions of the original text;

[0006] The encoder determines the first distribution parameter corresponding to the original text, the second distribution parameter corresponding to the positive sample, and the third distribution parameter corresponding to the negative sample. The first distribution parameter is used to limit the first semantic distribution, the second distribution parameter is used to limit the second semantic distribution, and the third distribution parameter is used to limit the third semantic distribution.

[0007] Based on the first distribution parameter, the second distribution parameter, and the third distribution parameter, the model parameters of the encoder are optimized by a loss function. The loss function includes a first term, which is used to constrain the similarity between the first semantic distribution and the second semantic distribution to be greater than the similarity between the first semantic distribution and the third semantic distribution.

[0008] A second aspect of the embodiments of this specification provides a text processing method, including:

[0009] The original text is input into the encoder to obtain distribution parameters, the encoder being trained according to the method described in the first aspect, the distribution parameters being used to define the semantic distribution;

[0010] Sampling is performed in the semantic distribution to obtain latent variables;

[0011] The hidden variables are input into the decoder to obtain the rewritten text.

[0012] A third aspect of the embodiments of this specification provides a model training apparatus, wherein the text generation model includes an encoder, and the apparatus includes:

[0013] The acquisition unit is used to acquire the original text, positive samples, and negative samples, wherein the positive samples and negative samples are rewritten texts of the original text;

[0014] The determining unit is used to determine, through the encoder, a first distribution parameter corresponding to the original text, a second distribution parameter corresponding to the positive sample, and a third distribution parameter corresponding to the negative sample. The first distribution parameter is used to limit the first semantic distribution, the second distribution parameter is used to limit the second semantic distribution, and the third distribution parameter is used to limit the third semantic distribution.

[0015] An optimization unit is used to optimize the encoder's model parameters based on a first distribution parameter, a second distribution parameter, and a third distribution parameter using a loss function. The loss function includes a first term, which constrains the similarity between the first semantic distribution and the second semantic distribution to be greater than the similarity between the first semantic distribution and the third semantic distribution.

[0016] A fourth aspect of the embodiments of this specification provides a text processing apparatus, including:

[0017] The first input unit is used to input the original text into the encoder to obtain distribution parameters, wherein the encoder is trained according to the method described in the first aspect, and the distribution parameters are used to define the semantic distribution.

[0018] A sampling unit is used to sample from a semantic distribution to obtain latent variables;

[0019] The second input unit is used to input the latent variables into the decoder to obtain the rewritten text.

[0020] A fifth aspect of the embodiments of this specification provides a computer device, including:

[0021] At least one processor;

[0022] A memory storing program instructions configured to be executed by the at least one processor, the program instructions including instructions for performing the methods as described in the first or second aspect.

[0023] The technical solutions provided in the embodiments of this specification can improve training effectiveness by employing comparative learning. Furthermore, the technical solutions provided in the embodiments of this specification can also rewrite text. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. The drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the model training method in the embodiments of this specification;

[0026] Figure 2 This is a schematic diagram of the training process of the model in the embodiments of this specification;

[0027] Figure 3 This is a flowchart illustrating the text processing method in the embodiments of this specification;

[0028] Figure 4 This is a schematic diagram of a search scenario in an embodiment of this specification;

[0029] Figure 5 This is a schematic diagram of the model training device in the embodiments of this specification;

[0030] Figure 6 This is a schematic diagram of the text processing device in the embodiments of this specification;

[0031] Figure 7 This is a schematic diagram of the structure of the computer device in the embodiments of this specification. Detailed Implementation

[0032] The technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0033] In natural language processing scenarios, it is often necessary to rewrite the original text. For example, in search systems, due to language conventions, the semantics of the search text entered by the user are not accurate. A rewritten text can be obtained from the search text. The rewritten text expresses relatively accurate semantics. Therefore, search results can be provided to the user based on the rewritten text.

[0034] In related technologies, a "teacher forcing" method can be used to train text generation models. This allows the generated rewritten text to be produced from the original text. However, training a text generation model using "teacher forcing" can introduce exposure bias. For example, during training, the text generation model predicts the next word based on the previous word in the training samples. During application, the model predicts the next word based on its own previous prediction. Because the model never uses its own previous prediction during training, if the previous word is predicted incorrectly during application, the next word will also be incorrect, leading to an accumulation of errors.

[0035] The text generation models described in this specification can include generative models such as Variational Auto Encoder (VAE) and Conditional Variational Auto Encoder (CVAE). Of course, the text generation models described in this specification can also include other generative models. For example, the text generation model can also include Generative Adversarial Networks (GANs). The text generation model can rewrite the original text into new text (hereinafter referred to as rewritten text). The rewritten text and the original text have the same semantics. Moreover, compared to the original text, the rewritten text expresses a relatively accurate semantic meaning. The text generation model can include an encoder and a decoder. The encoder is used to output distribution parameters based on the original text. The distribution parameters are used to define the semantic distribution corresponding to the original text. By sampling the semantic distribution, latent variables can be obtained. The decoder is used to output the rewritten text based on the latent variables. The encoder and the decoder can be neural network models. The neural network model may include multilayer perceptron (MLP), convolutional neural network (CNN) models, etc.

[0036] This specification provides a model training method. This model training method can be used to train a text generation model. Specifically, the model training method can train only the encoder in the text generation model. Alternatively, the model training method can train both the encoder and decoder in the text generation model simultaneously. This model training method can be applied to computer devices. The computer devices include, but are not limited to, personal computers, servers, and server clusters containing multiple servers. Please refer to the following documentation. Figure 1 and Figure 2 The model training method includes the following steps.

[0037] Step S11: Obtain the original text, positive samples, and negative samples.

[0038] In some embodiments, the original text, the positive sample, and the negative sample may be related. The positive sample and the negative sample may be rewritten versions of the original text. The positive sample may be a high-quality rewritten text. The positive sample and the original text have the same semantics. Compared to the original text, the positive sample expresses a relatively accurate semantic meaning. The negative sample may be a low-quality rewritten text. The negative sample and the original text have different semantics. Alternatively, compared to the original text, the negative sample expresses a relatively inaccurate semantic meaning.

[0039] The original text, the positive samples, and the negative samples may all include statements, paragraphs consisting of multiple statements, etc. A statement may include one or more keywords. A keyword may include one or more characters.

[0040] For example, the original text could be "China Merchants Bank Credit Card Application", the positive sample could be "China Merchants Bank Credit Card Application", and the negative sample could be "China Merchants Bank Credit Card Application". As another example, the original text could be "iQiyi Monthly Subscription", the positive sample could be "iQiyi Membership", and the negative sample could be "iQiyi VIP".

[0041] The original text, the positive samples, and the negative samples can be one or more. Positive samples may not have labels, or they may have corresponding labels. Negative samples may not have labels, or they may have corresponding labels. The labels are used to distinguish between positive and negative samples. The labels can be selected from 0 and 1. For example, the label corresponding to a positive sample can be 0, and the label corresponding to a negative sample can be 1. 0 and 1 here are merely examples. In practical applications, the labels can also be in other forms.

[0042] In some embodiments, the original text, the positive samples, and the negative samples can be obtained through acquisition. For example, the user-input search text can be acquired from search logs as the original text; rewritten text that significantly improves the quality of search results can be acquired from search logs as positive samples; and rewritten text that does not significantly improve the quality of search results can be acquired from search logs as negative samples. Alternatively, other devices can send the original text, the positive samples, and the negative samples to the computer device. The computer device can receive the original text, the positive samples, and the negative samples sent by other devices. Alternatively, a machine learning model can be used to generate corresponding positive and negative samples for the original samples.

[0043] Step S13: Obtain the first distribution parameter corresponding to the original text, the second distribution parameter corresponding to the positive sample, and the third distribution parameter corresponding to the negative sample through the encoder. The first distribution parameter is used to limit the first semantic distribution, the second distribution parameter is used to limit the second semantic distribution, and the third distribution parameter is used to limit the third semantic distribution.

[0044] In some embodiments, the first semantic distribution, the second semantic distribution, and the third semantic distribution may include a normal distribution. The first distribution parameter, the second distribution parameter, and the third distribution parameter may include the mean and variance. Of course, the first semantic distribution, the second semantic distribution, and the third semantic distribution may also be other probability distributions, and correspondingly, the first distribution parameter, the second distribution parameter, and the third distribution parameter may also be other distribution parameters. For example, the first semantic distribution, the second semantic distribution, and the third semantic distribution may also include a uniform distribution. The first distribution parameter, the second distribution parameter, and the third distribution parameter may also include boundary parameters.

[0045] Semantic distributions are used to represent the semantic space to which text belongs. Different points on the semantic distribution can represent different textual expressions under the same semantic meaning. The first semantic distribution is used to represent the semantic space to which the original text belongs, and can thus be understood as a prior distribution. The second semantic distribution is used to represent the semantic space to which positive samples belong, and can thus be understood as a posterior distribution. The third semantic distribution is used to represent the semantic space to which negative samples belong, and can thus be understood as a posterior distribution.

[0046] In some embodiments, the encoder may include a first encoder and a second encoder. The first encoder may be an encoder in a text generation model. In some scenario examples, the text generation model may include a conditional variational autoencoder. The second encoder may be an auxiliary encoder, which can be used to assist in the training of the text generation model. The model structures of the second encoder and the first encoder may be the same or different. A first distribution parameter can be obtained through the first encoder; a second distribution parameter and a third distribution parameter can be obtained through the second encoder.

[0047] The original text can be input into the first encoder to obtain the first distribution parameters. Alternatively, keywords in the original text can be used as constraints on the first semantic distribution. Adding constraints helps the encoder learn more accurate information, thereby improving the encoder's output. Specifically, keywords in the original text can be concatenated with the original text and then input into the first encoder to obtain the first distribution parameters. The keywords in the original text may include entity keywords. Named Entity Recognition (NER) can be performed on the original text to obtain entity keywords. These entity keywords represent entities with specific meanings, such as names of people, place names, and proper nouns. The concatenation may include splicing. For example, keywords in the original text can be concatenated with the original text and then input into the first encoder.

[0048] Positive samples can be input into the second encoder to obtain the second distribution parameters. Alternatively, the labels corresponding to the positive samples can be used as constraints on the second semantic distribution. By adding constraints, the encoder can learn more accurate information, thereby improving the encoder's output. Specifically, the labels corresponding to the positive samples and the positive samples themselves can be merged and input into the second encoder to obtain the second distribution parameters. Alternatively, the original text and the labels corresponding to the positive samples can be used as constraints on the second semantic distribution. Specifically, the original text, the labels corresponding to the positive samples, and the positive samples themselves can be merged and input into the second encoder to obtain the second distribution parameters. The merging can include concatenation. For example, the labels corresponding to the positive samples and the positive samples themselves can be concatenated and input into the second encoder. Another example is that the original text, the labels corresponding to the positive samples, and the positive samples themselves can be concatenated and input into the second encoder.

[0049] Negative samples can be input into the second encoder to obtain the third distribution parameters. Alternatively, the labels corresponding to the negative samples can be used as conditions to constrain the third semantic distribution. By adding constraints, the encoder can learn more accurate information, thereby improving the encoder's output. Specifically, the labels corresponding to the negative samples and the negative samples can be merged and input into the second encoder to obtain the third distribution parameters. Alternatively, the original text and the labels corresponding to the negative samples can be used as conditions to constrain the third semantic distribution. Specifically, the original text, the labels corresponding to the negative samples, and the negative samples can be merged and input into the second encoder to obtain the third distribution parameters. The merging can include concatenation. For example, the labels corresponding to the negative samples and the negative samples can be concatenated and input into the second encoder. Another example is that the original text, the labels corresponding to the negative samples, and the negative samples can be concatenated and input into the second encoder.

[0050] In some embodiments, the first distribution parameters, second distribution parameters, and third distribution parameters can be obtained solely through the encoder (i.e., the first encoder) in the text generation model. Specifically, the original text can be input into the encoder to obtain the first distribution parameters; positive samples can be input into the encoder to obtain the second distribution parameters; and negative samples can be input into the encoder to obtain the third distribution parameters. In some scenario examples, the text generation model can be a variational autoencoder.

[0051] Step S15: Optimize the encoder's model parameters using a loss function based on the first distribution parameter, the second distribution parameter, and the third distribution parameter. The loss function includes a first term, which constrains the similarity between the first semantic distribution and the second semantic distribution to be greater than the similarity between the first semantic distribution and the third semantic distribution.

[0052] In some embodiments, loss information can be calculated based on the loss function; the encoder's model parameters can be optimized based on the loss information. For example, the gradient of the model parameters can be calculated using the backpropagation mechanism; the model parameters can be adjusted based on the gradient.

[0053] In some embodiments, the loss function may include a first term. The first term is used to implement sentence-level contrastive learning. Specifically, the first term may be used to constrain the similarity between a first semantic distribution and a second semantic distribution to be greater than the similarity between the first semantic distribution and a third semantic distribution. Through the first term, the similarity between the first semantic distribution and the second semantic distribution can be increased, while the similarity between the first semantic distribution and the third semantic distribution can be decreased. Accordingly, the loss information may include the value of the first term. The value of the first term represents the sentence-level contrastive loss.

[0054] A first distance between a first semantic distribution and a second semantic distribution can be calculated based on the first and second distribution parameters; a second distance between the first semantic distribution and a third semantic distribution can be calculated based on the first and third distribution parameters; and the value of the first term can be calculated based on the first and second distances. The first and second distances are used to measure the similarity between semantic distributions. The first and second distances can include KL (Kullback-Leibler) divergence. Specifically, the smaller the KL divergence, the greater the similarity between semantic distributions. The larger the KL divergence, the smaller the similarity between semantic distributions. Of course, the first and second distances can also include JS (Jensen-Shannon) divergence, Maximum Mean Discrepancy, etc.

[0055] For example, the first term may include an InfoNoise Contrastive Estimationloss (InfoNCE loss) function. The first term can be expressed as a formula. y * y + Or y - y + This represents a positive sample. - Let represent a negative sample. Let x represent the original text. Let f(x) represent the first semantic distribution. Let f(y) represent the negative sample. + f(y) represents the second semantic distribution. - ) represents the third semantic distribution. h(f(y) + f(x)) represents the KL divergence between the first semantic distribution and the second semantic distribution. - Let f(x) represent the KL divergence between the first and third semantic distributions. τ is a hyperparameter representing the temperature coefficient. This formula is for illustrative purposes only; in practical applications, it can be modified or varied in other ways.

[0056] In some embodiments, keywords can influence the semantics of text. Even slight modifications to keywords in the text, such as changing "cosmology" to "astrophysics," can significantly alter the semantics of the text. Based on this, word-level contrastive learning can be introduced on top of sentence-level contrastive learning to further improve training effectiveness. Specifically, original keywords, positive sample keywords, and negative sample keywords can be determined based on the original text, positive samples, and negative samples, respectively. The encoder's model parameters can be optimized using a loss function based on these keywords. The loss function may also include a second term. This second term is used to implement word-level contrastive learning. Specifically, the second term constrains the similarity between the original keywords and positive sample keywords to be greater than the similarity between the original keywords and negative sample keywords. Through this second term, the similarity between the original keywords and positive sample keywords can be increased, while the similarity between the original keywords and negative sample keywords can be decreased. Correspondingly, the loss information may also include the value of the second term. The value of the second term represents the word-level contrastive loss.

[0057] At least one keyword can be extracted from the original text; this extracted keyword can be used as the original keyword. At least one keyword can be extracted from positive samples; this extracted keyword can be used as the positive sample keyword. At least one keyword can be extracted from negative samples; this extracted keyword can be used as the negative sample keyword. Alternatively, negative sample keywords can be determined through forgery. Specifically, at least one keyword can be extracted from negative samples as a reference keyword; a keyword network can be constructed based on the reference keyword and the positive sample keyword; one or more reference keywords adjacent to the positive sample keyword can be selected from the keyword network; and keywords can be forged as negative sample keywords based on the selected reference keywords. This allows for the construction of more difficult-to-identify negative sample keywords using adjacent reference keywords, which is beneficial for comparative learning.

[0058] A keyword network can be constructed using reference keywords and positive sample keywords as nodes. The keyword network can include nodes and edges. Nodes represent keywords. Edges represent the relationships between keywords. The relationships can be determined based on the samples to which the keywords belong. Specifically, if two keywords belong to two samples corresponding to the same original text, then the two keywords can have a relationship. The two nodes corresponding to the two keywords can be connected by edges. Further, the relationships can include positive and negative relationships. The edges can include positive and negative edges. Positive edges represent positive relationships. Negative edges represent negative relationships. Specifically, if two keywords are of the same type, then the two keywords can have a positive relationship. The two nodes corresponding to the two keywords can be connected by positive edges. If two keywords are of different types, then the two keywords can have a negative relationship. The two nodes corresponding to the two keywords can be connected by negative edges. The type of keyword can be determined by the samples to which the keyword belongs. Specifically, if the sample to which the keyword belongs is a positive sample, then the keyword can be a positive sample keyword. If the sample to which the keyword belongs is a negative sample, then the keyword can be a reference keyword.

[0059] For each positive sample keyword, a reference keyword with a proximity of less than or equal to K degrees can be selected from the keyword network. Based on the selected reference keyword, a fake keyword can be used as a negative sample keyword. The proximity can be represented by a degree. The degree can include the number of edges in the shortest path. For example, if the shortest path between two keywords contains K edges, the proximity between the two keywords can be considered to be K degrees. The value of K can be flexibly set according to actual needs, for example, it can be 1, 2, 4, 6, etc.

[0060] In the process of forging negative sample keywords, the embedding representation of the negative sample keywords can be calculated based on the embedding representation of the reference keywords. The embedding representation can include a vector. For example, it can be calculated using the formula... Calculate the embedding representation of the negative sample keywords. P represents the number of reference keywords. i This represents the embedding representation of the i-th reference keyword. For example, it can also be represented using the formula... Calculate the embedding representation of negative sample keywords. The W... i Used to represent weights. The W i It can be an empirical value, or it can be determined through machine learning.

[0061] A third distance can be calculated between the original keywords and the positive sample keywords based on their embedding representations. A fourth distance can be calculated between the original keywords and the negative sample keywords based on their embedding representations. The value of the second term can be calculated based on the third and fourth distances. The third and fourth distances are used to measure the semantic similarity between keywords. The third and fourth distances can include cosine similarity. A smaller cosine similarity indicates lower semantic similarity between keywords, while a larger cosine similarity indicates higher semantic similarity. The third and fourth distances can also include Hamming distance, minimum edit distance, etc.

[0062] For example, the second term could include a contrastive loss (InfoNCE loss) function. The second term can be expressed as a formula. u * For u out Or u imp u in Indicates the original keyword. out Indicates keywords for positive samples. imp Represents negative sample keywords. h(u in ,u out h(u) represents the cosine similarity between the original keywords and the keywords of the positive samples. in ,u imp The formula represents the cosine similarity between the original keyword and the negative sample keyword. This formula is for illustrative purposes only; in practical applications, it can have other variations or transformations.

[0063] It is worth noting that one-hot encoding algorithms or word2vec algorithms can be used to determine the keyword embedding representation. Alternatively, other methods can be used to determine the keyword embedding representation. The keyword embedding representation can include at least one of the following: the embedding representation of the original keyword, the embedding representation of the positive sample keyword, and the embedding representation of the negative sample keyword. For example, the keyword embedding representation can be determined based on the keyword network using a semantic representation model. The semantic representation model can include a semantic representation module, a multilayer perceptron module, and a graph attention (GAT) module. The semantic representation module can include a BERT (Bidirectional Encoder Representations from Transformers) model, an ERNIE (Enhanced Representation Knowledge Integration) model, etc. The semantic representation module is used to output the embedding representation of nodes in the keyword network. The multilayer perceptron module is used to determine the embedding representation of edges based on the embedding representation of nodes. For example, the embedding representation of the i-th node can be... The embedding representation of the j-th node can be The embedding representation of the edge between the i-th node and the j-th node can be... [] indicates concatenation. FFN represents the function corresponding to the multilayer perceptron. The graph attention module and the multilayer perceptron are combined to iteratively update the embedding representations of nodes and edges. In each iteration, the formula can be used... Update the edge embedding representation; it can be done according to the formula. Update the embedded representation of the node. Wherein, This represents the edge-upped embedding. This represents the embedding representation before the edge is updated. * represents either i or j. This represents the embedded representation after the node is updated. W q W k W r W v For model parameters, express and Attention weights between them.

[0064] In some embodiments, cross-level contrastive learning can be introduced in addition to sentence-level and word-level contrastive learning to further improve training effectiveness. The loss function may also include a third term. This third term is used to implement cross-level contrastive learning. Specifically, the third term constrains the distance between the first semantic distribution and positive sample keywords to be less than the distance between the first semantic distribution and negative sample keywords. Through this third term, the distance between the first semantic distribution and positive sample keywords can be reduced, while the distance between the first semantic distribution and negative sample keywords can be increased. Correspondingly, the loss information may also include the value of the third term. The value of the third term represents the cross-level contrastive loss.

[0065] The fifth distance between the first semantic distribution and the positive sample keywords can be calculated based on the first distribution parameters and the embedding representation of the positive sample keywords; the sixth distance between the first semantic distribution and the negative sample keywords can be calculated based on the first distribution parameters and the embedding representation of the negative sample keywords; and the loss of the third term can be calculated based on the fifth and sixth distances. The fifth and sixth distances may include Mahalanobis distance, Euclidean distance, etc.

[0066] For example, the third term may include an InfoNoise Contrastive Estimationloss (InfoNCE loss) function. The third term can be expressed as a formula. u * For u out Or u imp u out Indicates keywords for positive samples. imp Represents negative sample keywords. h(f(x),u out ) represents the Mahalanobis distance between the first semantic distribution and the positive sample keywords. imp The formula denoted by represents the Mahalanobis distance between the first semantic distribution and the negative sample keywords. This formula is merely an example; in practical applications, it can have other variations or transformations.

[0067] In some scenario examples, the loss function may include a first term, a second term, and a third term. Specifically, the loss function can be obtained by adding the first, second, and third terms together. The loss information may include the value of the first term, the value of the second term, and the value of the third term.

[0068] In some embodiments, the model parameters of the decoder in the text generation model can also be optimized. Specifically, sampling can be performed in a first semantic distribution to obtain latent variables; these latent variables can be input into the decoder; and the model parameters of the decoder can be optimized using a loss function based on the decoder's output and positive samples. Specifically, the loss function may further include a fourth term, which constrains the degree of difference between the decoder's output and positive samples. The fourth term may include a cross-entropy loss function, a maximum likelihood loss function (MLE), etc. Correspondingly, the loss information may also include the value of the fourth term.

[0069] The latent variables can be numerical values ​​or vectors. They can be obtained by random sampling within the first semantic distribution. Alternatively, other methods can be used to sample within the first semantic distribution. For example, the latent variables can be latent vectors. The first semantic distribution can be a normal distribution. The first distribution parameters can include the mean μ and variance σ. A reference vector ε can be obtained by sampling within a standard normal distribution. The latent vector can be calculated using the formula z = μ + σ*ε.

[0070] In some scenario examples, the loss function may include a first term, a second term, a third term, and a fourth term. Specifically, the loss function can be obtained by adding the first, second, third, and fourth terms together. The loss information may include the value of the first term, the value of the second term, the value of the third term, and the value of the fourth term.

[0071] The model training method described in this specification can improve training effectiveness by employing comparative learning.

[0072] This specification also provides a text processing method. The text processing method can be applied to computer devices. These computer devices include, but are not limited to, personal computers, servers, and server clusters containing multiple servers.

[0073] Please see Figure 3 and Figure 4 The text processing method may include the following steps.

[0074] Step S21: Input the original text into the encoder to obtain the distribution parameters.

[0075] Step S23: Sample from the semantic distribution to obtain latent variables.

[0076] Step S25: Input the hidden variables into the decoder to obtain the rewritten text.

[0077] In some embodiments, a rewritten text corresponding to the original text can be generated based on a text generation model. The rewritten text and the original text have the same semantics. Compared to the original text, the rewritten text has relatively accurate semantics. For example, the original text could be "Zhengda First Affiliated Hospital". The rewritten text could be "The First Affiliated Hospital of Zhengzhou University". Another example is that the original text could be "Newly Listed Car Inquiry". The rewritten text could be "New Car Listing Inquiry".

[0078] The text generation model may include an encoder and a decoder. The text generation model can be trained using the model training method described in the foregoing embodiments. Specifically, the encoder can be trained using the model training method described in the foregoing embodiments. The decoder can be trained using the model training method described in the foregoing embodiments, or it can be trained using other methods. Steps S21, S23, and S25 can be explained with reference to the foregoing embodiments and will not be repeated here.

[0079] In some embodiments, the text processing method can be applied to search scenarios. Specifically, it can receive user-input search text as raw text; input the raw text into an encoder to obtain distribution parameters; sample the semantic distribution to obtain latent variables; input the latent variables into a decoder to obtain rewritten text; perform a search based on the rewritten text; and provide search results to the user. This can improve the quality of search results, thereby enhancing the user experience.

[0080] Alternatively, the text processing method can also be applied to interactive question-and-answer scenarios. Specifically, it can receive user-inputted questions as raw text; input the raw text into an encoder to obtain distribution parameters; sample the semantic distribution to obtain latent variables; input the latent variables into a decoder to obtain rewritten text; obtain response information based on the rewritten text; and provide feedback to the user. This improves response quality and thus enhances the user experience.

[0081] The text processing method described in this specification can rewrite text.

[0082] Please see Figure 5 This specification also provides a model training apparatus, comprising the following units.

[0083] The acquisition unit 31 is used to acquire the original text, positive samples, and negative samples, wherein the positive samples and the negative samples are rewritten texts of the original text;

[0084] The determining unit 33 is used to determine, through the encoder, a first distribution parameter corresponding to the original text, a second distribution parameter corresponding to the positive sample, and a third distribution parameter corresponding to the negative sample. The first distribution parameter is used to limit the first semantic distribution, the second distribution parameter is used to limit the second semantic distribution, and the third distribution parameter is used to limit the third semantic distribution.

[0085] The optimization unit 35 is used to optimize the encoder's model parameters through a loss function based on the first distribution parameter, the second distribution parameter, and the third distribution parameter. The loss function includes a first term, which is used to constrain the similarity between the first semantic distribution and the second semantic distribution to be greater than the similarity between the first semantic distribution and the third semantic distribution.

[0086] Please see Figure 6 This specification also provides a text processing apparatus, including the following units.

[0087] The first input unit 41 is used to input the original text into the encoder to obtain distribution parameters. The encoder is trained according to the model training method of the aforementioned embodiment. The distribution parameters are used to limit the semantic distribution.

[0088] Sampling unit 43 is used to sample in the semantic distribution to obtain latent variables;

[0089] The second input unit 45 is used to input the hidden variables into the decoder to obtain the rewritten text.

[0090] The following describes an embodiment of the computer device described in this manual. Figure 7 This is a schematic diagram of the hardware structure of the computer device in this embodiment. For example... Figure 7 As shown, the computer device may include one or more (only one is shown in the figure) processors, memory, and transmission modules. Of course, those skilled in the art will understand that... Figure 7 The hardware structure shown is for illustrative purposes only and does not limit the hardware structure of the computer device described above. In practice, the computer device may also include more... Figure 7 Showing more or fewer component units; or having the same as Figure 7 The different configurations shown.

[0091] The memory may include high-speed random access memory; or it may include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. Of course, the memory may also include remotely accessible network memory. The memory can be used to store program instructions or modules of application software, such as those described in this specification. Figure 1 or Figure 3 The program instructions or modules corresponding to the embodiments.

[0092] The processor can be implemented in any suitable manner. For example, the processor can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc. The processor can read and execute program instructions or modules in the memory.

[0093] The transmission module can be used to transmit data via a network, such as the Internet, corporate intranet, local area network, or mobile communication network.

[0094] This specification also provides an embodiment of a computer storage medium. The computer storage medium includes, but is not limited to, Random Access Memory (RAM), Read-Only Memory (ROM), cache, hard disk drive (HDD), memory card, etc. The computer storage medium stores computer program instructions. When the computer program instructions are executed, they implement: this specification. Figure 1 or Figure 3 The program instructions or modules corresponding to the embodiments.

[0095] It should be noted that the various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on its differences from other embodiments. In particular, the device embodiments, computer device embodiments, and computer storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments. Furthermore, it is understood that those skilled in the art, after reading this specification, can arbitrarily combine some or all of the embodiments listed in this specification without creative effort, and such combinations are also within the scope of disclosure and protection of this specification.

[0096] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0097] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0098] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this specification can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of this specification, in essence or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments of this specification.

[0099] This manual can be used in a wide range of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices.

[0100] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0101] Although this specification has been described by way of examples, those skilled in the art will recognize that many variations and modifications are possible without departing from the spirit of this specification, and it is intended that the appended claims cover such variations and modifications without departing from the spirit of this specification.

Claims

1. A model training method, comprising: Obtain the original text, positive samples, and negative samples, wherein the positive samples and negative samples are rewritten versions of the original text; The encoder determines the first distribution parameter corresponding to the original text, the second distribution parameter corresponding to the positive sample, and the third distribution parameter corresponding to the negative sample. The first distribution parameter is used to limit the first semantic distribution, the second distribution parameter is used to limit the second semantic distribution, and the third distribution parameter is used to limit the third semantic distribution. Based on the first distribution parameter, the second distribution parameter, and the third distribution parameter, the encoder model parameters are optimized using a loss function. The loss function includes a first term, which constrains the similarity between the first semantic distribution and the second semantic distribution to be greater than the similarity between the first semantic distribution and the third semantic distribution. Optimizing the encoder model parameters using the loss function includes optimizing the encoder model parameters based on the original keywords, positive sample keywords, and negative sample keywords. The loss function also includes a second term, which constrains the similarity between the original keywords and the positive sample keywords to be greater than the similarity between the original keywords and the negative sample keywords. The original keywords, the positive sample keywords, and the negative sample keywords are determined based on the original text, positive samples, and negative samples, respectively.

2. The method according to claim 1, wherein the encoder comprises a first encoder and a second encoder, the first encoder being an encoder in a text generation model, and the second encoder being used to assist in the training of the text generation model; The steps for obtaining the first distribution parameter, the second distribution parameter, and the third distribution parameter include: The first encoder is used to obtain the first distribution parameters corresponding to the original text; The second encoder is used to obtain the second distribution parameters corresponding to positive samples and the third distribution parameters corresponding to negative samples.

3. The method according to claim 2, wherein the step of obtaining the first distribution parameter includes: The keywords in the original text are combined with the original text and then input into the first encoder to obtain the first distribution parameters; The steps for obtaining the second and third distribution parameters include: The original text, the labels corresponding to the positive samples, and the positive samples are merged and then input into the second encoder to obtain the second distribution parameters. The original text, the labels corresponding to the negative samples, and the negative samples are merged and input into the second encoder to obtain the third distribution parameters.

4. The method according to claim 1, wherein the step of determining negative sample keywords includes: Extract keywords from negative samples as reference keywords; Construct a keyword network based on reference keywords and positive sample keywords; Select one or more reference keywords that are adjacent to the positive sample keywords from the keyword network; Construct negative sample keywords based on one or more selected reference keywords.

5. The method according to claim 1, wherein the loss function further includes a third term, the third term being used to constrain the distance between the first semantic distribution and the positive sample keywords to be less than the distance between the first semantic distribution and the negative sample keywords.

6. The method according to claim 1, wherein the first distribution parameter, the second distribution parameter, and the third distribution parameter respectively include the mean and the variance, and the first semantic distribution, the second semantic distribution, and the third semantic distribution are normal distributions.

7. The method according to claim 1, further comprising: Sampling is performed in the first semantic distribution to obtain latent variables; Input the hidden variables into the decoder; Based on the decoder's output and positive samples, the decoder's model parameters are optimized using a loss function, which also includes a fourth term used to constrain the degree of difference between the decoder's output and the positive samples.

8. A text processing method, comprising: The original text is input into the encoder to obtain distribution parameters, wherein the encoder is trained according to any one of the methods in claims 1-6, and the distribution parameters are used to define the semantic distribution; Sampling is performed in the semantic distribution to obtain latent variables; The hidden variables are input into the decoder to obtain the rewritten text.

9. The method according to claim 8, wherein the decoder is trained according to the method according to claim 7.

10. A model training device, comprising: The acquisition unit is used to acquire the original text, positive samples, and negative samples, wherein the positive samples and negative samples are rewritten texts of the original text; The determining unit is used to determine, through the encoder, a first distribution parameter corresponding to the original text, a second distribution parameter corresponding to the positive sample, and a third distribution parameter corresponding to the negative sample. The first distribution parameter is used to limit the first semantic distribution, the second distribution parameter is used to limit the second semantic distribution, and the third distribution parameter is used to limit the third semantic distribution. An optimization unit is used to optimize the encoder's model parameters using a loss function based on a first distribution parameter, a second distribution parameter, and a third distribution parameter. The loss function includes a first term, which constrains the similarity between the first semantic distribution and the second semantic distribution to be greater than the similarity between the first semantic distribution and the third semantic distribution. Optimizing the encoder's model parameters using the loss function includes optimizing the encoder's model parameters based on the original keywords, positive sample keywords, and negative sample keywords. The loss function also includes a second term, which constrains the similarity between the original keywords and the positive sample keywords to be greater than the similarity between the original keywords and the negative sample keywords. The original keywords, the positive sample keywords, and the negative sample keywords are determined based on the original text, positive samples, and negative samples, respectively.

11. A text processing apparatus, comprising: The first input unit is used to input the original text into the encoder to obtain distribution parameters, wherein the encoder is trained according to any one of the methods in claims 1-6, and the distribution parameters are used to define the semantic distribution. A sampling unit is used to sample from a semantic distribution to obtain latent variables; The second input unit is used to input the latent variables into the decoder to obtain the rewritten text.

12. A computer device, comprising: At least one processor; A memory storing program instructions configured to be executed by the at least one processor, the program instructions including instructions for performing the method according to any one of claims 1-9.

Images