Text processing model training method and text processing method

Abstract

The invention provides a text processing model training method and a text processing method.The text processing model training method comprises the steps that noise is added to training sample content based on a target time step to obtain global noise adding content, and noise adding posterior distribution corresponding to a reference time step is calculated; inputting the global noise adding content into a text processing model, and obtaining prediction content output by the text processing model and noise reduction posteriori distribution corresponding to the reference time step; gradient compensation information is determined according to the noise adding posteriori distribution and the noise reduction posteriori distribution; obtaining a model target loss function according to the gradient compensation information, and calculating a model loss value based on the model target loss function; and adjusting model parameters of the text processing model according to the model loss value, and continuing to train the text processing model. The updated model target loss function balances the difference between gradients, so that the problem of insufficient precision caused by too small time step interval can be avoided, and the problem of gradient mismatching caused by continuous time step approximation is solved.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and in particular to methods for training text processing models. This specification also relates to text processing methods, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] With the development of technology, large language models have been widely used in various scenarios. Autoregressive models are the mainstream paradigm for large language models. However, the inherent decoding method of autoregressive models hinders their efficiency in parallel inference. Therefore, discrete diffusion large language models have been further developed.

[0003] The uniform diffusion model in discrete diffusion large language models has higher sampling efficiency, editable sampling methods, and a more controllable modeling approach; however, its modeling accuracy is weaker compared to the masked diffusion model. Therefore, improving the inference accuracy of the uniform diffusion model has become a pressing technical problem for engineers. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a method for training a text processing model. This specification also relates to a text processing method, a computing device, a computer-readable storage medium, and a computer program product, to address the aforementioned problems in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a method for training a text processing model is provided, comprising: Obtain training sample content, add noise to the training sample content based on the target time step to obtain global noisy content, and calculate the noisy posterior distribution corresponding to the reference time step based on the global noisy content and the training sample content. The globally denoised content is input into the text processing model to obtain the predicted content of the output of the text processing model, and the denoised posterior distribution corresponding to the reference time step in the process of generating the predicted content. The gradient compensation information corresponding to the reference time step is determined based on the noise-added posterior distribution and the noise-reduced posterior distribution, wherein the gradient compensation information characterizes the gradient difference information of the noise distribution between the reference time step and the target time step; The initial loss function is updated based on the gradient compensation information to obtain the model target loss function, and the model loss value is calculated based on the model target loss function. The model parameters of the text processing model are adjusted based on the model loss value, and the text processing model continues to be trained until the model training stops.

[0006] According to a second aspect of the embodiments of this specification, a text processing method is provided, comprising: Receive the text content to be processed; The text content to be processed is input into the text processing model to obtain the text prediction content output by the text processing model, wherein the text processing model is trained by the above-mentioned text processing model training method.

[0007] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0008] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0009] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0010] The training method for the text processing model provided in this specification includes: acquiring training sample content; adding noise to the training sample content based on a target time step to obtain globally noisy content; calculating the posterior distribution of the noisy content corresponding to a reference time step based on the globally noisy content and the training sample content; inputting the globally noisy content into the text processing model to obtain the predicted content output by the text processing model, and the posterior distribution of the noise reduction corresponding to the reference time step during the generation of the predicted content by the text processing model; determining the gradient compensation information corresponding to the reference time step based on the posterior distribution of the noisy content and the posterior distribution of the noise reduction, wherein the gradient compensation information characterizes the gradient difference information of the noise distribution between the reference time step and the target time step; updating the initial loss function based on the gradient compensation information to obtain the model target loss function, and calculating the model loss value based on the model target loss function; adjusting the model parameters of the text processing model based on the model loss value, and continuing to train the text processing model until the model training stopping condition is met.

[0011] The method provided in the embodiments of this specification updates the initial loss function based on gradient compensation information during the model training phase of the text processing model to obtain the model's target loss function. Finally, the model loss value is calculated based on the target loss function, and the model parameters of the text processing model are adjusted. The updated target loss function balances the differences between gradients, improving computational accuracy when the time step interval is short. This allows the target loss function to adapt to time intervals of any size, avoiding the accuracy problem caused by excessively small time step intervals and resolving the gradient mismatch problem caused by approximate continuous time steps. Attached Figure Description

[0012] Figure 1 This is a flowchart of a training method for a text processing model provided in one embodiment of this specification; Figure 2 This is a flowchart of a text processing method provided in one embodiment of this specification; Figure 3 This is a schematic diagram of the structure of a training device for a text processing model provided in one embodiment of this specification; Figure 4 This is a schematic diagram of the structure of a text processing device provided in one embodiment of this specification; Figure 5 This is an architecture diagram of a training system for a text processing model provided in one embodiment of this specification; Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0013] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0014] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0015] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0016] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0017] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0018] Kullback-Leibler divergence is an asymmetric measure in information theory used to measure the difference between two probability distributions. It has wide applications in machine learning, statistical inference, generative models, and other fields.

[0019] LLM (Large Language Model): A type of natural language processing model based on deep learning, capable of understanding, generating, and reasoning about human language. Large language models refer to deep neural network models with a large number of parameters, pre-trained on large amounts of text data, possessing powerful language understanding and generation capabilities. They can predict the next word (token) based on context, thus performing various tasks such as question answering, collaboration, translation, and programming.

[0020] Autoregressive (AR): Autoregression is a modeling approach that uses the historical values ​​of a variable to predict its current value. In deep learning, autoregression involves generating a sequence of elements one by one, with each step relying on previously generated elements.

[0021] dLLM (Diffusion Large Language Model): Diffusion models initially achieved significant success in image generation. Their core idea is to progressively add noise to an image until it becomes completely random noise (forward process), then train a model to learn how to remove noise step-by-step from the random noise, ultimately restoring a clear image (backward process). dLLM applies the ideas of image diffusion models to text generation. It includes a forward process, specifically adding noise to a clear and meaningful text to create random, meaningless text, thus obtaining text noise. Then, the dLLM learns how to remove noise step-by-step from the text noise, ultimately obtaining a fluent, coherent, and meaningful text.

[0022] MDLM (Masked Diffusion Language Model): Traditional image diffusion models corrupt images by adding continuous Gaussian noise. Since Gaussian noise cannot be added to text, the masked diffusion language model randomly selects a certain proportion of tokens in the text and replaces them with masks. After multiple steps, the entire sentence becomes a mask sequence. In the reverse process, based on the currently partially masked text, it predicts the tokens at all mask positions.

[0023] UDLM (Uniform Diffusion Language Model): The uniform diffusion language model mixes text with a uniform distribution. After a sufficient number of steps, all tokens in the text become uniformly distributed information. In the denoising process, through multiple iterations, the model progressively processes the uniformly distributed information at each position into the corresponding token.

[0024] In recent years, large language models and multimodal large language models have made significant progress. Their capabilities have been well-proven in the traditional realm of intelligence. Today's large and multimodal large language models consistently achieve high scores on standard benchmark tests measuring general knowledge, comprehension, and reasoning abilities, indicating that they are no longer simply text completion engines, but rather competent general-purpose intelligent agents.

[0025] Currently, autoregressive models have become the mainstream paradigm for large language models and multimodal large language models. However, despite their remarkable achievements, the inherent limitations of autoregressive models cannot be ignored. Specifically, their front-to-back decoding approach hinders parallel reasoning and reduces the efficiency of reasoning generation. Furthermore, they struggle to achieve precise structural control (such as length or format), resulting in low efficiency when using natural language in tasks requiring sophisticated orchestration tools and agent-based tasks. In addition, the causal attention mechanism forces the model to perform a one-time static perception of the input, limiting the model's ability to achieve dynamic task perception and understanding without requiring complex chained thinking or multi-round processing.

[0026] Therefore, discrete large language models have become a current research focus for engineers. In tasks such as code generation, planning, and Sudoku, discrete large language models can achieve better performance than autoregressive models. However, although diffusion models are widely used in image generation and signal processing, applying them to tasks involving discrete data (such as text modeling) presents some challenges.

[0027] First, standard diffusion models are practically unusable in terms of language. Second, the popular Masked Diffusion Model (MDLM) cannot edit the generated tokens in multiple rounds, and its token generation efficiency remains low; many current implementations still generate only one or two tokens per sample for higher modeling accuracy. On the other hand, the Uniform Diffusion Model (UDLM) has higher sampling efficiency, an editable sampling method, and a more controllable modeling approach, but it still lags behind MDLM in modeling accuracy.

[0028] Based on this, a method for training a text processing model is provided in this specification. This specification also relates to a text processing method, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail in the following embodiments.

[0029] Figure 1 A flowchart illustrating a training method for a text processing model according to an embodiment of this specification is shown, specifically including the following steps: Step 102: Obtain training sample content, add noise to the training sample content based on the target time step to obtain global noisy content, and calculate the noisy posterior distribution corresponding to the reference time step based on the global noisy content and the training sample content.

[0030] The text processing model provided in the embodiments of this specification is built on the Uniform Diffusion Language Model (UDLM) framework. During the research of the UDLM model, it was found that its accuracy is lower than that of the Masked Diffusion Language Model (MDLM). One reason for the lower accuracy of UDLM compared to MDLM is the inconsistency between the training and inference time intervals of the UDLM model; that is, the inconsistency between the training and inference time intervals is one of the reasons for the lower accuracy. To address this issue, the loss function of UDLM is adjusted in the method provided in the embodiments of this specification, thereby mitigating the problem caused by the mismatch between training and inference.

[0031] The training sample content can be understood as the training sample used to train the text processing model. In the method provided in the embodiments of this specification, the text processing model is trained. Therefore, the training sample content includes at least text information. Specifically, it can be a plain text sample, or it can be a multimodal information containing text information, such as image and text samples, audio text samples, or video text samples.

[0032] The text processing model in the method provided in the embodiments of this specification is a diffusion-based large language model. The idea of ​​the diffusion model is to denoise an image with added noise until the original clear image is restored. The diffusion-based large language model applies the idea of ​​the diffusion model to the field of text generation. Therefore, for the training sample data, a noise-adding process is first performed, and then the text processing model is used to denoise and restore the image.

[0033] Therefore, having obtained the training sample content, it is necessary to first perform forward noise addition on the training sample content. Specifically, the forward noise addition process can be represented by the following formula 1: Formula 1 in, Represents a discrete distribution. Indicates the time step from time step 0 to time step 1. t The state transition matrix, when representing the start from time step 0, can be simplified as follows: , This is the initial state, i.e., the content of the training samples (original real text). For time step t The text content after noise addition (global noise-added content).

[0034] In one specific embodiment provided in this specification, noise is added to the training sample content based on a target time step to obtain globally noisy content, including: Determine the first noise matrix and the second noise matrix; The target noise information corresponding to the target time step is determined based on the first noise matrix and the second noise matrix; Global noise-added content is generated based on the target noise-adding information and the training sample content.

[0035] In another specific embodiment provided in this specification, determining the target noise information corresponding to the target time step based on the first noise matrix and the second noise matrix includes: Determine the noise weighting information corresponding to the target time step; The target noise information corresponding to the target time step is calculated based on the noise weight information, the first noise matrix, and the second noise matrix.

[0036] In practical applications, the process of adding noise to the training sample content based on the target time step can be understood as moving from time step 0 to time step 1. t The process of adding the state transition matrix. Specifically, the state transition matrix can be calculated using the following formula 2: Formula 2 in, This represents the state transition matrix from time step s to time step t, which can be understood as the target noise information. The number of tokens in the vocabulary of a diffusion-based large language model. The matrix that represents the diagonal elements as 1s and the rest as 0s (i.e., the first noisy matrix). The representation is a matrix where all elements are 1 (i.e., the second noisy matrix). This is a scalar coefficient, with a value between 0 and 1, representing the degree of noise state transition. The closer the value is to 1, the lower the degree of state transition. This can be understood as noise-adding weighting information. In practical applications, for ease of representation, it can be denoted as... .

[0037] In another specific embodiment provided in this specification, the posterior distribution of the reference time step is calculated based on the global noise content and the training sample content, including: Determine the reference noise weight information corresponding to the reference time step; The posterior distribution of the reference time step is calculated based on the training sample content, the global noise content, the reference noise weight information, the first noise matrix, and the second noise matrix.

[0038] Based on Bayes' theorem and the Markov assumption of UDLM, the posterior distribution of the forward noise-adding process can be derived in the method provided in the embodiments of this specification. That is, based on the given target time step... t state of time Compared with the initial state Expected target time step t State probability distribution at time 1 ,in, , where e is a basis vector with the same dimension as x.

[0039] Determine the target time step t Given the state probability distribution at time step s, the noisy posterior distribution corresponding to the reference time step s is predicted, as detailed in Formula 3 below: Formula 3 in, Characterizing the reference time step s The corresponding noisy posterior distribution, For the target time step t Globally denoised content after noise addition This is the content of the training samples. Characterizing the reference time step s The corresponding state probability distribution, Characterization from reference time step s to the target time step t The degree of noise state transition, i.e., the reference noise weighting information.

[0040] Step 104: Input the globally noisy content into the text processing model to obtain the predicted content of the output of the text processing model, and the denoised posterior distribution corresponding to the reference time step in the process of generating the predicted content.

[0041] In the method provided in the embodiments of this specification, a text processing model is constructed. This model progressively denoises the input globally noisy content to obtain the predicted content output by the text processing model. Simultaneously, it can also obtain the denoised posterior distribution corresponding to the reference time step during the generation of the predicted content.

[0042] Specifically, refer to Formula 4 below to calculate the denoising posterior distribution.

[0043] Formula 4 in, Characterizing the reference time step s The corresponding denoised posterior distribution, Characterizes the predicted content output by the text processing model.

[0044] It should be noted that in the methods provided in this specification, the reference time step is from the start of noise addition at time step 0 to the end of noise addition at time step t. The reference time step s in the noise addition process is the same as the reference time step s in the noise reduction process. That is, the noise addition process is (0, 1, 2…s,…t-1, t), and the noise reduction process is (t, t-1,…2, 1, 0).

[0045] Step 106: Determine the gradient compensation information corresponding to the reference time step based on the noise-added posterior distribution and the noise-reduced posterior distribution, wherein the gradient compensation information characterizes the gradient difference information of the noise distribution between the reference time step and the target time step.

[0046] After the above steps, the posterior distributions of noise addition and denoising can be calculated in the text processing model. Based on the posterior distributions of noise addition and denoising corresponding to the reference time step s, the gradient compensation information at the reference time step s can be calculated. In the method provided in the embodiments of this specification, the gradient compensation information is used to characterize the gradient difference information of the noise distribution between the reference time step and the target time step.

[0047] During the training of the UDLM model, the KL divergence is usually calculated based on the denoised posterior distribution and the denoised posterior distribution, and the model loss value is calculated based on the KL divergence and the initial loss function.

[0048] KL divergence is a core metric for measuring the difference between two probability distributions, also known as relative entropy. In the loss function of model training, KL divergence is typically used to calculate the model loss value. Its core principle is to compare the true distribution of the diffusion process with the distribution predicted by the model, adjusting the model's ability to predict noise by reducing the KL divergence.

[0049] Specifically, the KL divergence can be calculated using the following formula 5: Formula 5 in, Characterizing the noisy posterior distribution at the reference time step s, Characterizes the denoised posterior distribution at the reference time step s.

[0050] Typically, the model loss function of a diffusion model is the Diffusion loss function. The core of the Diffusion loss function is to learn to predict the original noise from noisy samples. KL divergence is used to quantify the difference between the actual noise distribution and the noise distribution predicted by the model during the diffusion process, and finally transforms it into an adjustable loss term.

[0051] Based on this, when calculating the KL divergence according to Formula 5, the loss value can be calculated according to the initial loss function shown in Formula 6 below.

[0052] Formula 6 Formula 6 is the initial loss function during model training. When studying Formula 6, the technicians found that when the reference time step s is close to the target time step t, the first term will produce a division operation close to 0 / 0. This operation will lead to a large deviation when the calculation accuracy is not high.

[0053] In the current processing method, the formula 6 will be processed. The item will be further processed, that is, in We perform a first-order Taylor expansion. The first-order Taylor expansion is a way to approximate complex functions using linear functions, and its core purpose is to simplify calculation and analysis.

[0054] The first-order Taylor expansion is at some point (i.e.) of the function This describes an approximation method using linear functions to approximate variable functions. First-order Taylor expansions can simplify the calculation of complex functions, support the design of optimization algorithms, and reduce computational costs. Specifically, refer to Equation 7 below, and apply a first-order Taylor expansion to the above content according to Equation 7.

[0055] Formula 7 in, Characterization The term, after undergoing a first-order Taylor expansion, can be determined. and Differences between In the current processing method, because... It is so small that it will be ignored.

[0056] However, after research, technicians discovered that at this time... and Although numerically they are approximately aligned, their gradients are different. Specifically... and exist The gradient at that point is given by Equations 8 and 9 below.

[0057] Formula 8 Formula 9 in, represent exist gradient information at the location, represent exist The gradient information at that point. Through analysis of the initial loss function, we can obtain... and exist The gradient information differs at different points. The gradient is the core guiding signal for model training and parameter tuning, directly determining the direction and step size of parameter updates, ultimately affecting the model's convergence speed, training stability, and generalization performance. To ensure consistent time intervals between training and inference in UDLM, the gradient information can be... and exist The gradient information at each step is matched. This removes the assumption in the initial loss function that the time interval only approaches 0, allowing the model to train from any time step to the next time step (the time interval does not need to approach 0) and thus effectively alleviates the accuracy loss problem caused by the mismatch between training and inference.

[0058] In a specific embodiment provided in this specification, determining the gradient compensation information corresponding to the reference time step based on the noisy posterior distribution and the noisy posterior distribution includes: The reference gradient compensation information is determined based on the denoised posterior distribution and the predicted denoised posterior distribution corresponding to the predicted content. The gradient compensation information corresponding to the reference time step is determined based on the posterior distribution of the added noise, the global posterior distribution corresponding to the global added noise content, and the reference gradient compensation information.

[0059] Specifically, by analyzing Formulas 8 and 9 above, gradient compensation information can be calculated, as detailed in Formula 10 below: Formula 10 in, This indicates gradient truncation (stop gradient). Gradient truncation is used here to prevent this part from being recalculated.

[0060] Step 108: Update the initial loss function according to the gradient compensation information, obtain the model target loss function, and calculate the model loss value based on the model target loss function.

[0061] Once the gradient compensation information is determined, the initial loss function can be updated based on this information to obtain the model's target loss function. The model loss value can then be calculated based on this target loss function.

[0062] Specifically, in one specific embodiment provided in this specification, updating the initial loss function based on the gradient compensation information includes: Obtain the initial loss function; Determine the posterior distribution normalization term in the initial loss function; The posterior distribution normalization term is updated based on the gradient compensation information.

[0063] In this implementation, the initial loss function of the model is determined first. Specifically, the initial loss function is described in Equation 6 above. The posterior distribution normalization term can be determined within the initial loss function. The normalization term of the posterior distribution is updated based on the gradient compensation information.

[0064] In another specific embodiment provided in the embodiments of this specification, the posterior distribution normalization term includes loss adjustment information; Updating the posterior distribution normalization term based on the gradient compensation information includes: The loss adjustment information is updated based on the gradient compensation information.

[0065] Furthermore, in the posterior distribution normalization term This includes loss adjustment information. In the method provided in the embodiments of this specification, the loss adjustment information is updated according to the gradient compensation information, and the updated model target loss function is obtained as shown in the following formula 11: Formula 11 Formula 11 above is the loss function used in the model training method of the text processing model provided in the embodiments of this specification.

[0066] In another specific embodiment provided in the embodiments of this specification, calculating the model loss value based on the model objective loss function includes: The objective loss function of the model is converted into a logarithmic loss function; The model loss value is calculated based on the logarithmic loss function.

[0067] In practical applications, calculating the model loss value based on the model's objective loss function involves a large number of matrix multiplication operations. In order to further reduce the accuracy loss during model training, the model's objective loss function can be converted into a logarithmic loss function, and the model loss value can be calculated based on the logarithmic loss function.

[0068] Specifically, the same exponent can be used to convert matrix multiplication into addition in the logarithmic field, thereby transforming complex matrix multiplication into logarithmic addition, resulting in a larger representation range and a finer floating-point representation step size, reducing accuracy loss during model training.

[0069] Step 110: Adjust the model parameters of the text processing model according to the model loss value, and continue training the text processing model until the model training stops.

[0070] After calculating the model loss value using the aforementioned objective loss function, the model parameters of the text processing model can be adjusted based on this loss value. The text processing model can then be trained further using the training sample content until the training stops.

[0071] In the methods provided in the embodiments of this specification, the model training stopping condition may include at least one of the following: the model loss value is less than a preset threshold, or the number of training rounds reaches a preset number of rounds.

[0072] For example, if the model loss value is less than a preset threshold, it means that the model training has reached the model training stop condition when the calculated model loss value is less than the preset threshold.

[0073] For example, if the model training has reached a preset number of rounds, which can be set to 10 rounds, then the model training can be considered to have reached the model training stopping condition when the model training has reached 10 rounds.

[0074] For example, the model can be considered to have reached the stopping condition if either the model loss value is less than a preset threshold or the number of training epochs reaches a preset number. Both the model loss value and the number of training epochs can be checked simultaneously. If either condition is met, the model training can be considered to have reached the stopping condition. Alternatively, both conditions can be met for the model training to be considered to have reached the stopping condition.

[0075] In the methods provided in the embodiments of this specification, the specific content of the model training stopping condition is not limited, and the actual application shall prevail.

[0076] The method provided in the embodiments of this specification updates the initial loss function based on gradient compensation information during the model training phase of the text processing model to obtain the model's target loss function. Finally, the model loss value is calculated based on the target loss function, and the model parameters of the text processing model are adjusted. The updated target loss function balances the differences between gradients, improving computational accuracy when the time step interval is short. This allows the target loss function to adapt to time intervals of any size, avoiding the accuracy problem caused by excessively small time step intervals and resolving the gradient mismatch problem caused by approximate continuous time steps.

[0077] In addition, during the calculation of the loss function, complex matrix multiplication operations are converted into logarithmic addition operations in the logarithmic field, reducing the accuracy loss during model training.

[0078] See Figure 2 , Figure 2 A flowchart illustrating a text processing method provided in one embodiment of this specification is shown, as follows: Figure 2 As shown, the method includes the following steps: Step 202: Receive the text content to be processed.

[0079] Step 204: Input the text content to be processed into the text processing model to obtain the text prediction content output by the text processing model, wherein the text processing model is trained by the above-mentioned text processing model training method.

[0080] The method provided in the embodiments of this specification provides a text processing method. The text processing method provided in this embodiment uses the text processing model trained by the above-mentioned text processing model training method, and the generated text prediction content has higher prediction accuracy.

[0081] Corresponding to the above method embodiments, this specification also provides embodiments of a training device for a text processing model. Figure 3 A schematic diagram of a training apparatus for a text processing model according to an embodiment of this specification is shown. Figure 3 As shown, the device includes: The acquisition module 302 is configured to acquire training sample content, add noise to the training sample content based on the target time step to obtain global noise-added content, and calculate the noise-added posterior distribution corresponding to the reference time step based on the global noise-added content and the training sample content. The prediction module 304 is configured to input the globally noisy content into the text processing model to obtain the predicted content output by the text processing model, and the denoised posterior distribution corresponding to the reference time step in the process of generating the predicted content by the text processing model. The determining module 306 is configured to determine the gradient compensation information corresponding to the reference time step based on the noise-added posterior distribution and the noise-reduced posterior distribution, wherein the gradient compensation information characterizes the gradient difference information of the noise distribution between the reference time step and the target time step; The calculation module 308 is configured to update the initial loss function according to the gradient compensation information, obtain the model target loss function, and calculate the model loss value based on the model target loss function; The training continuation module 310 is configured to adjust the model parameters of the text processing model based on the model loss value and continue training the text processing model until the model training stop condition is met.

[0082] In one specific embodiment provided in this specification, the acquisition module 302 is further configured as follows: Determine the first noise matrix and the second noise matrix; The target noise information corresponding to the target time step is determined based on the first noise matrix and the second noise matrix; Global noise-added content is generated based on the target noise-adding information and the training sample content.

[0083] In one specific embodiment provided in this specification, the acquisition module 302 is further configured as follows: Determine the noise weighting information corresponding to the target time step; The target noise information corresponding to the target time step is calculated based on the noise weight information, the first noise matrix, and the second noise matrix.

[0084] In one specific embodiment provided in this specification, the acquisition module 302 is further configured as follows: Determine the reference noise weight information corresponding to the reference time step; The posterior distribution of the reference time step is calculated based on the training sample content, the global noise content, the reference noise weight information, the first noise matrix, and the second noise matrix.

[0085] In one specific embodiment provided in this specification, the determining module 306 is further configured as follows: The reference gradient compensation information is determined based on the denoised posterior distribution and the predicted denoised posterior distribution corresponding to the predicted content. The gradient compensation information corresponding to the reference time step is determined based on the posterior distribution of the added noise, the global posterior distribution corresponding to the global added noise content, and the reference gradient compensation information.

[0086] In one specific embodiment provided in this specification, the computing module 308 is further configured as follows: Obtain the initial loss function; Determine the posterior distribution normalization term in the initial loss function; The posterior distribution normalization term is updated based on the gradient compensation information.

[0087] In one specific embodiment provided in the embodiments of this specification, the posterior distribution normalization term includes loss adjustment information; The computing module 308 is further configured as follows: The loss adjustment information is updated based on the gradient compensation information.

[0088] In one specific embodiment provided in this specification, the computing module 308 is further configured as follows: The objective loss function of the model is converted into a logarithmic loss function; The model loss value is calculated based on the logarithmic loss function.

[0089] The apparatus provided in the embodiments of this specification updates the initial loss function based on gradient compensation information during the model training phase of the text processing model to obtain the target loss function. Finally, the model loss value is calculated based on the target loss function, and the model parameters of the text processing model are adjusted. The updated target loss function balances the differences between gradients, improving computational accuracy when the time step interval is short. This allows the target loss function to adapt to time intervals of any size, avoiding the accuracy problem caused by excessively small time step intervals and solving the gradient mismatch problem caused by approximate continuous time steps.

[0090] In addition, during the calculation of the loss function, the complex matrix multiplication operation is transformed into a logarithmic addition operation in the logarithmic field, which improves the calculation efficiency of the loss function and reduces the training time of the model.

[0091] The above is a schematic scheme of a text processing model training device according to this embodiment. It should be noted that the technical solution of this text processing model training device and the technical solution of the text processing model training method described above belong to the same concept. For details not described in detail in the technical solution of the text processing model training device, please refer to the description of the technical solution of the text processing model training method described above.

[0092] Corresponding to the above method embodiments, this specification also provides embodiments of a text processing device. Figure 4 A schematic diagram of the structure of a text processing apparatus according to an embodiment of this specification is shown. Figure 4 As shown, the device includes: The receiving module 402 is configured to receive text content to be processed; The prediction module 404 is configured to input the text content to be processed into the text processing model and obtain the text prediction content output by the text processing model, wherein the text processing model is trained by the above-mentioned text processing model training method.

[0093] In the method provided in the embodiments of this specification, a text processing device is provided. The text processing device provided in this embodiment deploys the text processing model trained by the above-mentioned text processing model training method, and the generated text prediction content has higher prediction accuracy.

[0094] The above is an illustrative scheme of a text processing device according to this embodiment. It should be noted that the technical solution of this text processing device and the technical solution of the above-described text processing method belong to the same concept. For details not described in detail in the technical solution of the text processing device, please refer to the description of the technical solution of the above-described text processing method.

[0095] See Figure 5 , Figure 5 This specification shows an architecture diagram of a text processing model training system according to an embodiment of the present specification. The text processing model training system may include a client 100 and a server 200. Client 100 is used to send training sample content to server 200; Server 200 is configured to add noise to the training sample content based on a target time step to obtain globally denoised content; calculate the denoised posterior distribution corresponding to the reference time step based on the globally denoised content and the training sample content; input the globally denoised content into a text processing model to obtain the predicted content output by the text processing model, and the denoised posterior distribution corresponding to the reference time step during the generation of the predicted content by the text processing model; determine the gradient compensation information corresponding to the reference time step based on the denoised posterior distribution and the denoised posterior distribution, wherein the gradient compensation information characterizes the gradient difference information of the noise distribution between the reference time step and the target time step; update the initial loss function based on the gradient compensation information to obtain the model target loss function, and calculate the model loss value based on the model target loss function; adjust the model parameters of the text processing model based on the model loss value, and continue training the text processing model until the model training stop condition is reached, obtain the text processing model parameters of the text processing model, and send the text processing model parameters to client 100. Client 100 is also used to receive text processing model parameters sent by server 200 and deploy text processing models based on these parameters.

[0096] The text processing model training system may include multiple clients 100 and a server 200. Clients 100 can be referred to as edge devices, and server 200 can be referred to as cloud devices. Multiple clients 100 can establish communication connections through server 200. In the text processing model training scenario, server 200 is used to provide training services for the text processing model among multiple clients 100. Each client 100 can act as a sender or receiver, communicating through server 200.

[0097] Users can interact with server 200 through client 100 to receive data sent by other clients 100, or send data to other clients 100, etc. In the training scenario of the text processing model, users can publish data streams to server 200 through client 100, server 200 can generate text processing model parameters based on the data stream, and push the text processing model parameters to other clients that have established communication.

[0098] In this system, client 100 and server 200 establish a connection via a network. The network provides the medium for communication between client 100 and server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. Data transmitted by client 100 may need to undergo encoding, transcoding, compression, or other processing before being published to server 200.

[0099] Client 100 can be a browser, an app (application), a web application such as an H5 (HyperText Markup Language 5) application, a lightweight application (also known as a mini-program), or a cloud application. Client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by server 200, such as a real-time communication (RTC) SDK. Client 100 can be deployed on a computing device and depends on the device or certain apps on the device to run. The computing device may have a display screen and support information browsing, such as a personal mobile terminal like a mobile phone, tablet, or personal computer. Various other types of applications can also be configured on the computing device, such as human-computer interaction applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, and social media platform software.

[0100] Server 200 may include servers providing various services, such as servers providing communication services to multiple clients, servers supporting backend training of models used on clients, and servers processing data sent by clients. It should be noted that server 200 can be implemented as a distributed server cluster composed of multiple servers, or as a single server. The server can also be a server in a distributed system, or a server integrated with blockchain. The server can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0101] It is worth noting that the training method for the text processing model provided in the embodiments of this specification is generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server, thereby executing the training method for the text processing model provided in the embodiments of this specification. In other embodiments, the training method for the text processing model provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0102] Figure 6 A structural block diagram of a computing device 600 according to an embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0103] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0104] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0105] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 600 can also be a mobile or stationary server.

[0106] The processor 620 is used to execute the following computer program / instruction, which, when executed by the processor, implements the training method and text processing method steps of the above-mentioned text processing model.

[0107] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the training method of the text processing model and the technical solution of the text processing method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the training method of the text processing model and the technical solution of the text processing method described above.

[0108] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the training method and text processing method steps of the above-described text processing model.

[0109] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solution of the text processing model training method and the text processing method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the text processing model training method and the text processing method described above.

[0110] An embodiment of this specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the training method and text processing method steps of the above-described text processing model.

[0111] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the training method of the text processing model and the technical solution of the text processing method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the training method of the text processing model and the technical solution of the text processing method described above.

[0112] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0113] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or certain intermediate forms. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0114] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this specification is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this specification. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this specification.

[0115] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0116] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. These embodiments have been selected and specifically described in this specification to better explain the principles and practical applications of this specification, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A training method for a text processing model, characterized in that, The method comprises the following steps: obtaining training sample content, adding noise to the training sample content based on a target time step to obtain global noise-added content, and calculating a noise-added posterior distribution corresponding to the reference time step according to the global noise-added content and the training sample content; inputting the global noise-added content into a text processing model to obtain predicted content of the output of the text processing model, and a noise-reduced posterior distribution corresponding to the reference time step in the process of generating the predicted content by the text processing model; determining gradient compensation information corresponding to the reference time step according to the noise-added posterior distribution and the noise-reduced posterior distribution, wherein the gradient compensation information represents gradient difference information of noise distribution between the reference time step and the target time step; updating an initial loss function according to the gradient compensation information to obtain a model target loss function, and calculating a model loss value based on the model target loss function; adjusting model parameters of the text processing model according to the model loss value, and continuing to train the text processing model until a model training stop condition is reached.

2. The method of claim 1, wherein, adding noise to the training sample content based on a target time step to obtain global noise-added content, comprising: determining a first noise-added matrix and a second noise-added matrix; determining target noise-added information corresponding to the target time step according to the first noise-added matrix and the second noise-added matrix; generating global noise-added content according to the target noise-added information and the training sample content.

3. The method of claim 2, wherein, determining target noise-added information corresponding to the target time step according to the first noise-added matrix and the second noise-added matrix, comprising: determining noise-added weight information corresponding to the target time step; calculating target noise-added information corresponding to the target time step according to the noise-added weight information, the first noise-added matrix and the second noise-added matrix.

4. The method of claim 3, wherein, calculating a noise-added posterior distribution corresponding to the reference time step according to the global noise-added content and the training sample content, comprising: determining reference noise-added weight information corresponding to the reference time step; calculating a noise-added posterior distribution corresponding to the reference time step according to the training sample content, the global noise-added content, the reference noise-added weight information, the first noise-added matrix and the second noise-added matrix.

5. The method of claim 1, wherein, determining gradient compensation information corresponding to the reference time step according to the noise-added posterior distribution and the noise-reduced posterior distribution, comprising: determining reference gradient compensation information according to the noise-reduced posterior distribution and predicted noise-added posterior distribution corresponding to the predicted content; determining gradient compensation information corresponding to the reference time step according to the noise-added posterior distribution, global noise-added posterior distribution corresponding to the global noise-added content and the reference gradient compensation information.

6. The method of claim 1, wherein, updating an initial loss function according to the gradient compensation information, comprising: obtaining an initial loss function; determining a posterior distribution normalization term in the initial loss function; updating the posterior distribution normalization term according to the gradient compensation information.

7. The method of claim 6, wherein, The posterior distribution normalization term includes loss adjustment information; updating the posterior distribution normalization term according to the gradient compensation information, comprising: updating the loss adjustment information according to the gradient compensation information.

8. The method of claim 1, wherein, calculating a model loss value based on the model target loss function, comprising: Converting the model target loss function into a log loss function; Calculating a model loss value based on the log loss function.

9. A text processing method characterized by, The method comprises the steps of: Receiving text content to be processed; Inputting the text content to be processed into a text processing model to obtain text prediction content output by the text processing model, wherein the text processing model is trained by the training method of any one of claims 1-8.

10. A computing device, comprising: The method comprises the steps of: A memory and a processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which realize the steps of the method of any one of claims 1 to 9 when executed by the processor.

11. A computer readable storage medium storing computer programs / instructions, characterized in that, The computer programs / instructions realize the steps of the method of any one of claims 1 to 9 when executed by the processor.

12. A computer program product comprising computer programs / instructions, characterized in that, The computer programs / instructions realize the steps of the method of any one of claims 1 to 9 when executed by the processor. The computer programs / instructions realize the steps of the method of any one of claims 1 to 9 when executed by the processor.

Images