Text sampling generation method and system based on probability statistical model

Abstract

The invention provides a text sampling generation method and system based on a probability statistical model, and relates to the field of natural language processing, and the method comprises the following steps: obtaining an initial text and a text constraint condition of a to-be-generated field; inputting the initial text and the text constraint condition into a pre-training language model to perform text probability distribution prediction, and performing iterative loop sampling in combination with a probability statistical model to obtain a generated text; and preprocessing the generated text to obtain a final available text. The method is used for generating the text by utilizing the pre-training data of the non-large language model and adopting the sampling algorithm based on probability statistics when the large language model cannot be used due to resource limitation, the problems that hardware resources are limited and text generation is relatively weak in interpretability do not exist, and the method also has flexibility and is easy to implement. And various sampling algorithms can be easily replaced, so that richness and diversity of text generation are realized.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a text sampling generation method and system based on a probability statistics model. Background Art

[0002] The field of text generation technology, as an important branch of natural language processing (NLP), aims to enable computers to understand and generate natural language text. With the continuous development of artificial intelligence technology, text generation technology has made significant progress and has shown wide application value in many fields.

[0003] At present, text generation technology can be used in news reporting, content marketing, social media, intelligent customer service, educational assistance and other fields. The current mainstream text generation technology is mainly divided into three directions: 1. Rule-based text generation technology; 2. Template-based text generation technology; 3. Neural network-based text generation technology. As an important part of the field of artificial intelligence, text generation technology has broad application prospects and huge market potential. The widely used text generation technology is the text generation technology based on the large language model. Among them, the text generation technology based on the large language model has the following obvious defects in the application of special fields.

[0004] (1) There is a risk of information leakage: Large language models with good text generation quality are developed by large technology companies and are not open source. Therefore, there is a risk of information leakage when using such large language models.

[0005] (2) There is a problem of insufficient hardware resources: Although the text generation quality of open source large language models is acceptable, they require a relatively high consumption of hardware resources. For example, ChatGLM4 and Qwen2, which currently have relatively outstanding results, both require at least 20G of video memory to run at a relatively high speed.

[0006] (3) There is a problem of weak interpretability: Large language models are essentially neural networks, and the text they directly generate is like black box output. Therefore, the text generated by large language models is relatively weak in interpretability. Summary of the invention

[0007] The present invention provides a text sampling generation method and system based on a probability statistical model, which solves the problem that the existing text generation technology is based on a large language model, so there are limited hardware resources and weak interpretability of text generation.

[0008] In a first aspect, an embodiment of the present invention provides a text sampling generation method based on a probability statistical model, the method comprising the following process:

[0009] Obtaining the initial text and text constraints of the domain to be generated;

[0010] Input the initial text and text constraints into the pre-trained language model to predict the text probability distribution, and perform iterative cycle sampling in combination with the probability statistics model to obtain the generated text;

[0011] The generated text is preprocessed to obtain the final usable text.

[0012] In the above embodiment, the present invention uses a pre-trained language model that is not a large language model as a source for obtaining text probability distribution, and then combines iterative sampling based on a probability statistical model, and then combines it with the constraint Schema rules of the domain text to be generated, thereby achieving high-quality and diverse text generation.

[0013] As some optional implementation modes of the present application, the text constraints include subject requirements, word usage norms, sentence quantity restrictions, word count restrictions, and phonetic and rhyme requirements.

[0014] In the above embodiment, the text constraint conditions of the present invention, namely the constraint Schema rules of the domain text to be generated, mean that the generated text must comply with the domain-specific constraint conditions.

[0015] As some optional implementations of the present application, the pre-trained language model includes a Bert language model, a Roberta language model or an ALBERT language model.

[0016] In the above embodiment, the present invention uses a pre-trained model of a non-language model to determine that the operation will not consume too many hardware resources such as graphics cards.

[0017] As some optional implementation methods of the present application, the initial text and text constraints are input into the pre-trained language model to predict the text probability distribution, and the process of iterative cyclic sampling is performed in combination with the probability statistics model as follows:

[0018] Based on the text constraints of the domain to be generated, randomly select character positions from the initial text of the domain to be generated;

[0019] Perform mask replacement on the character positions corresponding to the initial text, and input the mask-replaced initial text into the pre-trained language model to predict the text probability distribution of the corresponding character positions, so as to obtain the probability distribution of the corresponding character positions;

[0020] The probability distribution of the character position is sampled iteratively and cyclically using a probability statistics model, and it is determined whether the iterative cycle reaches a stop condition. If the stop condition is reached, the generated text is updated.

[0021] In the above embodiment, the present invention adopts a sampling generation method based on a probability statistical model, which can output the text generation process transparently throughout the whole process and perceive the changes in the probability distribution of the text at each position in real time, thereby having a certain degree of interpretability.

[0022] As some optional implementations of the present application, the array size of the probability distribution is the same as the vocabulary size of the pre-trained model.

[0023] In the above embodiment, each element in the probability distribution array of the present invention is a floating point number between 0 and 1, and the sum of all elements in the array is 1.

[0024] As some optional implementations of the present application, the process of iteratively sampling the probability distribution of character positions using a probability statistical model is as follows:

[0025] Obtain a uniform probability distribution corresponding to the probability distribution, and perform probability accumulation on the probability distribution to obtain a cumulative probability distribution corresponding to the probability distribution;

[0026] Perform interval loop traversal on the cumulative probability distribution, and verify the interval loop traversal result of the cumulative probability distribution in combination with the uniform probability distribution to obtain the sampling position;

[0027] According to the sampling position, the corresponding character is searched in the vocabulary of the pre-trained language model, and the corresponding character position of the initial text is updated to obtain the sampled text.

[0028] In the above-mentioned embodiment, the present invention adopts the probability statistical model to perform iterative cyclic sampling, which means gradually approximating the model parameters or distribution through an iterative process. It is mainly used to solve the parameter estimation problem in complex probability models, especially in the case of high-dimensional or multimodal distribution. Iterative cyclic sampling can overcome the difficulties encountered by traditional methods (such as maximum likelihood estimation or least squares method) in processing complex models to a certain extent. The core idea of ​​its iterative cyclic sampling is to gradually update the parameter estimation or sample distribution in an iterative manner until a stable solution is reached.

[0029] As some optional implementation methods of the present application, it is determined whether the iterative loop reaches the stop condition. If the stop condition is reached, the process of updating the generated text is as follows:

[0030] Determine whether the iterative loop reaches a preset stop condition, wherein the stop condition includes a preset number of iterations;

[0031] If the stopping condition is reached, the final generated sample is updated according to the sampled text obtained each time. If the stopping condition is not reached, the iterative cycle sampling continues.

[0032] In the above embodiment, the present invention can choose direct random sampling and stop after one cycle, or can choose GIBBIS-based random sampling and need to cycle to a preset number of iterations; preferably, the embodiment of the present invention selects a preset number of iterations to perform iterative cycle sampling.

[0033] As some optional implementation methods of the present application, the process of preprocessing the generated text is as follows:

[0034] Perform deduplication filtering on the generated text, and perform perplexity calculation on the generated text after deduplication filtering to obtain the perplexity of the generated text;

[0035] The perplexity values ​​of the generated texts are arranged and filtered, and the arranged and filtered generated texts are rule filtered based on the text constraints of the domain to be generated to obtain the final usable text.

[0036] In the above embodiment, the present invention uses a preprocessing method to filter the generated text, so as to obtain a higher quality generated text.

[0037] As some optional implementations of the present application, the formula for calculating the perplexity is as follows:

[0038]

[0039] Among them, p(xi) represents the probability distribution of the i-th position predicted by the pre-trained language model, and n represents the total number of words in the initial text.

[0040] In the above embodiments, the perplexity involved in the present invention is an important indicator for measuring the prediction ability of the pre-trained language model. It is used to evaluate the prediction accuracy of the model for unseen data, especially in the field of natural language processing (NLP). The lower the perplexity, the stronger the prediction ability of the pre-trained language model for unknown data. The perplexity can also be used to measure whether a sentence conforms to language habits. The lower the perplexity, the less grammatical errors there are in the current text and the smoother the grammar and logic.

[0041] As some optional implementations of the present application, the formula for calculating the perplexity is as follows:

[0042]

[0043] Among them, p(xi) represents the probability distribution of the i-th position predicted by the pre-trained language model, and n represents the total number of words in the initial text.

[0044] In the above embodiments, in actual use of the present invention, the perplexity is usually calculated using logarithms to avoid numerical overflow problems.

[0045] In a second aspect, the present invention provides a text sampling generation system based on a probability statistics model, the system comprising:

[0046] A domain data unit, wherein the domain data unit is used to obtain an initial text and text constraints of a domain to be generated;

[0047] A text generation unit, wherein the text generation unit is used to input the initial text and text constraints into the pre-trained language model to predict the text probability distribution, and perform iterative cycle sampling in combination with the probability statistics model to obtain generated text;

[0048] The text post-processing unit is used to pre-process the generated text to obtain the final usable text.

[0049] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the text sampling generation method based on a probability statistical model when executing the computer program.

[0050] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the text sampling generation method based on a probability statistical model is implemented.

[0051] The beneficial effects of the present invention are as follows: When a large language model cannot be used due to resource limitations, the present invention is used to utilize pre-trained data of a non-large language model and adopt a sampling algorithm based on probability statistics for text generation. There are no problems of limited hardware resources and weak interpretability of text generation. The present invention is also flexible and can be easily replaced with a variety of sampling algorithms, thereby achieving richness and diversity in text generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0053] Figure 1 is a schematic diagram of the computer device structure of the hardware operating environment described in the embodiment of the present invention;

[0054] Figure 2 is a flowchart of the text sampling generation method according to an embodiment of the present invention;

[0055] Figure 3It is a structural block diagram of the text sampling generation system described in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In order to solve the problem that the existing text generation technology is based on a large language model, so there are limited hardware resources and weak interpretability of text generation. The embodiment of the present invention provides a text sampling generation method and system based on a probability statistical model. Before introducing the specific technical solution of the present application, the hardware operating environment involved in the embodiment of the present application is first introduced.

[0058] See also Figure 1 , Figure 1 The figure is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present invention.

[0059] like Figure 1 As shown, the computer device may include: a processor, such as a central processing unit (CPU), a communication bus, a user interface, a network interface, and a memory. Among them, the communication bus is used to realize the connection and communication between these components. The user interface may include a display screen (Display), an input unit such as a keyboard (Keyboard), and optionally, the user interface may also include a standard wired interface and a wireless interface. The network interface may be a wired interface or a wireless interface (such as a wireless fidelity interface). The memory may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory may optionally be a storage device independent of the aforementioned processor.

[0060] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the computer device, and may include more or less components than shown in the figure, or combine some components, or arrange the components differently.

[0061] like Figure 1 As shown, the memory as a storage medium may include an operating system module, a network communication module, a user interface module and a data storage module.

[0062] exist Figure 1In the computer device shown, the network interface is mainly used for data communication with the network server; the user interface is mainly used for data interaction with the user; the processor and memory in the computer device of the present application can be set in the computer device, and the computer device calls the text sampling generation system based on the probability statistical model stored in the data storage module through the processor, and executes the text sampling generation method based on the probability statistical model provided in the embodiment of the present application.

[0063] Based on the hardware environment of the aforementioned embodiment, the embodiment of the present application provides a text sampling generation method based on a probability statistical model, please refer to Figure 2 , Figure 2 The flowchart of the text sampling generation method is as follows:

[0064] (1) Obtain the initial text of the domain to be generated T0 = (t 0,1 ,t 0,2 ,...,t 0,L ) and text constraints; the text constraints are constraint Schema rules for text generation, including but not limited to subject requirements, word usage specifications, sentence quantity restrictions, word count restrictions, and rhyme requirements.

[0065] (2) The initial text and text constraints are input into the pre-trained language model to predict the text probability distribution, and the probabilistic statistical model is combined for iterative cyclic sampling to obtain the generated text.

[0066] In an embodiment of the present invention, the pre-trained language model is a pre-trained model of a non-large language model, which is mainly used for obtaining text probability distribution; specifically, the pre-trained language model includes a Bert language model, a Roberta language model or an ALBERT language model, etc.; wherein, the probability statistical model includes but is not limited to a direct random sampling probability statistical model, a GIBBIS sampling probability statistical model, etc., wherein the random sampling probability statistical model stops after one cycle, and the GIBBIS sampling probability statistical model needs to cycle to a preset number of iterations.

[0067] In an embodiment of the present invention, the initial text and text constraints are input into the pre-trained language model to predict the text probability distribution, and the process of iterative cyclic sampling is performed in combination with the probability statistics model as follows:

[0068] (2.1) Based on the text constraints of the domain to be generated, the initial text T0 of the domain to be generated is 0,1 ,t 0,2 ,...,t 0,L ) randomly selects character position k in .

[0069] (2.2) For the initial text T0 = (t0,1 ,t 0,2 ,...,t 0,L ) performs mask replacement on the corresponding character position k, and inputs the mask-replaced initial text into the pre-trained language model to predict the text probability distribution of the corresponding character position k, so as to obtain the probability distribution P(t 0,P |t 0,1 ,...,t 0,k-1 ,t 0,k+1 ,...,t 0,L ); wherein the size of the probability distribution array is the same as the vocabulary size of the pre-trained model, and each element in the array is a floating point number between 0 and 1, and the sum of all elements in the array is 1, so the probability distribution P(t 0,P |t 0,1 ,...,t 0,k-1 ,t 0,k+1 ,...,t 0,L ) is expressed as P = (P1, P2, ..., P vocab ), where P1+P2+,...,+P vocab =1.

[0070] (2.3) The probability distribution of character positions is sampled iteratively using a probabilistic statistical model, and it is determined whether the iterative cycle reaches a stopping condition. If the stopping condition is reached, the generated text is updated.

[0071] In the embodiment of the present invention, according to the probability distribution P of the character position k=(P1, P2, ..., P vocab ) The process of iterative cycle sampling is as follows:

[0072] (2.4) Obtain the probability distribution P = (P1, P2, ..., P vocab ), generate a random number rand from the uniform probability distribution U(0,1), and perform probability accumulation on the probability distribution to obtain the probability distribution P = (P1, P2, ..., P vocab ) corresponds to the cumulative probability distribution Q = (q1, q2, ..., q vocab )=(p1,p1+p2,...,p1+p2+,...,+q vocab ).

[0073] (2.5) For the cumulative probability distribution Q = (q1, q2, ..., q vocab ) to perform interval loop traversal, and verify the interval loop traversal result of the cumulative probability distribution in combination with the uniform probability distribution to obtain the sampling position; that is, initialize n = 1, and set Q = (q1, q2, ..., q vocab) performs interval loop traversal, if q n <<rand<<q n+1 , then stop the loop to get the sampling position n, otherwise set n = n + 1 and execute q again n <<rand<<q n+1 Judge until the sampling position n is obtained.

[0074] (2.6) According to the sampling position n, search for the corresponding character in the vocabulary of the pre-trained language model, and update the corresponding character position of the initial text to obtain the sampled text G, and update T i , and get T i+1 =(t i,1 ,t i,2 ,...,t i,k-1 ,G,t i,i+1 ,...,t i,L ).

[0075] In the embodiment of the present invention, it is determined whether the iterative loop reaches the stop condition. If the stop condition is reached, the process of updating the generated text is as follows:

[0076] (2.7) Determine whether the iterative loop reaches a preset stop condition, wherein the stop condition includes a preset number of iterations.

[0077] (2.8) If the stopping condition is reached, the final generated samples T1, T2, ..., T are updated according to the sampled text obtained each time. i , if the stopping condition is not reached, the iterative cycle sampling continues.

[0078] (3) Preprocess the generated text to obtain the final usable text.

[0079] In the embodiment of the present invention, since the generated text composed of the sampling samples has text duplication, grammatical errors, etc., it is necessary to pre-process the generated text.

[0080] Specifically, the process of preprocessing the generated text is as follows:

[0081] (3.1) The generated text is filtered for deduplication, and the perplexity of the generated text after deduplication is calculated to obtain the perplexity of the generated text.

[0082] In an embodiment of the present invention, the process of performing deduplication filtering on the generated text is as follows:

[0083] (3.11) Assuming that after M iterations of cyclic sampling, the generated texts T1, T2, ..., T M ;

[0084] (3.12) Initialize j = 1, and sequentially sample T j Add to string array A, if the sample text T j If it is not in the string array A, the sampled text T j Add to string array A; if the sample text T j In string array A, let j=j+1 and execute the next loop.

[0085] In an embodiment of the present invention, the perplexity (PP) is an important indicator for measuring the prediction ability of a pre-trained language model. It is used to evaluate the prediction accuracy of the model for unseen data, especially in the field of natural language processing (NLP). The lower the perplexity, the stronger the prediction ability of the pre-trained language model for unknown data. The perplexity can also be used to measure whether a sentence conforms to language habits. The lower the perplexity, the less grammatical errors there are in the current text and the smoother the grammar and logic.

[0086] Specifically, the formula for calculating the perplexity is as follows:

[0087]

[0088] Among them, p(xi) represents the probability distribution of the i-th position predicted by the pre-trained language model, and n represents the total number of words in the initial text.

[0089] Specifically, in order to avoid the problem of numerical overflow, the embodiment of the present invention can use logarithms to calculate the perplexity, and the formula for calculating the perplexity is as follows:

[0090]

[0091] Among them, p(xi) represents the probability distribution of the i-th position predicted by the pre-trained language model, and n represents the total number of words in the initial text.

[0092] Specifically, the process of using logarithms to calculate confusion is as follows:

[0093] (3.13) Initialize perplexity PP = 0 and loop flag i = 0;

[0094] (3.14) The initial text with a total number of words n is looped through, and the probability distribution P(x i );

[0095] (3.15) Let PP = PP + logp(x i )), let i=i+1, and continue to execute step (3.14) until the loop ends;

[0096] (3.16) Let PP = exp(-PP / N), and output the perplexity PP.

[0097] (3.2) Perform permutation filtering on the perplexity values of the generated text, and perform rule filtering on the generated text after permutation filtering based on the text constraint conditions of the field to be generated, so as to obtain the final available text.

[0098] Specifically, calculate the perplexity for each generated text in the string array A, sort the perplexities, and select the generated text with lower perplexity for rule filtering to obtain the final available text.

[0099] Among them, the rule filtering is performed according to the text constraint conditions of the field to be generated. For example, in the field of poetry generation, the generated text should conform to the rhyme requirements of the last characters of five-character quatrains and seven-character regulated verses, and poems with inconsistent rhymes cannot be generated; for example, the last character of the poem should conform to the probability of the same final vowel.

[0100] The following takes the generation of text for poetry exam questions based on the direct random sampling probability statistical model as an example to elaborate the detailed steps of the embodiments of the present invention:

[0101] S10: Obtain the initial text of the field to be generated and the constraint Schema rules. Among them, the initial text is the sentence "Moisten things silently" in ancient poetry. The constraint Schema rule is to mask a certain non-final character in the sentence.

[0102] S20: Input the initial text and the text constraint conditions into the pre-trained language model for text probability distribution prediction, and perform iterative loop sampling in combination with the probability statistical model to obtain the generated text.

[0103] In the embodiments of the present invention, the direct random sampling algorithm is used for one iteration of sampling, and the specific process is as follows:

[0104] S201: First, according to the constraint Schema rule, randomly perform MASK mask replacement on the character position k in the sentence; for example, replace the sentence "Moisten things silently" with "Moisten things [MASK]ly"; then the Bert language model predicts the probability distribution of the [MASK] position to obtain the probability distribution P. For example, the probability distribution P = [1.5860291e-11, 2.3655845e-11, 2.4104111e-11,..., 2.5079147e-10, 1.6116232e-11, 4.0486461e-11], where the size of the array of the probability distribution is the vocabulary size of the Bert pre-trained language model (the vocabulary size of the Bert pre-trained language model is 21128), and the probability distribution P is converted into a floating-point number array P = [0, 1, 2,..., 21127].

[0105] S202: Directly randomly sample a character according to the probability distribution P to obtain a candidate character array. Directly randomly sample according to the probability distribution in an array of the vocabulary size, obtaining the sampled number 3299. At this time, reverse look up according to the vocabulary of the Bert pre-trained language model to obtain "声".

[0106] S203: Determine whether the iterative loop reaches the stop condition. If it reaches the stop condition, update to obtain the generated text. Since the direct random sampling only performs one iterative loop and then stops, replace the original initial text to obtain the sampled text "润物细声声".

[0107] S30: Preprocess the generated text to obtain the final available text. Since there is only one generated text for "润物细声声" and it is not repeated with the original text, there is no need to process or sort; and based on the constraint Schema rules for filtering, the generated text conforms to the rhythm of the five-character regulated poem and does not need to be filtered.

[0108] Repeat S10 - S30 two more times to obtain the following generated texts in Table 1: "润物细微声", "润物细有声".

[0109] Original text Generating text Moisturizing things silently moisten things with a subtle sound, moisten things with a subtle sound, moisten things with a subtle sound

[0110] The above generated approximate texts have a very practical application scenario - generating Chinese language test questions for the compulsory education stage, generating questions to select the correct characters for poems. The above texts can be compiled into the following questions:

[0111] 1. 随风潜入夜,润物细__声。

[0112] A. 无 B. 微 C. 有 D. 声

[0113] In summary, when the large language model cannot be used due to resource constraints, the present invention utilizes the pre-trained data of non-large language models and adopts a sampling algorithm based on probability statistics for text generation. It does not have the problems of hardware resource constraints and weak interpretability of text generation, and also has flexibility, being able to easily replace with various sampling algorithms, thus achieving the richness and diversity of text generation.

[0114] In addition, in one embodiment, based on the same inventive concept as the foregoing embodiment, the embodiment of the present invention provides a text sampling generation system based on a probability statistical model. Please refer to Figure 3 , Figure 3 which is the structural block diagram of the text sampling generation system. The system corresponds one-to-one with the method of Embodiment 1. The system includes:

[0115] A domain data unit, which is used to obtain the initial text and text constraint conditions of the domain to be generated;

[0116] A text generation unit, wherein the text generation unit is used to input the initial text and text constraints into the pre-trained language model to predict the text probability distribution, and perform iterative cycle sampling in combination with the probability statistics model to obtain generated text;

[0117] The text post-processing unit is used to pre-process the generated text to obtain the final usable text.

[0118] It should be noted that each unit in the text sampling generation system based on the probability statistical model in this embodiment corresponds one-to-one to each step in the text sampling generation method based on the probability statistical model in the aforementioned embodiment. Therefore, the specific implementation method and technical effects achieved by this embodiment can refer to the implementation method of the aforementioned text sampling generation method based on the probability statistical model, and will not be repeated here.

[0119] In addition, in one embodiment, the present application also provides a computer device, which includes a processor, a memory, and a computer program stored in the memory, and the computer program implements the method in the aforementioned embodiment when executed by the processor.

[0120] In addition, in one embodiment, the present application further provides a computer storage medium, on which a computer program is stored, and the computer program implements the method in the aforementioned embodiment when executed by a processor.

[0121] In some embodiments, the computer readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories. The computer may be various computing devices including intelligent terminals and servers.

[0122] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0123] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0124] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0125] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0126] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0127] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a multimedia terminal device (which can be a mobile phone, a computer, a television receiver, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0128] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A text sampling generation method based on a probability statistical model, characterized in that: The method comprises the following steps: Obtaining the initial text and text constraints of the domain to be generated; Input the initial text and text constraints into the pre-trained language model to predict the text probability distribution, and perform iterative cycle sampling in combination with the probability statistics model to obtain the generated text; The generated text is preprocessed to obtain the final usable text.

2. A text sampling generation method based on a probability statistical model according to claim 1, characterized in that: The text constraints include subject requirements, word usage specifications, sentence quantity restrictions, word count restrictions, and rhyme requirements.

3. A text sampling generation method based on a probability statistical model according to claim 1, characterized in that: The pre-trained language model includes a Bert language model, a Roberta language model or an ALBERT language model.

4. A text sampling generation method based on a probability statistical model according to claim 1, characterized in that: The process of inputting the initial text and text constraints into the pre-trained language model to predict the text probability distribution and combining the probability statistics model for iterative cycle sampling is as follows: Based on the text constraints of the domain to be generated, randomly select character positions from the initial text of the domain to be generated; Perform mask replacement on the character positions corresponding to the initial text, and input the mask-replaced initial text into the pre-trained language model to predict the text probability distribution of the corresponding character positions, so as to obtain the probability distribution of the corresponding character positions; The probability distribution of the character position is sampled iteratively and cyclically using a probability statistics model, and it is determined whether the iterative cycle reaches a stop condition. If the stop condition is reached, the generated text is updated.

5. A text sampling generation method based on a probability statistical model according to claim 4, characterized in that: The array size of the probability distribution is the same as the vocabulary size of the pre-trained model.

6. A text sampling generation method based on a probability statistical model according to claim 4, characterized in that: The process of iteratively sampling the probability distribution of character positions using a probabilistic statistical model is as follows: Obtain a uniform probability distribution corresponding to the probability distribution, and perform probability accumulation on the probability distribution to obtain a cumulative probability distribution corresponding to the probability distribution; Perform interval loop traversal on the cumulative probability distribution, and verify the interval loop traversal result of the cumulative probability distribution in combination with the uniform probability distribution to obtain the sampling position; According to the sampling position, the corresponding character is searched in the vocabulary of the pre-trained language model, and the corresponding character position of the initial text is updated to obtain the sampled text.

7. A text sampling generation method based on a probability statistical model according to claim 6, characterized in that: Determine whether the iterative loop reaches the stop condition. If the stop condition is reached, the process of updating the generated text is as follows: Determine whether the iterative loop reaches a preset stop condition, wherein the stop condition includes a preset number of iterations; If the stopping condition is reached, the final generated sample is updated according to the sampled text obtained each time. If the stopping condition is not reached, the iterative cycle sampling continues.

8. A text sampling generation method based on a probability statistical model according to claim 1, characterized in that: The process of preprocessing the generated text is as follows: Perform deduplication filtering on the generated text, and perform perplexity calculation on the generated text after deduplication filtering to obtain the perplexity of the generated text; The perplexity values ​​of the generated texts are arranged and filtered, and the arranged and filtered generated texts are rule filtered based on the text constraints of the domain to be generated to obtain the final usable text.

9. A text sampling generation method based on a probability statistical model according to claim 8, characterized in that: The formula for calculating the perplexity is as follows: Among them, p(xi) represents the probability distribution of the i-th position predicted by the pre-trained language model, and n represents the total number of words in the initial text.

10. The text sampling generation method based on a probability statistical model according to claim 8 is characterized in that: The formula for calculating the perplexity is as follows: Among them, p(xi) represents the probability distribution of the i-th position predicted by the pre-trained language model, and n represents the total number of words in the initial text.

11. A text sampling generation system based on a probability statistical model, characterized in that: The system comprises: A domain data unit, wherein the domain data unit is used to obtain an initial text and text constraints of a domain to be generated; A text generation unit, wherein the text generation unit is used to input the initial text and text constraints into the pre-trained language model to predict the text probability distribution, and perform iterative cycle sampling in combination with the probability statistics model to obtain generated text; The text post-processing unit is used to pre-process the generated text to obtain the final usable text.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for generating text sampling based on a probability statistical model as described in any one of claims 1 to 10 is implemented.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for generating text sampling based on a probability statistical model described in any one of claims 1 to 10 is implemented.

Images