Text generation method, device, electronic device and storage medium

Abstract

The present disclosure provides a text generation method, device, electronic device and storage medium. A specific implementation of the method includes: obtaining a first text; inputting a text vector of the first text into a language model pre-trained using a differential privacy model training method to obtain a first occurrence probability of different candidate words in a preset candidate word set appearing after the first text; based on the first occurrence probability of each of the candidate words, generating a second text corresponding to the first text in the current application scenario. This implementation improves the privacy protection of generating a second text corresponding to the first text in different application scenarios, and can reduce the risk of leakage of training data information of the language model used to generate the second text.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the technical field of natural language processing, and in particular to a text generation method, device, electronic device, and storage medium. Background Art

[0002] In the field of natural language processing, large-scale language models have been widely used. Most language models are pre-trained using various training data. However, in the process of language model training, there is a risk of leaking training data information due to the use of a large amount of corpus data. In actual applications, the use of the above language models also has the risk of leaking training data information in different application scenarios. Summary of the invention

[0003] The embodiments of the present disclosure provide a text generation method, device, electronic device, and storage medium.

[0004] In a first aspect, an embodiment of the present disclosure provides a text generation method, which includes: obtaining a first text; inputting a text vector of the first text into a language model pre-trained using a differential privacy model training method to obtain a first occurrence probability of different candidate words in a preset candidate word set appearing after the first text, wherein, in the process of training the language model using a training data set, the gradient corresponding to the training data in the training data set is clipped, the noise to be added is determined according to the privacy degree of the sample text in the training data set, and the noise to be added is added to the average clipped gradient corresponding to the training data set; based on the first occurrence probability of each of the candidate words, a second text corresponding to the first text in the current application scenario is generated.

[0005] In some optional implementations, the method further includes: presenting the second text.

[0006] In some optional implementations, the current application scenario is a text input scenario or a speech recognition scenario, and the first text is input text or recognized text.

[0007] In some optional embodiments, the second text corresponding to the first text in the current application scenario is generated based on the first occurrence probability of each candidate word, including: sorting each candidate word in descending order according to the first occurrence probability; generating a candidate word subset using the candidate words among the candidate words sorted within a first preset higher occurrence probability range; and forming the second text based on each candidate word in the candidate word subset and the corresponding first occurrence probability.

[0008] In some optional embodiments, the current application scenario is an auxiliary writing scenario, and the first text is an input text; and based on the first occurrence probability of each of the candidate words, generating a second text corresponding to the first text in the current application scenario, includes: splicing the candidate words with the highest first occurrence probability among the candidate words after the first text to form a spliced ​​text; performing the following splicing operations a preset number of times: inputting the text vector of the spliced ​​text into the language model to obtain the second occurrence probability of each of the candidate words appearing after the spliced ​​text; splicing the candidate words with the highest second occurrence probability among the candidate words after the spliced ​​text; and generating the second text using text other than the first text in the spliced ​​text.

[0009] In some optional embodiments, the current application scenario is a question-and-answer scenario, and the first text is a question text; and the second text corresponding to the first text in the current application scenario is generated based on the occurrence probability of each of the candidate words, including: sorting each of the candidate words in order from large to small according to the first occurrence probability; generating a set of reply candidate words using the candidate words sorted within a second preset higher occurrence probability range among the candidate words; determining the reply text corresponding to each of the reply candidate words according to the correspondence between the preset reply keywords and the reply text; and generating the second text based on the reply text corresponding to each of the reply candidate words.

[0010] In some optional embodiments, the language model is pre-trained through the following training steps: obtaining an initial language model and at least one training data set, the training data including sample text, sample text vector and the probability of occurrence of the annotation of each candidate word; performing the following parameter adjustment operation on the training data set in the at least one training data set until the preset training end condition is met, the parameter adjustment operation including: adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise; clipping the gradient corresponding to each training data in the training data set to obtain the corresponding clipped gradient, and determining the average clipped gradient corresponding to the training data set; adding the obtained privacy-adjusted random noise to the average clipped gradient corresponding to the training data set to obtain the noise gradient corresponding to the training data set; based on the noise gradient corresponding to the training data set, using a preset gradient descent optimization algorithm to adjust the model parameters of the initial language model; determining the initial language model as the pre-trained language model.

[0011] In some optional embodiments, before adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise, the parameter adjustment operation also includes: performing the following privacy calculation operation for the training data in the training data set: inputting the sample text vector in the training data into the initial language model to obtain the predicted occurrence probability corresponding to the training data, which is used to predict the occurrence of each candidate word after the sample text in the training data; determining the privacy degree corresponding to the training data, which is used to characterize the privacy degree of the sample text in the training data in the training data set, based on the obtained predicted occurrence probability of each candidate word; and determining the privacy weight of the training data set, which is used to characterize the overall privacy degree of each sample text in the training data set, based on the privacy degree corresponding to each training data in the training data set.

[0012] In some optional embodiments, determining the privacy weight of the training data set for characterizing the overall privacy level of each sample text in the training data set based on the privacy level corresponding to each training data in the training data set includes: determining the mean of the privacy levels corresponding to each training data in the training data set as the privacy weight of the training data set.

[0013] In some optional embodiments, the step of clipping the gradients corresponding to the training data in the training data set to obtain corresponding clipped gradients includes: performing the following gradient clipping operations on the training data in the training data set: determining the loss function value between the predicted occurrence probability and the labeled occurrence probability of the candidate words corresponding to the training data; determining the gradient corresponding to the training data based on the determined loss function value; and clipping the gradient corresponding to the training data according to a preset gradient clipping norm to obtain the clipped gradient corresponding to the training data.

[0014] In some optional embodiments, the step of adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise includes: adjusting the preset random noise according to the privacy weight of the training data set and the preset gradient clipping norm to obtain the privacy-adjusted random noise of the training data set.

[0015] In some optional embodiments, determining the privacy degree corresponding to the training data and used to characterize the degree of privacy of the sample text in the training data in the training data set based on the obtained predicted occurrence probability of each of the candidate words includes: determining the perplexity of the sample text in the training data predicted by the initial language model based on the obtained predicted occurrence probability of each of the candidate words; determining the privacy degree corresponding to the training data according to the perplexity of the sample text in the training data predicted by the initial language model, wherein the privacy degree corresponding to the training data is positively linearly correlated with the perplexity of the sample text in the training data predicted by the initial language model.

[0016] In a second aspect, an embodiment of the present disclosure provides a text generation device, which includes: an acquisition unit, configured to acquire a first text; an input unit, configured to input a text vector of the first text into a language model pre-trained using a differential privacy model training method, to obtain a first occurrence probability of different candidate words in a preset candidate word set appearing after the first text, wherein, in the process of training the language model using a training data set, the gradient corresponding to the training data in the training data set is clipped, the noise to be added is determined according to the privacy degree of the sample text in the training data set, and the noise to be added is added to the average clipped gradient corresponding to the training data set; a generation unit, configured to generate a second text corresponding to the first text in the current application scenario based on the first occurrence probability of each of the candidate words.

[0017] In some optional implementations, the device further includes: a presentation unit configured to present the second text.

[0018] In some optional implementations, the current application scenario is a text input scenario or a speech recognition scenario, and the first text is input text or recognized text.

[0019] In some optional embodiments, the generation unit is further configured to: sort each of the candidate words in order from large to small according to the first occurrence probability; generate a candidate word subset using the candidate words among the candidate words sorted within a first preset higher occurrence probability range; and form the second text based on each candidate word in the candidate word subset and the corresponding first occurrence probability.

[0020] In some optional embodiments, the current application scenario is an auxiliary writing scenario, and the first text is an input text; and the generation unit is further configured to: splice the candidate word with the highest first occurrence probability among the candidate words after the first text to form a spliced ​​text; perform the following splicing operations a preset number of times: input the text vector of the spliced ​​text into the language model to obtain the second occurrence probability of each candidate word after the spliced ​​text; splice the candidate word with the highest second occurrence probability among the candidate words after the spliced ​​text; and generate the second text using the text in the spliced ​​text except the first text.

[0021] In some optional embodiments, the current application scenario is a question-and-answer scenario, the first text is a question text; and the generation unit is further configured to: sort each of the candidate words in order from large to small according to a first probability of occurrence; generate a set of reply candidate words using the candidate words ranked within a second preset higher probability of occurrence among the candidate words; determine the reply text corresponding to each of the reply candidate words according to the correspondence between preset reply keywords and reply texts; and generate the second text based on the reply text corresponding to each of the reply candidate words.

[0022] In some optional embodiments, the language model is pre-trained through the following training steps: obtaining an initial language model and at least one training data set, the training data including sample text, sample text vector and the probability of occurrence of the annotation of each candidate word; performing the following parameter adjustment operation on the training data set in the at least one training data set until the preset training end condition is met, the parameter adjustment operation including: adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise; clipping the gradient corresponding to each training data in the training data set to obtain the corresponding clipped gradient, and determining the average clipped gradient corresponding to the training data set; adding the obtained privacy-adjusted random noise to the average clipped gradient corresponding to the training data set to obtain the noise gradient corresponding to the training data set; based on the noise gradient corresponding to the training data set, using a preset gradient descent optimization algorithm to adjust the model parameters of the initial language model; determining the initial language model as the pre-trained language model.

[0023] In some optional embodiments, before adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise, the parameter adjustment operation also includes: performing the following privacy calculation operation for the training data in the training data set: inputting the sample text vector in the training data into the initial language model to obtain the predicted occurrence probability corresponding to the training data, which is used to predict the occurrence of each candidate word after the sample text in the training data; determining the privacy degree corresponding to the training data, which is used to characterize the privacy degree of the sample text in the training data in the training data set, based on the obtained predicted occurrence probability of each candidate word; and determining the privacy weight of the training data set, which is used to characterize the overall privacy degree of each sample text in the training data set, based on the privacy degree corresponding to each training data in the training data set.

[0024] In some optional embodiments, determining the privacy weight of the training data set for characterizing the overall privacy level of each sample text in the training data set based on the privacy level corresponding to each training data in the training data set includes: determining the mean of the privacy levels corresponding to each training data in the training data set as the privacy weight of the training data set.

[0025] In some optional embodiments, the step of clipping the gradients corresponding to the training data in the training data set to obtain corresponding clipped gradients includes: performing the following gradient clipping operations on the training data in the training data set: determining the loss function value between the predicted occurrence probability and the labeled occurrence probability of the candidate words corresponding to the training data; determining the gradient corresponding to the training data based on the determined loss function value; and clipping the gradient corresponding to the training data according to a preset gradient clipping norm to obtain the clipped gradient corresponding to the training data.

[0026] In some optional embodiments, the step of adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise includes: adjusting the preset random noise according to the privacy weight of the training data set and the preset gradient clipping norm to obtain the privacy-adjusted random noise of the training data set.

[0027] In some optional embodiments, determining the privacy degree corresponding to the training data and used to characterize the degree of privacy of the sample text in the training data in the training data set based on the obtained predicted occurrence probability of each of the candidate words includes: determining the perplexity of the sample text in the training data predicted by the initial language model based on the obtained predicted occurrence probability of each of the candidate words; determining the privacy degree corresponding to the training data according to the perplexity of the sample text in the training data predicted by the initial language model, wherein the privacy degree corresponding to the training data is positively linearly correlated with the perplexity of the sample text in the training data predicted by the initial language model.

[0028] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0029] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by one or more processors, implements the method described in any implementation manner in the first aspect.

[0030] In order to reduce the risk of leaking training data information in the existing text generation process, the text generation method, device, electronic device and storage medium provided by the embodiments of the present disclosure, in the scenario specifically applied to the language model, first obtain a first text, then input the first text into a language model pre-trained by a differential privacy model training method, so as to obtain the first occurrence probability of different candidate words in a preset candidate word set appearing after the first text, wherein, in the process of training the language model using the training data set, the gradient corresponding to the training data in the training data set is clipped, the noise to be added is determined according to the privacy degree of the sample text in the training data set, and the noise to be added is added to the average clipped gradient corresponding to the training data set; finally, based on the first occurrence probability of each candidate word, a second text corresponding to the first text in the current application scenario is generated. Since the language model is pre-trained by the differential privacy model training method, and the noise amplitude of the added noise is appropriately adjusted according to the privacy degree of the sample text in the training data during the training process, the risk of the language model leaking the training data in the specific application scenario can be reduced, and the effect of the language model protecting the privacy text and using the non-privacy text normally can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Other features, objects and advantages of the present disclosure will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings. The drawings are only for the purpose of illustrating specific embodiments and are not to be considered as limiting the present invention. In the drawings:

[0032] Figure 1 is an exemplary system architecture diagram in which an embodiment of the present disclosure may be applied;

[0033] Figure 2 is a flow chart of an embodiment of a text generation method according to the present disclosure;

[0034] Figure 3 is a decomposed flow chart of one embodiment of step 203 according to the present disclosure;

[0035] Figure 4 is a decomposed flow chart of yet another embodiment of step 203 according to the present disclosure;

[0036] Figure 5 is a decomposed flow chart of another embodiment of step 203 according to the present disclosure;

[0037] Figure 6 is a flow chart of one embodiment of the training steps according to the present disclosure;

[0038] Figure 7 is a structural schematic diagram of an embodiment of a text generation device according to the present disclosure;

[0039] Figure 8 It is a structural diagram of a computer system of an electronic device suitable for implementing an embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] The present disclosure is further described in detail below in conjunction with the accompanying drawings and embodiments. It is understood that the specific embodiments described herein are only used to explain the relevant invention, rather than to limit the invention. It is also necessary to explain that, for ease of description, only the parts related to the relevant invention are shown in the accompanying drawings.

[0041] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present disclosure may be combined with each other. The present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0042] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the text generation method, apparatus, electronic device, and storage medium of the present disclosure can be applied.

[0043] like Figure 1As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0044] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as text editing applications, writing assistance applications, intelligent question and answer applications, natural language processing applications, voice recognition applications, short video social applications, audio and video conferencing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0045] Terminal devices 101, 102, 103 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with sound acquisition devices (such as microphones), video acquisition devices (such as cameras) and display screens, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III, Moving Picture Experts Compression Standard Audio Layer 3), MP4 (Moving Picture Experts Group Audio Layer IV, Moving Picture Experts Compression Standard Audio Layer 4) players, laptop computers and desktop computers, etc. When terminal devices 101, 102, 103 are software, they can be installed in the terminal devices listed above. It can be implemented as multiple software or software modules (for example, used to provide text generation services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0046] In some cases, the text generation method provided by the present disclosure may be executed by the terminal devices 101, 102, 103, and accordingly, the text generation apparatus may be disposed in the terminal devices 101, 102, 103. In this case, the system architecture 100 may not include the server 105.

[0047] In some cases, the text generation method provided by the present disclosure can be jointly performed by the terminal devices 101, 102, 103 and the server 105. For example, the step of "obtaining the first text" can be performed by the terminal devices 101, 102, 103, and the steps of "inputting the first text into a language model pre-trained using a differential privacy model training method" can be performed by the server 105. The present disclosure does not limit this. Accordingly, the text generation device can also be respectively set in the terminal devices 101, 102, 103 and the server 105.

[0048] In some cases, the text generation method provided by the present disclosure may be executed by the server 105 , and accordingly, the text generation device may also be set in the server 105 . In this case, the system architecture 100 may not include the terminal devices 101 , 102 , and 103 .

[0049] It should be noted that the server 105 can be hardware or software. When the server 105 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server 105 is software, it can be implemented as multiple software or software modules (for example, for providing distributed services), or it can be implemented as a single software or software module. No specific limitation is made here.

[0050] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0051] Continue to refer Figure 2 , which shows a process 200 of an embodiment of a text generation method according to the present disclosure, the text generation method comprises the following steps:

[0052] Step 201, obtaining a first text.

[0053] In this embodiment, the execution subject of the text generation method (for example Figure 1 The terminal device shown) can adopt corresponding methods to obtain the first text according to different specific current application scenarios.

[0054] In some optional implementations, the above-mentioned execution entity (for example Figure 1 The terminal device shown) can obtain the first text locally.

[0055] In some optional implementations, the above-mentioned execution entity (for example Figure 1 The server shown in the figure can also be remotely connected to the execution subject network from other electronic devices (such as Figure 1 The terminal device shown) obtains the first text.

[0056] In some optional implementations, the first text may be text that has been input by the user using the terminal device. For example, the first text may be text on the screen that has been input by the user using the input method application installed on the terminal device and has not been confirmed to be selected (for example, text displayed in the input field of the input method). For another example, the first text may also be text that has been input by the user using the input method application installed on the terminal device and has been confirmed to be selected.

[0057] In some optional implementations, the first text may also be a recognized text that has been recognized during a process of performing speech recognition on certain speech data.

[0058] Step 202 : Input the text vector of the first text into a language model pre-trained using a differential privacy model training method to obtain the first occurrence probability of different candidate words in a preset candidate word set appearing after the first text.

[0059] In this embodiment, the execution subject may first perform vector representation on the first text to obtain a text vector of the first text. Then, the text vector of the first text is input into a language model pre-trained using a differential privacy model training method to obtain the first occurrence probability of different candidate words in the preset candidate word set appearing after the first text.

[0060] It should be noted that how to vectorize text is an existing technology that is widely studied and applied at present, and is not specifically limited here. For example, the text can be first segmented to obtain a segmentation sequence; then, each segmentation in the analysis sequence is vectorized by various word vector representation methods to obtain a segmentation vector sequence; finally, the mean vector of each segmentation vector in the segmentation vector sequence is calculated to obtain the text vector. Among them, various methods can also be used to vectorize the segmentation. For example, the One Hot method and the distributed representation method can be used, and the distributed representation method can include: matrix-based distributed representation, clustering-based distributed representation, or neural network-based distributed representation, etc. Of course, in practice, existing word vector representation tools, such as Glove, word2vec, fasttext, WordRank, etc., can be used to vectorize the segmentation.

[0061] Here, the language model may be a model obtained by training with various text data, which is used to represent the correspondence between a text vector representing a text and the probability of each candidate word in a preset candidate word set appearing after the text.

[0062] For example, the language model can be N-Gram, BERT (Bidirectional Encoder Representation from Transformers), GPT (Generative Pre-Training), etc.

[0063] In the process of training the language model, the differential privacy model training method can be used to pre-train the above language model. Specifically, in the process of training the language model using a training data set (which can also be understood as a batch of training data), the average gradient of each training data in the batch of training data is not directly back-propagated to optimize the model parameters of the language model. Instead, it is performed as follows: the gradient corresponding to the training data in the training data set is clipped, the noise to be added is determined according to the privacy level of the sample text in the training data set, and the noise to be added is added to the average clipped gradient corresponding to the training data set.

[0064] In the process of training each batch of training data, the gradient corresponding to each training data in the batch is clipped to obtain the clipped gradient; then the noise to be added is determined according to the privacy degree of the sample text in the batch of training data; then, the noise to be added is added to the average clipped gradient corresponding to the batch of training data; finally, the average clipped gradient of the batch of training data after adding noise is back-propagated using the parameter optimization method to update the model parameters. In order to protect the privacy information of the sample text in the training data, when determining the noise to be added, a higher noise amplitude can be determined for the sample text with a higher privacy degree, so that it is difficult for the language model to learn in the sample text with a higher privacy degree, so as to improve the protection of the language model for the sample text with a higher privacy degree; conversely, a lower noise amplitude can be determined for the sample text with a lower privacy degree, so that it is easy for the language model to learn in the sample text with a higher privacy degree, so as to improve the learning degree of the language model for the sample text with a lower privacy degree. Thus, the effect of protecting the private text and using the non-private text normally is achieved. In addition, since random noise is added to the gradient in the above process, the risk of the language model leaking the training data information can be reduced.

[0065] Step 203: Generate a second text corresponding to the first text in the current application scenario based on the first occurrence probability of each candidate word.

[0066] In this embodiment, the execution subject may generate a second text corresponding to the first text in the current application scenario based on the first occurrence probability of each candidate word by adopting a corresponding method according to different specific current application scenarios.

[0067] In some optional implementations, the current application scenario is a text input scenario or a speech recognition scenario. In the text input scenario, the first text is the input text. In the speech recognition scenario, the first text is the recognized text. Accordingly, step 203 may include the following: Figure 3 Steps 2031a to 2033a are shown:

[0068] Step 2031a, sort the candidate words in descending order according to the first occurrence probability.

[0069] Step 2032a: Generate a candidate word subset using the candidate words ranked within a first preset higher probability range among the candidate words.

[0070] Here, the first preset higher occurrence probability range may correspond to the candidate words ranked before the first preset proportion among the candidate words. For example, all the candidate words ranked in the top 5% of the first occurrence probability. Accordingly, the candidate word subset may be composed of the candidate words ranked before the first preset proportion among the candidate words.

[0071] The first preset higher occurrence probability range may also correspond to the candidate words ranked in the first preset position and before among the candidate words. For example, the candidate words ranked in the top three by the first occurrence probability. Accordingly, the candidate word subset may be composed of the candidate words ranked in the first preset position and before among the candidate words by the first occurrence probability.

[0072] In summary, the candidate word subset is composed of the candidate words with the highest first appearance probability in the preset candidate word set.

[0073] Step 2033a: forming a second text based on each candidate word in the candidate word subset and the corresponding first occurrence probability.

[0074] For example, the second text may be composed of N second subtexts, where N may be the number of candidate words in the candidate word subset. Each second subtext may correspond to each candidate word in the candidate word subset. For example, the second subtext may only include candidate words. For another example, the second subtext may also include the ranking sequence number of the first occurrence probability corresponding to the candidate word in the candidate word subset and the corresponding candidate word. In addition, each second subtext in the second text may also be arranged according to the ranking of the first occurrence probability of the corresponding candidate word in the candidate word subset.

[0075] For text input scenarios, this method can be used to sort words with a higher first occurrence probability at a front position, making it easier for users to select and improving the efficiency of user text input.

[0076] For speech recognition scenarios, this method can be used to sort the words with the highest probability of first appearance at a front position, thereby facilitating the selection of homophonic or similar-sounding words during speech recognition and improving the accuracy of speech recognition.

[0077] In some optional implementations, the current application scenario may be an auxiliary writing scenario, and the first text may be an input text. Accordingly, step 203 may include the following steps: Figure 4 Steps 2031b to 2033b are shown:

[0078] Step 2031b: concatenate the candidate word with the highest first occurrence probability among the candidate words after the first text to form a concatenated text.

[0079] After step 2031b, a candidate word with the highest probability of appearing first after the first text is added to the concatenated text based on the first text.

[0080] Step 2032b, performing a preset number of splicing operations.

[0081] Here, the splicing operation may include:

[0082] First, the text vector of the concatenated text is input into the language model to obtain the second occurrence probability of each candidate word after the concatenated text.

[0083] Secondly, the candidate word with the second highest probability of occurrence among the candidate words is concatenated after the concatenated text.

[0084] It is understandable that, without a single concatenation operation, the concatenated text is added with a candidate word that has the second highest probability of appearing after the concatenated text on the basis of the original text.

[0085] Assume that the preset number is a positive integer M. After step 2031b, the concatenated text adds one candidate word based on the first text, and after step 2032b, the concatenated text adds (1+M) candidate words based on the first text. Thus, auxiliary writing is achieved based on the first text, which can reduce the writing burden of the user and improve the writing efficiency of the user.

[0086] Step 2033b, generating a second text using the text in the concatenated text except the first text.

[0087] That is, the second text is generated here using the text spliced ​​to the first text in steps 2031b and 2032b.

[0088] In some optional implementations, the current application scenario may be a question-and-answer scenario, and the first text may be a question text. Accordingly, step 203 may include: Figure 5Steps 2031c to 2033c shown:

[0089] Step 2031c, sort the candidate words in descending order according to the first occurrence probability.

[0090] Step 2032c, generating a set of answer candidate words using the candidate words ranked within a second preset higher probability range among the candidate words.

[0091] Here, the second preset higher occurrence probability range may correspond to the candidate words ranked before the second preset ratio among the candidate words. For example, all the candidate words ranked in the top 5% of the second occurrence probability. Accordingly, the reply candidate word set may be composed of the candidate words ranked before the second preset ratio among the candidate words with the first occurrence probability.

[0092] The second preset higher occurrence probability range may also correspond to the candidate words ranked in the first second preset position and before among the candidate words. For example, the candidate words ranked in the first three in the occurrence probability ranking. Accordingly, the reply candidate word set may be composed of the candidate words ranked in the first second preset position and before among the candidate words.

[0093] In summary, the answer candidate word set is composed of the candidate words with the highest probability of first appearance in the preset candidate word set.

[0094] Step 2033c, according to the correspondence between the preset reply keywords and the reply text, determine the reply text corresponding to each reply candidate word.

[0095] Here, a corresponding correspondence table may be preset for the reply keywords and the reply texts, and based on the correspondence table, the reply texts corresponding to the reply candidate words in step 2032c may be queried.

[0096] Step 2034c, generating a second text based on the reply text corresponding to each reply candidate word.

[0097] For example, the second text may be composed of S second sub-texts, where S may be the number of candidate words in the set of candidate answer words. Among them, each second sub-text may correspond to each candidate answer word in the set of candidate answer words. For example, the second sub-text may only include the answer text corresponding to the candidate answer word. For another example, the second sub-text may also include the ranking number of the first occurrence probability corresponding to the answer candidate word in the set of candidate answer words and the answer text corresponding to the corresponding answer candidate word. In addition, each second sub-text in the second text may also be arranged according to the ranking of the first occurrence probability of the corresponding answer candidate word in the set of candidate answer words.

[0098] In some optional implementations, the above process 200 may further include the following steps 204:

[0099] Step 204: present the second text.

[0100] For example, for text input scenarios, auxiliary writing scenarios, and question-and-answer scenarios, the second text can be displayed in a preset area relative to the first text. For speech recognition scenarios, the second text can be continuously displayed at a position adjacent to the first text display position.

[0101] In some cases, this embodiment may have the following optional implementations:

[0102] Optional implementation (I): The language model can be obtained by Figure 6 The training step 600 shown is obtained by pre-training, and the training step 600 includes the following steps 601 to 603:

[0103] Step 601: Obtain an initial language model and at least one training data set.

[0104] Here, the initial language model may be a model of the correspondence between various text vectors used to represent text and the probability of each candidate word in a preset candidate word set appearing after the text. This application does not make any specific limitation on this.

[0105] Assume that the initial language model is Lm, and in step 601, a positive integer T training data sets are obtained, which are training data sets B t , t is a positive integer between 1 and T. Assume that the training data set B t It includes a positive integer J training data, which are training data x j (j is a positive integer between 1 and J). Assume that the preset candidate word set includes a positive integer I candidate words w i , i is a positive integer between 1 and I.

[0106] Here, the training data x j It can include sample text s, sample text vector v of sample text s s And each candidate word w in I candidate words i The probability of the tag appearing after the sample text s

[0107] In practice, the probability of labeling It can be obtained through manual annotation or statistical analysis of corpus data.

[0108] Step 602: Perform parameter adjustment operations on a training data set in at least one training data set until a preset training end condition is met.

[0109] Here, for the training data set B in step 601, t , perform parameter adjustment operations until the preset training end conditions are met.

[0110] Here, the parameter adjustment operation may include the following steps 6021 to 6024:

[0111] Step 6021: Adjust the preset random noise according to the privacy weight of the training data set to obtain privacy-adjusted random noise.

[0112] Here, each training data set B t There may be a corresponding privacy weight for characterizing the privacy degree of the sample text content in each training data in the training data set. Training data set B t The higher the privacy level of the sample text content in each training data, the higher the privacy level of the training data set B. t Privacy weight The higher the value, the more the training data set B is needed to reduce the leakage of the trained language model. t The risk of sample text content in each training data can be calculated by adding the training data set B t The corresponding average clipped gradient adds a higher amplitude noise. That is, in step 6021, when the preset random noise is adjusted, the privacy-adjusted random noise can be consistent with the training data set B t Privacy weight Positive correlation. Optionally, step 6021 can be performed according to the following formula 1:

[0113]

[0114] Where N is the preset random noise, is the privacy weight of the training data set The privacy-adjusted random noise is obtained by adjusting the preset random noise N.

[0115] Training data set B t Privacy weight It can be determined in various ways. For example, the training data set B t Privacy weight It may be manually specified according to the privacy level of the sample text content in each training data in the training data set.

[0116] Optionally, here, the preset random noise N may be various random noises, for example, Gaussian noise.

[0117] Step 6022, clipping the gradient corresponding to each training data in the training data set to obtain the corresponding clipped gradient, and determining the average clipped gradient corresponding to the training data set.

[0118] Here, assume that the training data set B t The training data x in j After inputting the initial language model, the corresponding output result will be obtained, that is, the training data x will be used to predict j After the sample text appears each candidate word w i The predicted probability of Next, various loss function calculation methods can be used to determine the training data x j The corresponding candidate words w i The predicted probability of and the probability of labeling The loss function value between Furthermore, based on the determined loss function value Determine the training data x j The corresponding gradient

[0119] Here, various gradient clipping methods can be used to train the data set B t Each training data x j The corresponding gradient Perform clipping to obtain the corresponding clipping gradient It should be noted that here, the training data x j Clipped gradient Usually less than or equal to its original gradient To avoid exploding gradients.

[0120] After getting the training data set B t Each training data x j Clipped gradient After that, the training data set B can be calculated t Each training data x j Clipped gradient The mean of the training data set B t The average clipped gradient

[0121] Step 6023, adding the obtained privacy-adjusted random noise to the average clipped gradient corresponding to the training data set to obtain the noise gradient corresponding to the training data set.

[0122] That is, the training data set B obtained in step 6022 t The corresponding average clipped gradient Add the training data set B obtained in step 6021t Privacy-adjusted random noise To obtain the training data set B t The corresponding noise gradient Specifically, the following formula (2) can be used:

[0123]

[0124] In practice, considering and The magnitude difference between j The corresponding gradient The magnitude is generally in e -4 , where e is a natural number, and the training data x after trimming j Clipped gradient The magnitude of will be smaller, and then the training data set B t The corresponding average clipped gradient The magnitude of is also relatively small, and the training data set B t Privacy-adjusted random noise The magnitude of the noise is relatively large), and the following formula (3) can also be used to add noise:

[0125]

[0126] Step 6024: Based on the noise gradient corresponding to the training data set, a preset gradient descent optimization algorithm is used to adjust the model parameters of the initial language model.

[0127] For example, the following gradient descent (GD) optimization algorithms may be used, including but not limited to: batch gradient descent (BGD), mini-batch gradient descent (MBGD), stochastic gradient descent (SGD), gradient descent with momentum (GDM), Nesterov accelerated gradient (Nesterov Accelerated Gradient, NAG), RMSprop (Root Mean Square Prop) algorithm, Adaptive Moment Estimation (Adaptive Moment Estimation, Adam) algorithm, etc.

[0128] The preset training conditions in step 602 may be various preset conditions for determining the convergence of the language model. For example, the preset training conditions may include at least one of the following conditions 1 to 4:

[0129] Condition 1: The number of times the parameter adjustment operation is performed is greater than or equal to a preset number. For example, the preset number may be the number of training data sets acquired in step 6021.

[0130] Condition 2: The time required to perform the parameter adjustment operation exceeds the preset training time.

[0131] Condition 3: the sum of the loss function values ​​corresponding to each training data in the training data set during the parameter adjustment operation is less than the preset difference and threshold, or the mean of the loss function values ​​corresponding to each training data in the training data set during the parameter adjustment operation is less than the preset difference mean threshold.

[0132] Condition 4, before step 602, a verification data set is obtained in advance. The verification data in the verification data set includes a verification text, a verification text feature vector, and a probability of occurrence of an annotation used to characterize each candidate word appearing after the verification text. Then, the difference between the sum of the loss function values ​​of the verification data set before and after the current parameter adjustment operation of the initial language model is calculated. Condition 4 is that the difference obtained by the above calculation is less than the preset loss function difference threshold. That is, the loss function of the initial language model no longer decreases or decreases very little on the verification data set.

[0133] After step 602, the model parameters of the initial language model are optimized, and random noise is added during the optimization process, which can reduce the risk of leakage of training data information.

[0134] Step 603: determine the initial language model as a pre-trained language model.

[0135] Optional implementation (two): In the parameter adjustment operation, the following steps 6021' and 6021" may be included before step 6021:

[0136] Step 6021', performing a privacy calculation operation on the training data in the training data set.

[0137] That is, for the training data set B t The training data x in j Perform a privacy calculation operation. Here, the privacy calculation operation includes:

[0138] First, the training data set B t The training data x in j The sample text vector v in s Input the initial language model Lm and get the same as the training data x jCorresponding, for predicting training data x j Each candidate word w appears after the sample text s i The predicted probability of

[0139] Then, based on the obtained candidate words w i The predicted probability of Determine the training data x j Corresponding, used to characterize the training data x j The sample text s in the training data set B t Medium privacy level

[0140] Specifically, the applicant found that text data containing private information will not appear in large quantities in large-scale corpora. In other words, it can be considered that the probability of common text data being private text is low, so the judgment of the degree of privacy of the text can be transformed into the judgment of the degree of commonness of the text. In other words, the degree to which the text doc is a private text, i.e., the degree of privacy pri(doc), is inversely proportional to its degree of commonness ord(doc), i.e.:

[0141] Since the language model is a probabilistic generative model, given the text vector of the prefix text, it can predict the probability of the next word being the occurrence of each candidate word in the preset candidate word set. The perplexity is an indicator to measure the accuracy of the language model in predicting the text sequence. Assuming that the perplexity of the language model Lm for the text doc is ppl(doc), when the perplexity ppl(doc) is very low, it means that the language model Lm predicts a higher average probability of the text doc. In practice, language models are trained using hundreds of billions of text data, and the probability distribution of the result text predicted by the language model conforms to the distribution of the text in real language scenarios, that is, the greater the predicted probability of the result text, the more common the result text is.

[0142] In summary, the perplexity ppl(doc) of the language model Lm for the text doc is inversely proportional to the commonness ord(doc) of the text doc, that is, The following conclusion is drawn: the perplexity ppl(doc) of the text doc is proportional to the privacy pri(doc) of the text doc. Therefore, the privacy pri(doc) of the text doc can be calculated by the perplexity ppl(doc) of the text doc. That is, here, based on the obtained candidate words w i The predicted probability of Determine the training data x j Corresponding, used to characterize the training data x j The sample text s in the training data set B t Medium privacy level This can be done as follows:

[0143] First, based on the predicted occurrence probability of each candidate word, the perplexity of the initial language model predicting the sample text in the training data is determined.

[0144] Here, various perplexity calculation methods can be used to calculate the perplexity ppl(doc) of the text doc.

[0145] Assume that the text doc is the following word sequence c1,c2,…,c n , that is, doc=c1,c2,…,c n , where n is the number of characters in doc. Then the perplexity ppl(doc) of the text doc can be calculated by the following formula 4 or formula 5:

[0146]

[0147]

[0148] here, To convert c1, c2, …, c k-1 After the text vector of the text composed of is input into the language model Lm, the occurrence probability of each candidate word obtained is the same as c k The corresponding probability value.

[0149] Optionally, cross entropy may be used to calculate the perplexity ppl(doc) of the text doc in the language model Lm.

[0150] Then, the privacy degree corresponding to the training data is determined based on the perplexity of the sample text in the training data predicted by the initial language model, wherein the privacy degree corresponding to the training data is positively linearly correlated with the perplexity of the sample text in the training data predicted by the initial language model.

[0151] Here, in order to obtain the training data x j The sample text s in the training data set B t Medium privacy level The training data x can be calculated by using the above formula 4, formula 5 or cross entropy method. j The perplexity of the sample text s in the initial language model Then according to Calculate the privacy In practice, the training data x can be calculated by the normalization method shown in the following formula 6: j The sample text s in the training data set B t Medium privacy level

[0152]

[0153] in, and They are training data set B t The training data x in j The corresponding privacy The minimum and maximum values ​​in .

[0154] By adopting this optional implementation, the privacy degree of the sample text can be automatically calculated by first calculating the perplexity using the occurrence probability of the sample text in the training data, and then calculating the privacy degree through the perplexity, without the need for manual labeling, thereby reducing labor costs.

[0155] In step 6021, based on the privacy degree corresponding to each training data in the training data set, a privacy weight of the training data set is determined, which is used to characterize the overall privacy degree of each sample text in the training data set.

[0156] Here, various methods may be used to determine the privacy weight of the training data set for characterizing the overall privacy level of each sample text in the training data set based on the privacy level corresponding to each training data in the training data set.

[0157] Optionally, the mean of the privacy degrees corresponding to the training data in the training data set can be determined as the privacy weight of the training data set. In this way, when the privacy degree corresponding to the training data is between 0 and 1, the privacy weight of the obtained training data set is also between 0 and 1.

[0158] Optionally, the training data in the training data set may be first sorted in descending order according to the corresponding privacy degree. Then, the privacy degree of each training data is weighted and summed according to the preset weight coefficient corresponding to its sorting position, and the sum result is normalized to obtain the privacy weight of the training data set.

[0159] Optional implementation method (three): Based on the above optional implementation method (two), in step 6022, the gradient corresponding to each training data in the training data set is clipped to obtain the corresponding clipped gradient, which can be performed as follows:

[0160] For the training data set B t The training data x in j , perform the following gradient clipping operation:

[0161] First, various loss functions can be used to determine the loss function value L(xj ). For example, including but not limited to L1 norm, L2 norm, etc.

[0162] Secondly, based on the determined loss function value, determine the training data x j The corresponding gradient.

[0163] That is, the current model parameter θ of the initial language model Lm can be adjusted according to the determined loss function value. t Find the partial derivative to get the training data x j The gradients of the model parameters of the corresponding initial language model Specifically, it can be expressed as follows using Formula 7:

[0164]

[0165] Finally, according to the preset gradient clipping norm C and the training data x j The corresponding gradient Crop and get the same value as the training data x j The corresponding clipping gradient Specifically, it can be expressed as follows using Formula 8:

[0166]

[0167] It can be understood that when C is less than 1, the above formula can be used to convert the training data x j The corresponding clipping gradient Clip to between 0 and C, that is, clip the gradient The amplitude range is below the preset gradient clipping norm C. If C is greater than or equal to 1, the gradient is clipped and Therefore, in practice, in order to clip the gradient, C can be set to a value greater than 0 and less than 1.

[0168] Optional implementation mode (four): Based on the above optional implementation mode (three), in step 6021, the preset random noise is adjusted according to the privacy weight of the training data set to obtain the privacy-adjusted random noise, which can be performed as follows:

[0169] The preset random noise is adjusted according to the privacy weight of the training data set and the preset gradient clipping norm to obtain the privacy-adjusted random noise of the training data set. Specifically, it can be expressed as follows in formula (9):

[0170]

[0171] By adopting this optional implementation (four), the magnitudes of the clipped gradient and the privacy-adjusted random noise can be unified to between 0 and C, that is, the magnitudes are consistent, which facilitates the execution of the subsequent step 6023, that is, when the privacy-adjusted random noise is added to the average clipped gradient corresponding to the training data set, the two are at the same magnitude, which can avoid the inconsistency of their distribution.

[0172] The text generation method provided by the above-mentioned embodiment of the present disclosure first obtains a first text in a specific application scenario of a language model, then inputs the first text into a language model pre-trained using a differential privacy model training method to obtain the first occurrence probability of different candidate words in a preset candidate word set appearing after the first text; finally, based on the first occurrence probability of each candidate word, a second text corresponding to the first text in the current application scenario is generated. Since the language model is pre-trained using the differential privacy model training method, the risk of the language model leaking training data in a specific application scenario can be reduced.

[0173] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a text generation device, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0174] like Figure 7 As shown, the text generation device 700 of this embodiment includes: an acquisition unit 701, an input unit 702 and a generation unit 703. Among them, the acquisition unit 701 is configured to acquire a first text; the input unit 702 is configured to input the text vector of the first text into a language model pre-trained by a differential privacy model training method to obtain the first occurrence probability of different candidate words in a preset candidate word set appearing after the first text; the generation unit 703 is configured to generate a second text corresponding to the first text in the current application scenario based on the first occurrence probability of each candidate word.

[0175] In this embodiment, the specific processing of the acquisition unit 701, the input unit 702 and the generation unit 703 of the text generation device 700 and the technical effects thereof can be referred to in Figure 2 The relevant descriptions of step 201, step 202 and step 203 in the corresponding embodiment are not repeated here.

[0176] In some optional implementations, the apparatus 700 may further include: a presenting unit 704 configured to present the second text.

[0177] In some optional embodiments, the current application scenario may be a text input scenario or a speech recognition scenario, the first text may be an input text or a recognized text; and the generation unit 703 may be further configured to: sort each of the candidate words in order from large to small according to the first occurrence probability; generate a candidate word subset using the candidate words in the candidate words sorted within a first preset higher occurrence probability range; and form the second text based on each candidate word in the candidate word subset and the corresponding first occurrence probability.

[0178] In some optional embodiments, the current application scenario may be an auxiliary writing scenario, the first text may be an input text; and the generation unit 703 may be further configured to: concatenate the candidate word with the highest first occurrence probability among the candidate words after the first text to form a concatenated text; perform the following concatenation operations a preset number of times: input the text vector of the concatenated text into the language model to obtain the second occurrence probability of each candidate word after the concatenated text; concatenate the candidate word with the highest second occurrence probability among the candidate words after the concatenated text; and generate the second text using the text in the concatenated text except the first text.

[0179] In some optional embodiments, the current application scenario may be a question-and-answer scenario, the first text may be a question text; and the generation unit 703 may be further configured to: sort each of the candidate words in order from large to small according to a first probability of occurrence; generate a set of reply candidate words using the candidate words ranked within a second preset higher probability of occurrence among the candidate words; determine the reply text corresponding to each of the reply candidate words according to the correspondence between preset reply keywords and reply texts; and generate the second text based on the reply text corresponding to each of the reply candidate words.

[0180] In some optional embodiments, the language model may be pre-trained through the following training steps: obtaining an initial language model and at least one training data set, the training data including sample text, sample text vector and the probability of occurrence of the annotation of each candidate word; performing the following parameter adjustment operation on the training data set in the at least one training data set until a preset training end condition is met, the parameter adjustment operation including: adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise; clipping the gradient corresponding to each training data in the training data set to obtain the corresponding clipped gradient, and determining the average clipped gradient corresponding to the training data set; adding the obtained privacy-adjusted random noise to the average clipped gradient corresponding to the training data set to obtain the noise gradient corresponding to the training data set; based on the noise gradient corresponding to the training data set, using a preset gradient descent optimization algorithm to adjust the model parameters of the initial language model; determining the initial language model as the pre-trained language model.

[0181] In some optional embodiments, before adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise, the parameter adjustment operation may also include: performing the following privacy calculation operation for the training data in the training data set: inputting the sample text vector in the training data into the initial language model to obtain the predicted occurrence probability corresponding to the training data, which is used to predict the occurrence of each candidate word after the sample text in the training data; determining the privacy degree corresponding to the training data, which is used to characterize the privacy degree of the sample text in the training data in the training data set, based on the obtained predicted occurrence probability of each candidate word; and determining the privacy weight of the training data set, which is used to characterize the overall privacy degree of each sample text in the training data set, based on the privacy degree corresponding to each training data in the training data set.

[0182] In some optional embodiments, determining the privacy weight of the training data set for characterizing the overall privacy level of each sample text in the training data set based on the privacy level corresponding to each training data in the training data set may include: determining the mean of the privacy levels corresponding to each training data in the training data set as the privacy weight of the training data set.

[0183] In some optional embodiments, the clipping of the gradients corresponding to each training data in the training data set to obtain the corresponding clipped gradients may include: performing the following gradient clipping operations on the training data in the training data set: determining the loss function value between the predicted occurrence probability and the labeled occurrence probability of each candidate word corresponding to the training data; determining the gradient corresponding to the training data based on the determined loss function value; and clipping the gradient corresponding to the training data according to a preset gradient clipping norm to obtain the clipped gradient corresponding to the training data.

[0184] In some optional embodiments, adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise may include: adjusting the preset random noise according to the privacy weight of the training data set and the preset gradient clipping norm to obtain the privacy-adjusted random noise of the training data set.

[0185] In some optional embodiments, determining the privacy degree corresponding to the training data and used to characterize the degree of privacy of the sample text in the training data in the training data set based on the obtained predicted occurrence probability of each of the candidate words may include: determining the perplexity of the sample text in the training data predicted by the initial language model based on the obtained predicted occurrence probability of each of the candidate words; determining the privacy degree corresponding to the training data according to the perplexity of the sample text in the training data predicted by the initial language model, wherein the privacy degree corresponding to the training data is positively linearly correlated with the perplexity of the sample text in the training data predicted by the initial language model.

[0186] It should be noted that the implementation details and technical effects of each unit in the text generation device provided in the embodiments of the present disclosure can be referred to the description of other embodiments in the present disclosure, and will not be repeated here.

[0187] Reference below Figure 8 , which shows a schematic diagram of the structure of a computer system 800 suitable for implementing the electronic device of the present disclosure. Figure 8 The computer system 800 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0188] like Figure 8As shown, the computer system 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the computer system 800 are also stored. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0189] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication devices 809 may allow the computer system 800 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 The computer system 800 of an electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0190] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0191] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0192] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0193] The computer readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device implements the following Figure 2 The text generation method shown in the embodiment and its optional implementation manner, and / or, as Figure 3 The illustrated embodiment and its optional implementations illustrate a method for generating text.

[0194] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0195] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0196] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, an acquisition unit may also be described as a "unit for acquiring a first text".

[0197] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

Claims

1. A text generation method, comprising: Get the first text; Inputting the text vector of the first text into a language model pre-trained using a differential privacy model training method, obtaining first occurrence probabilities of different candidate words in a preset candidate word set appearing after the first text, wherein, in the process of training the language model using a training data set, the gradient corresponding to the training data in the training data set is clipped, the noise to be added is determined according to the privacy degree of the sample text in the training data set, and the noise to be added is added to the average clipped gradient corresponding to the training data set; Based on the first occurrence probability of each of the candidate words, a second text corresponding to the first text in the current application scenario is generated, The language model is pre-trained through the following training steps: Acquire an initial language model and at least one training data set, wherein the training data includes sample text, sample text vectors, and annotated occurrence probabilities of each of the candidate words; Performing a parameter adjustment operation on a training data set in the at least one training data set until a preset training end condition is met; and The initial language model is determined to be the pre-trained language model.

2. The method according to claim 1, wherein: The method further comprises: The second text is presented.

3. The method according to claim 1, wherein: The current application scenario is a text input scenario or a speech recognition scenario, and the first text is input text or recognized text.

4. The method according to claim 3, wherein: The generating, based on the first occurrence probability of each of the candidate words, a second text corresponding to the first text in the current application scenario includes: Sorting the candidate words in descending order according to the first occurrence probability; Generating a candidate word subset using the candidate words ranked in a first preset higher occurrence probability range among the candidate words; The second text is formed based on each candidate word in the candidate word subset and the corresponding first occurrence probability.

5. The method according to claim 1, wherein: The current application scenario is an assisted writing scenario, and the first text is an input text; as well as The generating, based on the first occurrence probability of each of the candidate words, a second text corresponding to the first text in the current application scenario includes: splicing the first candidate word with the highest probability of occurrence among the candidate words after the first text to form a spliced ​​text; Perform the following splicing operation for a preset number of times: input the text vector of the spliced ​​text into the language model to obtain the second occurrence probability of each candidate word after the spliced ​​text; splice the candidate word with the highest second occurrence probability among the candidate words after the spliced ​​text; The second text is generated by using the text in the concatenated text except the first text.

6. The method according to claim 1, wherein: The current application scenario is a question-and-answer scenario, and the first text is a question text; as well as The generating, based on the occurrence probability of each of the candidate words, a second text corresponding to the first text in the current application scenario includes: Sorting the candidate words in descending order according to the first occurrence probability; Generate a set of answer candidate words by using the candidate words ranked in a second preset higher probability range among the candidate words; According to the correspondence between the preset reply keywords and the reply texts, the reply texts corresponding to the candidate reply words are determined; The second text is generated based on the reply text corresponding to each of the reply candidate words.

7. The method according to claim 1, wherein the parameter adjustment operation comprises: Adjusting the preset random noise according to the privacy weight of the training data set to obtain privacy-adjusted random noise; The gradient corresponding to each training data in the training data set is clipped to obtain a corresponding clipped gradient, and an average clipped gradient corresponding to the training data set is determined; the obtained privacy-adjusted random noise is added to the average clipped gradient corresponding to the training data set to obtain a noise gradient corresponding to the training data set; based on the noise gradient corresponding to the training data set, a preset gradient descent optimization algorithm is used to adjust the model parameters of the initial language model.

8. The method according to claim 7, wherein: Before adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise, the parameter adjustment operation further includes: For the training data in the training data set, the following privacy calculation operation is performed: inputting the sample text vector in the training data into the initial language model to obtain the predicted occurrence probability corresponding to the training data, which is used to predict the occurrence of each candidate word after the sample text in the training data; determining the privacy degree corresponding to the training data, which is used to characterize the privacy degree of the sample text in the training data in the training data set, based on the obtained predicted occurrence probability of each candidate word; and Based on the privacy degree corresponding to each training data in the training data set, a privacy weight of the training data set is determined for characterizing the overall privacy degree of each sample text in the training data set.

9. The method according to claim 8, wherein: The determining, based on the privacy degree corresponding to each training data in the training data set, a privacy weight of the training data set for characterizing the overall privacy degree of each sample text in the training data set comprises: The mean of the privacy degrees corresponding to the training data in the training data set is determined as the privacy weight of the training data set.

10. The method according to claim 8, wherein: The step of clipping the gradient corresponding to each training data in the training data set to obtain a corresponding clipped gradient includes: For the training data in the training data set, the following gradient clipping operation is performed: determining the loss function value between the predicted occurrence probability and the labeled occurrence probability of each candidate word corresponding to the training data; determining the gradient corresponding to the training data based on the determined loss function value; and clipping the gradient corresponding to the training data according to a preset gradient clipping norm to obtain a clipped gradient corresponding to the training data.

11. The method according to claim 10, wherein: The step of adjusting the preset random noise according to the privacy weight of the training data set to obtain the privacy-adjusted random noise includes: The preset random noise is adjusted according to the privacy weight of the training data set and the preset gradient clipping norm to obtain the privacy-adjusted random noise of the training data set.

12. The method according to claim 8, wherein: The step of determining the privacy level corresponding to the training data and used to characterize the privacy level of the sample text in the training data in the training data set based on the predicted occurrence probability of each of the candidate words includes: Determining the perplexity of the initial language model in predicting the sample text in the training data based on the predicted occurrence probability of each of the candidate words; The privacy degree corresponding to the training data is determined according to the perplexity of the sample text in the training data predicted by the initial language model, wherein there is a positive linear correlation between the privacy degree corresponding to the training data and the perplexity of the sample text in the training data predicted by the initial language model.

13. A text generation device, comprising: An acquiring unit, configured to acquire a first text; An input unit is configured to input the text vector of the first text into a language model pre-trained using a differential privacy model training method to obtain a first occurrence probability of different candidate words in a preset candidate word set appearing after the first text, wherein, in the process of training the language model using a training data set, a gradient corresponding to the training data in the training data set is clipped, noise to be added is determined according to the privacy degree of the sample text in the training data set, and the noise to be added is added to an average clipped gradient corresponding to the training data set; a generating unit configured to generate a second text corresponding to the first text in a current application scenario based on the first occurrence probability of each of the candidate words, The language model is pre-trained through the following training steps: Acquire an initial language model and at least one training data set, wherein the training data includes sample text, sample text vectors, and annotated occurrence probabilities of each of the candidate words; Performing a parameter adjustment operation on a training data set in the at least one training data set until a preset training end condition is met; and The initial language model is determined to be the pre-trained language model.

14. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon, When the one or more programs are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 12.

15. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by one or more processors, the method according to any one of claims 1 to 12 is implemented.

Images