Synonymous character generation model training method and device, and computing equipment

Abstract

The invention provides a training method and device of a synonymous character generation model and computing equipment, and belongs to the technical field of artificial intelligence. According to the training method of the synonymous character generation model provided by the invention, the synonymous character generation model is pre-trained in an unsupervised training mode based on the sensitive word data set, so that the synonymous character generation model can learn the association rules of the characters in the sensitive words, and therefore, after doping, the sensitivity of the sensitive words can be improved. Whether the semantics of the sensitive words is changed or not is judged according to the association rules of the words in the sensitive words, and then a synonym generation model obtained through pre-training is trained in a supervised training mode on the basis of the labeled doped corpus, so that the synonym generation model can learn the rules of doping the characters into the sensitive words; according to the method, the model can obtain synonymous characters doped with other characters, and the synonymous characters and sensitive words have the same semantics, so that the obtained synonymous characters are more comprehensive, and the efficiency of obtaining the synonymous characters is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method, device, and computing equipment for a synonymous word generation model. Background Art

[0002] Synonymous words refer to words with the same meaning as a given word. Synonymous word generation methods are a type of method in the field of natural language processing and are often applied in various fields. For example, when intercepting sensitive words through a risk control system, it is necessary to generate synonyms of the above sensitive words and record the synonyms in the risk control system so that the risk control system can intercept the synonyms of the sensitive words. The synonyms of the above sensitive words include words with similar pronunciation or glyphs to the sensitive words, symbols corresponding to the sensitive words, pinyin or other languages, synonyms mixed with other characters, and combinations of the above synonyms.

[0003] At present, synonymous characters can be obtained through variant word recognition models, and the training method for this model is to use a large number of given words and the similarity between the given word and other characters in the text library for training to obtain a variant word recognition model, which is used to obtain the probability that the character to be recognized is a synonym of the given word.

[0004] Since the similarity between synonyms mixed with other characters and given words is small, the variant word recognition model obtained by the above training method has poor performance and is unable to obtain synonyms mixed with other characters. Therefore, there is an urgent need for a model training method for obtaining synonyms mixed with other characters, so that the obtained synonyms are more comprehensive, thereby improving the efficiency of obtaining synonyms. Summary of the invention

[0005] The embodiment of the present application provides a training method, device, and computing device for a synonym generation model, which can obtain synonyms mixed with other characters, making the obtained synonyms more comprehensive, thereby improving the efficiency of obtaining synonyms. The technical solution is as follows.

[0006] In a first aspect, a training method for a synonym generation model is provided, the method comprising: pre-training the synonym generation model in an unsupervised training manner based on a sensitive word data set, the sensitive word data set comprising a plurality of sensitive words; obtaining a labeled mixed corpus, the labeled mixed corpus comprising a plurality of question-answer pairs, the questions in the question-answer pairs comprising a start word, a sensitive word and a conclusion word, the answers in the question-answer pairs being a set of mixed synonyms of the sensitive words obtained based on the questions; and training the pre-trained synonym generation model in a supervised training manner based on the labeled mixed corpus.

[0007] Among them, the multiple sensitive words included in the sensitive word data set include both sensitive words that have not been transformed into synonyms and synonymous characters of sensitive words, such as characters that are similar to the pronunciation or shape of sensitive words, symbols corresponding to sensitive words, pinyin or other languages, sensitive words after Chinese characters are decomposed, and combinations of the above-mentioned synonymous characters, etc. The position of each character in the sensitive word data set remains unchanged during one iteration of the synonym generation model training process.

[0008] The embodiment of the present application provides a training method for a synonym generation model. During the training process of the model, the synonym generation model is pre-trained in an unsupervised training manner based on a sensitive word data set, so that the synonym generation model can learn the association rules of characters in sensitive words, and the association rules reflect the semantics of sensitive words, so that after doping, it can be judged whether the semantics of sensitive words have changed according to the association rules of characters in the sensitive words. Then, based on a labeled doped corpus, the pre-trained synonym generation model is trained in a supervised training manner, so that the synonym generation model can learn the rules of doping characters into sensitive words, so that the model can obtain synonyms doped with other characters, and the synonyms have the same semantics as the sensitive words, so that the obtained synonyms are more comprehensive, thereby improving the efficiency of obtaining synonyms.

[0009] In some embodiments, the above-mentioned pre-training of the synonymous character generation model in an unsupervised training manner based on the sensitive word data set includes: in each iteration process, the sensitive word data set is input into the synonymous character generation model to obtain the first probability of each character in the sensitive word data set, and the first probability refers to the probability that the character is the next character of the first number of characters in the sensitive word data set; based on the first probability of each character in the sensitive word data set, the synonymous character generation model is trained. The above-mentioned method trains the synonymous character generation model based on the first probability of each character in the sensitive word data set, so that the synonymous character generation model can learn the association rules between each character in the sensitive word data set and other characters, so that the semantics of sensitive words learned by the synonymous character generation model are more accurate.

[0010] In some embodiments, the above-mentioned inputting the sensitive word data set into the synonymous text generation model to obtain the first probability of each word in the sensitive word data set includes: inputting the sensitive word data set into the synonymous text generation model to obtain a first attention score and a second attention score, and based on the first attention score and the second attention score, obtaining the first probability of each word in the sensitive word data set through the synonymous text generation model, the first attention score is used to indicate the attention score between the word in the sensitive word data set and other words, the second attention score is used to indicate the attention score between the word in the sensitive word data set and other words under different occlusion conditions, and the attention score is used to indicate the degree of association between words. The above-mentioned process obtains the first probability through the first attention score and the second attention score, comprehensively considering the attention score of each word with other words under no occlusion and the attention score of each word with other words under different occlusion conditions, so that the first probability of each word in the sensitive word data set is more accurate, thereby making the synonymous text generation model more accurate.

[0011] In some embodiments, the training of the synonymous character generation model based on the first probability of each character in the sensitive word data set includes: taking the sum of the first probabilities of each character in the sensitive word data set as the first objective function value, and adjusting the parameters of the synonymous character generation model so that the first objective function value is greater than or equal to the first threshold. Taking the sum of the first probabilities of each character in the sensitive word data set as the basis for adjusting the parameters of the synonymous character generation model comprehensively considers the association between each character and other characters, which can make the synonymous character generation model more accurate.

[0012] In some embodiments, the above-mentioned training of the pre-trained synonym generation model in a supervised training manner based on the labeled mixed corpus includes: in each iteration process, for each sensitive word, the question-answer pair combination of the sensitive word in the labeled mixed corpus is input into the synonym generation model to obtain the second probability of each question-answer pair combination, the second probability refers to the probability that the output is the reference label of the sensitive word when the input is the question-answer pair combination, and the reference label of the sensitive word refers to the set of mixed synonyms of the sensitive word; based on the second probability of each question-answer pair combination, the synonym generation model is trained. The above-mentioned second probability reflects the similarity between the answer in the question-answer pair combination and the reference label of the sensitive word. Model training through the second probability can make the answer generated by the synonym generation model closer to the reference label, thereby making the synonym generation model more accurate.

[0013] In some embodiments, the above-mentioned inputting the question-answer pair combination of sensitive words in the labeled doped corpus into the synonymous text generation model to obtain the second probability of each question-answer pair combination includes: inputting the question-answer pair combination of sensitive words in the labeled doped corpus into the synonymous text generation model to obtain the third attention score and the fourth attention score of each question-answer pair in the question-answer pair combination, and based on the third attention score and the fourth attention score, obtaining the second probability of each question-answer pair combination through the synonymous text generation model, the third attention score is used to indicate the attention score of each word of the sensitive word in the question-answer pair, and the fourth attention score is used to indicate the attention score of each word of the answer in the question-answer pair. The above-mentioned third attention score and the fourth attention score reflect the degree of semantic similarity between the sensitive words in the question-answer pair and the doped synonymous text in the answer. Therefore, the second probability obtained based on the third attention score and the fourth attention score takes into account the semantics of the sensitive words and the doped synonymous text in the answer, and the model training based on the second probability enables the synonymous text generation model to generate doped synonymous texts with the same semantics as the sensitive words, so that the answers generated by the synonymous text generation model are more accurate.

[0014] In some embodiments, the training of the synonymous word generation model based on the second probability of each question-answer pair combination includes: taking the sum of the second probabilities of each question-answer pair combination as the second objective function, adjusting the parameters of the synonymous word generation model so that the second objective function value is greater than or equal to the second threshold. Taking the sum of the second probabilities of each question-answer pair combination as the second objective function, the similarity between the answer in each question-answer pair combination and the reference label is comprehensively considered, so that the synonymous word generation model is more accurate.

[0015] In some embodiments, the above-mentioned ending remarks are at least one of the following two ending remarks: the ending remarks of obtaining the sensitive words mixed with synonymous words, and the ending remarks of obtaining the sensitive words mixed with synonymous words under a preset mixing method.

[0016] In a second aspect, a method for generating synonymous characters is provided, the method comprising: obtaining a plurality of first doped synonymous character sets of sensitive words and the probabilities of the plurality of first doped synonymous character sets, the first doped synonymous character set being a doped synonymous character set obtained by doping sensitive words in a plurality of doping methods; obtaining a first target set from the plurality of first doped synonymous character sets, and using the doped synonymous characters in the first target set as doped synonymous characters of sensitive words, the first target set being a first doped synonymous character set in the plurality of first doped synonymous character sets whose probabilities are greater than or equal to a first probability threshold. Through the method for generating synonymous characters, the acquisition of synonymous characters doped with other characters is realized, so that the obtained synonymous characters are more comprehensive, thereby improving the efficiency of acquiring synonymous characters.

[0017] In some embodiments, the above method further includes: based on sensitive words and the doping methods of sensitive words, obtaining multiple second doped synonymous word sets and the probabilities of multiple second doped synonymous word sets of sensitive words, the second doped synonymous word set refers to the doped synonymous word set obtained by doping sensitive words according to the doping method; obtaining a second target set from multiple second doped synonymous word sets, and using the doped synonymous words in the second target set as the doped synonymous words of sensitive words in the doping method, the second target set is the second doped synonymous word set in multiple second doped synonymous word sets with a probability greater than or equal to a second probability threshold. The above synonymous word generation method realizes the acquisition of synonymous words doped with other characters in the specified doping method by specifying the doping method, and realizes the customization of synonymous words.

[0018] In a third aspect, a training device for a synonymous character generation model is provided, which is used to execute the training method for the synonymous character generation model. Specifically, the training device for a synonymous character generation model includes a functional module for executing the training method for the synonymous character generation model provided in the first aspect or any optional method of the first aspect.

[0019] In a fourth aspect, a synonymous character generation device is provided, which is used to execute the above-mentioned synonymous character generation method. Specifically, the synonymous character generation device includes a functional module for executing the synonymous character generation method provided by the above-mentioned second aspect or any optional manner of the above-mentioned second aspect.

[0020] In a fifth aspect, a computing device or a computing device cluster is provided, which includes a processor for executing program code so that the computing device or the computing device cluster executes to implement the operations performed by the training method of the synonymous word generation model or the synonymous word generation method as described above.

[0021] In a sixth aspect, a computer-readable storage medium is provided, in which at least one program code is stored. The program code is read by a processor to enable a computing device to execute operations such as those performed by the training method of the synonym generation model or the synonym generation method described above.

[0022] In the seventh aspect, a computer program product or a computer program is provided, which includes a program code, and the program code is stored in a computer-readable storage medium. A processor of a computing device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device executes the methods provided in various optional implementations of the first and second aspects above.

[0023] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0025] Figure 2 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0026] Figure 3 It is a structural diagram of a text content supervision and risk control system provided by an embodiment of the present application;

[0027] Figure 4 It is a flowchart of a training method for a synonymous character generation model provided in an embodiment of the present application;

[0028] Figure 5 It is a structural schematic diagram of a synonymous character generation model provided in an embodiment of the present application;

[0029] Figure 6 It is a flowchart of another method for training a synonymous character generation model provided in an embodiment of the present application;

[0030] Figure 7 It is a structural diagram of a multi-head attention encoding and decoding module provided in an embodiment of the present application;

[0031] Figure 8 It is a flowchart of another method for training a synonymous character generation model provided in an embodiment of the present application;

[0032] Fig. 9 It is a structural schematic diagram of a training device for a synonymous character generation model provided in an embodiment of the present application;

[0033] Fig.10 It is a structural schematic diagram of a synonymous character generation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0035] Figure 1 1 is a schematic diagram of a computing device provided in an embodiment of the present application. It should be understood that the computing device described below can implement any function in any of the following methods. Generally, the computing device 100 includes: a processor 101 and a memory 102.

[0036] The processor 101 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 101 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 101 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 101 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 101 may also include an artificial intelligence (AI) processor, which is used to process computing operations related to machine learning.

[0037] The memory 102 may include one or more computer-readable storage media, which may be non-transitory. The memory 102 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 102 is used to store at least one program, which is used to be executed by the processor 101 to implement the training method of the synonymous character generation model or the synonymous character generation method provided in the method embodiment of the present application.

[0038] In some embodiments, the computing device 100 may further include: a peripheral device interface 103 and at least one peripheral device. The processor 101, the memory 102 and the peripheral device interface 103 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 103 via a bus, a signal line or a circuit board.

[0039] In the embodiment of the present application, the computing device 100 can be configured as a terminal or a server, and the terminal or the server serves as an execution subject to implement the technical solution provided in the embodiment of the present application.

[0040] In some embodiments, the computing device 100 may be a portable mobile terminal, such as a smart phone, a tablet computer, a moving picture experts group audio layer III (MP3) player, a moving picture experts group audio layer IV (MP4) player, a laptop computer or a desktop computer. The computing device 100 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal or other names.

[0041] In some embodiments, computing device 100 may be a separate physical server, or may be implemented as Figure 2 The computing device cluster shown is a server cluster or distributed file system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Taking the computing device as a cloud server as an example, the computing device can also be called a cloud platform (i.e., the abbreviation of cloud computing platform), which refers to services based on hardware resources and software resources, providing computing, network and storage capabilities. Through the network "cloud", the huge data computing processing is processed and analyzed at the remote end and then returned to the user, which has the characteristics of large-scale, distributed, virtualized, high availability, scalability, on-demand service and security. The cloud platform can realize the rapid issuance and release of configurable computing resources with a small management cost or a low interaction complexity between users and service providers.

[0042] Figure 3 1 is a structural diagram of a text content supervision and risk control system provided according to an embodiment of the present application, and the text content supervision and risk control system is used to apply the training method or synonymous word generation method of the synonymous word generation model provided in the embodiment of the present application. Figure 3As shown, the text content supervision and risk control system includes a text content module, a sensitive word data set module, a labeled mixed corpus module, a mixed synonymous text generation module, a risk control interception module and a text content publishing module. Among them, the text content module contains the text content that needs to be reviewed and supervised by the above-mentioned text content supervision and risk control system. These text contents include text messages, blog texts, etc., which may include text content without sensitive words, and may also include text content containing sensitive words. The sensitive word data set module contains multiple sensitive words. These sensitive words are extracted and accumulated from past text content by the text content supervision and risk control system, including both sensitive words that have not been synonymously transformed and synonymous texts of sensitive words obtained after synonymous transformation. The labeled doped corpus module includes a sensitive word part, an impurity part, and a question-answer pair part. The sensitive word part includes multiple sensitive words, which can be sensitive words in the sensitive word dataset module or obtained by other methods. The impurity part includes multiple visible characters, which are all visible characters in encodings such as Unicode transformation format (UTF-8 / 16 / 32) and Chinese internal code specification (GBK), including numbers, English, punctuation marks, etc., which are used to be doped into sensitive words. The question-answer pair part includes multiple question-answer pairs obtained based on the sensitive word part and the impurity part. The questions in the question-answer pair include starting words, sensitive words, and ending words. The answers in the question-answer pair are a set of doped synonymous characters of sensitive words obtained based on the questions. The questions in multiple question-answer pairs may be the same. The doped synonym generation module is used to train the synonym generation model based on the sensitive word data set module and the labeled doped corpus module, and to summarize and infer the doping method according to the training method of the synonym generation model provided in the embodiment of the present application, and to obtain the doped synonyms of sensitive words through the model in a question-and-answer manner. The above-mentioned question-and-answer manner refers to inputting questions into the model, and the model outputs answers based on the questions. The risk control interception module is a module with enhanced interception capability obtained through soft transformation, which is used to record the doped synonyms generated by the above-mentioned doped synonym generation module, and intercept text content based on the recorded doped synonyms. The soft transformation includes updating the sensitive words and synonyms stored in the risk control interception module, upgrading the interception mechanism, etc. The text content publishing module is used to publish text content that has not been intercepted by the risk control interception module, and the text content intercepted by the risk control interception module will not be published through the text content publishing module.

[0043] Figure 4 is a flowchart of a training method for a synonymous character generation model provided in an embodiment of the present application, such as Figure 4As shown, the following is an example of a method for training the above-mentioned synonymous character generation model executed by a server, and the method includes the following steps:

[0044] 401. The server pre-trains a synonym generation model in an unsupervised training manner based on a sensitive word dataset, where the sensitive word dataset includes multiple sensitive words.

[0045] Among them, unsupervised training refers to the use of unlabeled sample data for model training. In the embodiment of the present application, the above-mentioned sensitive word data set is the unlabeled sample data used for unsupervised training of the synonymous word generation model.

[0046] 402. The server obtains a labeled mixed corpus, which includes a plurality of question-answer pairs. The questions in the question-answer pairs include a start word, a sensitive word, and a closing word. The answers in the question-answer pairs are a set of mixed synonymous words of sensitive words obtained based on the questions.

[0047] Among them, the starting words include: "Excuse me", "Please answer", "Please output", etc. The sensitive words involved in step 402 can be sensitive words in the sensitive word data set in the above step 401, or sensitive words obtained by other methods. The ending words include at least one of the following two ending words: (1) an ending word used to indicate the acquisition of mixed synonyms, such as "mixed synonyms". Based on this type of ending words, the model can be instructed to obtain the ending words of mixed synonyms of sensitive words under multiple mixing methods based on the input questions; (2) an ending word used to indicate the mixing method, such as "mixed split words", "mixed homophones", etc. Based on this type of ending words, the model can be instructed to obtain the ending words of mixed synonyms of sensitive words under a preset mixing method based on the input questions. The starting words, sensitive words and ending words are combined into multiple questions by Cartesian product. The answers in multiple question-answer pairs of the labeled mixed corpus can be manually input or obtained by the server executing related programs, and the answers are used as labels of the questions, thereby obtaining labeled mixed corpus.

[0048] In the process of obtaining labeled mixed corpus, questions are first generated based on the starting words, sensitive words and ending words, and then answers are generated based on the questions. Since the trained synonym generation model can generate mixed synonyms, it is only necessary to generate answers for some questions in the labeled mixed corpus to obtain some question-answer pairs, use these part of the question-answer pairs to train the synonym generation model, and then use the trained synonym generation model to generate answers for other questions in the labeled mixed corpus.

[0049] 403. The server trains the pre-trained synonym generation model based on the labeled mixed corpus in a supervised training manner.

[0050] The training method of the synonym generation model provided in the embodiment of the present application is based on a sensitive word data set, and the synonym generation model is pre-trained in an unsupervised training manner, so that the synonym generation model can learn the association rules of characters in sensitive words, and the association rules reflect the semantics of sensitive words, so that after doping, it can be judged whether the semantics of sensitive words have changed according to the association rules of characters in the sensitive words, and then based on the labeled doped corpus, the pre-trained synonym generation model is trained in a supervised training manner, so that the synonym generation model can learn the rules of doping characters into sensitive words, so that the model can obtain synonyms doped with other characters, and the synonyms have the same semantics as the sensitive words, so that the obtained synonyms are more comprehensive, thereby improving the efficiency of obtaining synonyms.

[0051] The training method of the synonymous word generation model provided in the embodiment of the present application first performs unsupervised pre-training on the synonymous word generation model based on the sensitive word data set. After the pre-training is completed, the synonymous word generation model is supervisedly trained based on the labeled mixed corpus. Both the pre-training and the supervised training process involve multiple rounds of iterations. The unsupervised pre-training and supervised training of the synonymous word generation model are described below.

[0052] The unsupervised pre-training and supervised training of the above synonymous word generation model share the same synonymous word generation model structure, and the structure of the synonymous word generation model is as follows: Figure 5 is a structural diagram of a synonymous character generation model provided according to an embodiment of the present application, such as Figure 5As shown, the synonymous text generation model includes an input processing module, a multi-head attention encoding and decoding module, and an output processing module. Among them, the input processing module is used to receive a sensitive word data set or a question-answer pair combination of sensitive words in a labeled doped corpus, and perform data preprocessing on the text in the sensitive word data set or the question-answer pair in the question-answer pair combination, so as to represent the text or question-answer pair in the sensitive word data set in the form of a vector, and output it to the multi-head attention encoding and decoding module. The multi-head attention encoding and decoding module is used to calculate the attention score based on the vector output by the input processing module, and then train the model based on the attention score. In the unsupervised pre-training stage, the multi-head attention encoding and decoding module is trained based on the vector converted by the text in the sensitive word data set. In the supervised training stage, the multi-head attention encoding and decoding module that has undergone unsupervised pre-training is trained based on the vector converted by the question-answer pair. In each multi-head attention encoding and decoding module, the attention score of the vector is calculated in the absence of occlusion, and the attention score of the vector under different occlusion conditions is calculated through the multi-head attention mechanism. The obtained attention score is layer-normalized to realize the forward propagation of the model. The vector output by the multi-head attention encoder-decoder module of the last layer is linearly transformed and predicted by the fully connected layer, and then output to the output processing module. The output processing module is used to process the prediction results of the multi-head attention encoder-decoder module and convert the vector into text or question-answer pairs. Figure 5 In some embodiments, a module in an upper layer may output data to all modules in a lower layer. In some embodiments, a module in an upper layer may output data to one or more modules in a lower layer.

[0053] exist Figure 5 Under the structure of the synonym generation model shown in FIG. 1 , the unsupervised pre-training process of the synonym generation model includes multiple rounds of iterations. Figure 6 This is a flowchart of a training method for a synonymous character generation model provided in accordance with an embodiment of the present application. Figure 5 and Figure 6 Taking the server executing the training method of the above-mentioned synonymous character generation model as an example, an iterative process in the unsupervised pre-training of the synonymous character generation model is explained. Figure 6 As shown, the method includes the following steps.

[0054] 601. The server performs data preprocessing on the sensitive word data set through the input processing module of the synonymous word generation model to obtain a word vector for each word in the sensitive word data set.

[0055] The server inputs the sensitive word dataset into Figure 5In the input processing module of the synonymous word generation model, the input processing module performs word embedding and position embedding on the words in the sensitive word data set, converts the words into word embedding vectors and position embedding vectors, adds the word embedding vectors to the position embedding vectors to obtain an intermediate word vector, and normalizes the intermediate word vector using the following formula (1), thereby converting the intermediate word vector into a word vector that can be processed by the multi-head attention encoding and decoding module.

[0056] Among them, the word embedding vector is used to indicate the relative semantic neighborhood relationship between words, that is, the probability of the word appearing at the same time as other words. The position embedding vector is used to indicate the position of the word in the sensitive word, which can be an absolute position or a relative position.

[0057] h0=xw e +w p (1)

[0058] Among them, h0 is used to indicate the word vector, x is used to indicate the intermediate word vector, and w e Used to indicate the word embedding matrix, w p Used to indicate the position embedding matrix. The word embedding matrix and the position embedding matrix are used to normalize the intermediate word vectors of different words.

[0059] 602. The server obtains a first attention score and a second attention score for each character in the sensitive word data set based on the character vector of each character in the sensitive word data set through the multi-head attention encoding and decoding module of the synonymous character generation model, and outputs an updated character vector based on the first attention score and the second attention score. The first attention score is used to indicate the attention score between the character in the sensitive word data set and other characters, and the second attention score is used to indicate the attention score between the character in the sensitive word data set and other characters under different occlusion conditions.

[0060] Among them, the first attention score and the second attention score are obtained in the same way, except that the word vector used when obtaining the first attention score is the word vector under no occlusion, and the word vector used when obtaining the second attention score is the word vector under different occlusion conditions. The different occlusion conditions refer to occluding some words in the sensitive word data set according to multiple preset schemes. The preset scheme can be to occlude the nth word in the sensitive word data set, or to occlude the second preset number of words every first preset number of words. The above attention score is used to indicate the degree of association between words. If two words often appear together, the attention score between the two words is high. By obtaining the attention score of each word, the synonymous word generation model can summarize the association rules between words at different positions and other words, so that the next word can be predicted based on the words at different positions. By obtaining the second attention score, the synonymous word generation model can summarize the association rules based on the attention scores under different occlusion conditions, and comprehensively consider the association rules between words at different positions and other words and the association rules under different occlusion conditions, so that the synonymous word generation model can be more accurate. The acquisition of the attention score is explained below.

[0061] In the embodiment of the present application, the multi-head attention encoding and decoding module obtains the first attention score and the second attention score based on the word vector output by the input processing module. Figure 7 As shown, Figure 5 Each layer of the multi-head attention encoding and decoding module of the synonymous text generation model includes three weight matrices, namely w q 、w k 、w v , the weight matrix is ​​a random matrix before model training. For any layer of the multi-head attention encoding and decoding module, the word vector is respectively combined with the weight matrix w q 、w k 、w v Multiply them together. If the current layer is the first layer, the word vector used by the current layer comes from the input processing module. If the current layer is not the first layer, the word vector used by the current layer comes from the previous layer of the multi-head attention encoding and decoding module. The Queries vector, Keys vector and Values ​​vector are obtained, as shown in the following formula (2):

[0062]

[0063] Among them, Q, K, and V are used to indicate the Queries vector, Keys vector, and Values ​​vector respectively, h0 is used to indicate the word vector, and w q 、w k 、w v They are used to indicate the weight matrices required to obtain the Queries vector, Keys vector, and Values ​​vector respectively.

[0064] The above Queries vector, Keys vector and Values ​​vector are used to calculate the attention score between each word and other words in the sensitive word dataset, as shown in the following formula (3):

[0065]

[0066] Among them, head is used to indicate the attention score of the text and other texts, s is used to indicate the label of the attention score, Attention() is used to indicate the function of obtaining the attention score, softmax() is used to indicate the softmax function of this layer, Q, K, V are used to indicate the Queries vector, Keys vector and Values ​​vector respectively, T is used to indicate the transposition operation, d k Used to indicate w q and w k The row dimension of .

[0067] The attention scores obtained in the above process are concatenated and linearly transformed to obtain the updated word vector, as shown in the following formula (4):

[0068]

[0069] Among them, h l It is used to represent the word vector output by the lth layer. MultiHead Attention() and Concat() are used to indicate the functions for concatenating and linearly transforming the attention scores. Q, K, and V are used to indicate the Queries vector, the Keys vector, and the Values ​​vector, respectively. Head is used to indicate the attention score of the text and other texts. The attention score can be the first attention score or the second attention score. s is used to indicate the label of the attention score. n is used to indicate the number of layers in the multi-head attention encoding and decoding module.

[0070] In some embodiments, word masking in the process of obtaining the second attention score is performed based on the MaskedMulti-head Attention mechanism.

[0071] The above steps 601 to 602 are a possible implementation method of inputting the sensitive word dataset into the synonymous word generation model to obtain the first attention score and the second attention score.

[0072] 603. The server obtains the first probability of each word in the sensitive word data set through the multi-head attention encoding and decoding module of the synonymous word generation model based on the word vector output by the last layer of the module. The first probability refers to the probability that the word in the sensitive word data set is the next word before the first number of words in the sensitive word data set.

[0073] The first probability is obtained by the association probability distribution between the characters in the sensitive word data set, and the probability distribution is obtained by the following formula (5):

[0074]

[0075] Among them, x is used to indicate the sensitive word dataset, P(x) is used to indicate the association probability distribution between the characters in the sensitive word dataset, softmax() is used to indicate the softmax function of the last layer, and h n The word vector used to indicate the output of the last layer, w e is used to indicate the word embedding matrix, and T is used to indicate the transposition operation.

[0076] The above steps 601 to 603 are a possible implementation method of inputting the sensitive word data set into the synonymous character generation model to obtain the first probability of each character in the sensitive word data set. In this implementation method, the server inputs the sensitive word data set into the synonymous character generation model to obtain the first attention score and the second attention score. Based on the first attention score and the second attention score, the first probability of each character in the sensitive word data set is obtained through the synonymous character generation model.

[0077] 604. The server trains a synonymous word generation model based on the first probability of each word in the sensitive word data set.

[0078] In order to make the parameters of the synonymous word generation model converge in the back propagation, the sum of the first probabilities of each word in the sensitive word data set is used as the first objective function value, and the parameters of the synonymous word generation model are adjusted so that the first objective function value is greater than or equal to the first threshold. The above first objective function value is obtained by the following formula (6):

[0079] L1(x)=∑ i logP(x i |x i-k , ..., x i-1 ;θ) (6)

[0080] Among them, x is used to indicate the sensitive word dataset, i is used to indicate the position of the text in the sensitive word dataset, and x iIt is used to indicate the i-th word in the sensitive word dataset, L1(x) is used to indicate the first objective function value of the sensitive word dataset, P is used to indicate the conditional probability, k is used to indicate the number of words before the word, θ is used to indicate the parameters in the multi-head attention encoding and decoding module, P(x i |x i-k , ..., x i-1 ; θ) is used to indicate the first probability, that is, under the θ parameter, x i The probability that the represented word is the next word of the first k words in the sensitive word dataset.

[0081] In the above-mentioned process of training the synonymous word generation model, it is necessary to calculate the above-mentioned first objective function value in each iteration, and adjust the parameters in the model based on the first objective function value until the preset conditions are met. The preset conditions can be that the first objective function value obtained in a certain iteration is greater than or equal to the first threshold, or that a preset number of iterations are completed.

[0082] When the preset conditions are met, the synonym generation model has learned the association rules between sensitive words and their synonyms, that is, the word grouping method, and the pre-training is completed. The following describes the process of supervised training of the pre-trained synonym generation model based on labeled mixed corpus.

[0083] In the above Figure 5 Under the structure of the synonym generation model shown in FIG. 1 , the supervised training process of the synonym generation model includes multiple rounds of iterations. Figure 8 This is a flowchart of a training method for a synonymous character generation model provided in accordance with an embodiment of the present application. Figure 5 and Figure 8 Taking the server executing the training method of the above synonymous character generation model as an example, an iterative process in the supervised training of the synonymous character generation model is described. Figure 8 As shown, the method includes the following steps.

[0084] 800. The server obtains a labeled mixed corpus, where the labeled mixed corpus includes multiple question-answer pairs, where the questions in the question-answer pairs include start words, sensitive words, and end words, and the answers in the question-answer pairs are a set of mixed synonymous words of sensitive words obtained based on the questions.

[0085] 801. The server performs data preprocessing on the question-answer pairs and reference labels of sensitive words in the labeled mixed corpus through the input processing module of the synonym generation model to obtain the vector of each question-answer pair and the vector of each reference label in the labeled mixed corpus. The reference label of a sensitive word refers to a set of mixed synonyms of the sensitive word.

[0086] For each sensitive word, the server inputs the question-answer pair combination of the sensitive word in the label-doped corpus and the reference label of the sensitive word into Figure 5 In the input processing module of the synonymous word generation model, the input processing module performs word embedding and position embedding on the question-answer pair of sensitive words in the label-doped corpus and the reference label of the sensitive word, and converts the question-answer pair and the reference label into a word embedding vector and a position embedding vector. The word embedding vector is used to indicate the association between the parts in the question-answer pair, and the position embedding vector is used to indicate the relative position of the parts in the question-answer pair. The parts in the question-answer pair include the start word, sensitive word, end word and answer. The word embedding vector and the position embedding vector are added to obtain an intermediate vector, and the intermediate vector is normalized by the above formula (1), so as to convert the intermediate vector into a vector that can be processed by the multi-head attention encoding and decoding module.

[0087] 802. For each sensitive word, the server obtains the third attention score and the fourth attention score of each question-answer pair in the question-answer pair combination based on the vector of each question-answer pair in the labeled doped corpus through the multi-head attention encoding and decoding module of the synonymous word generation model, and outputs the updated vector based on the third attention score and the fourth attention score. The third attention score is used to indicate the attention score of each word of the sensitive word in the question-answer pair, and the fourth attention score is used to indicate the attention score of each word of the answer in the question-answer pair.

[0088] The acquisition of the third attention score and the fourth attention score is similar to the acquisition method of the first attention score, and the embodiments of the present application are not described in detail here. By comparing the third attention score and the fourth attention score, it is possible to determine whether the semantics of the sensitive word changes after being mixed with other characters, so that the synonymous word generation model can generate mixed synonymous words with the same semantics as the sensitive word.

[0089] The above steps 801 to 802 are a possible implementation method of inputting the question-answer pair combination with sensitive words in the labeled corpus into the synonymous word generation model to obtain the third attention score and the fourth attention score of each question-answer pair in the question-answer pair combination.

[0090] 803. The server obtains the second probability of each question-answer pair combination based on the vector output by the last layer of the multi-head attention encoding and decoding module of the synonymous word generation model, where the second probability refers to the probability that the output is the reference label of the sensitive word when the input is a question-answer pair combination.

[0091] The second probability is obtained by the following formula (7):

[0092]

[0093] Among them, y′ is used to indicate the reference label of the sensitive word, y1, y2, ...y m Used to indicate the question-answer pair combination, P(y′|y1,y2,...y m ) is used to indicate that when the input is y1, y2, ...y m Under the condition, the prediction result is the conditional probability of y′, and softmax() is used to indicate the softmax function of the last layer. A vector indicating the output of the last layer, w v′ Used to indicate the network parameters of the last layer that is randomly initialized based on y′.

[0094] The above steps 801 to 803 are a possible implementation method for inputting the question-answer pair combination of sensitive words in the labeled corpus into the synonymous text generation model for each sensitive word to obtain the second probability of each question-answer pair combination. In this implementation method, the server inputs the question-answer pair combination of sensitive words in the labeled corpus into the synonymous text generation model to obtain the third attention score and the fourth attention score of each question-answer pair in the question-answer pair combination, and based on the third attention score and the fourth attention score, obtains the second probability of each question-answer pair combination through the synonymous text generation model.

[0095] 804. The server trains a synonym generation model based on the second probability of each question-answer pair combination.

[0096] In order to make the parameters of the synonymous word generation model converge in the back propagation, the sum of the second probabilities of each question-answer pair combination is used as the second objective function, and the parameters of the synonymous word generation model are adjusted so that the second objective function value is greater than or equal to the second threshold. The above first objective function value is obtained by the following formula (8):

[0097] L2(y)=∑ yy′ logP(y′|y1,...,y m ;θ) (8)

[0098] Among them, L2(y) is used to indicate the second objective function value of the sensitive word, y is used to indicate the question-answer pair set of the sensitive word, y′ is used to indicate the reference label, and y m is used to indicate the mth question-answer pair in y, θ is used to indicate the parameters in the multi-head attention encoder-decoder module, P(y′|y1,...,y m ; θ) is used to indicate the second probability, that is, under the θ parameter, when the input is y1, y2, ...y m Under this condition, the predicted result is the conditional probability of y′.

[0099] In the above-mentioned process of training the synonymous word generation model, it is necessary to calculate the above-mentioned second objective function value in each iteration, and adjust the parameters in the model based on the second objective function value until the preset condition is met. The preset condition can be that the second objective function value obtained in a certain iteration is greater than or equal to the second threshold, or it can be that a preset number of iterations are completed.

[0100] When the preset conditions are met and the synonym generation model has learned the doping method, the training can be terminated.

[0101] The training method of the synonym generation model provided in the embodiment of the present application can obtain synonyms mixed with other characters, so that the obtained synonyms are more comprehensive, thereby improving the efficiency of obtaining synonyms. In the training process of the model, based on the sensitive word data set, the synonym generation model is pre-trained in an unsupervised training manner, so that the synonym generation model can learn the association rules of the characters in the sensitive words, and the association rules reflect the semantics of the sensitive words, so that after mixing, it can be judged whether the semantics of the sensitive words have changed according to the association rules of the characters in the sensitive words. Then, based on the labeled mixed corpus, the pre-trained synonym generation model is trained in a supervised training manner, so that the synonym generation model can learn the rules of mixing characters into sensitive words, so that the model can mix the blank positions of sensitive words in the form of filling in the blanks, so that the model can obtain synonyms mixed with other characters, and the synonyms have the same semantics as the sensitive words, so that the obtained synonyms are more comprehensive, thereby improving the efficiency of obtaining synonyms.

[0102] The embodiment of the present application also provides a synonymous text generation method, which obtains mixed synonymous texts of sensitive words based on the above-mentioned synonymous text generation model. According to different needs, the synonymous text generation method can generate synonymous texts of sensitive words under multiple mixed methods, or can generate synonymous texts of sensitive words under preset mixed methods. The following two implementation methods are taken as examples to illustrate the synonymous text generation method under the above-mentioned different needs.

[0103] In an embodiment of the present application, the server inputs a first question into a synonym generation model, the first question includes a start word, a sensitive word, and a first end word, the first end word is used to indicate the acquisition of mixed synonyms of the sensitive word, based on the first question and the synonym generation model, multiple first mixed synonym sets of sensitive words and the probabilities of multiple first mixed synonym sets are acquired, the first mixed synonym set refers to a mixed synonym set obtained by mixing sensitive words in multiple mixing methods, a first target set is acquired from the multiple first mixed synonym sets, the mixed synonyms in the first target set are used as mixed synonyms of the sensitive word, the first target set is a first mixed synonym set in the multiple first mixed synonym sets whose probability is greater than or equal to a first probability threshold.

[0104] In some embodiments, a second question is input into a synonym generation model, the second question includes a start word, a sensitive word, and a second end word, the second end word is used to indicate the acquisition of doped synonyms of sensitive words under a preset doping method, based on the second question and the synonym generation model, multiple second doped synonym sets of sensitive words and the probabilities of multiple second doped synonym sets are acquired, the second doped synonym set refers to a doped synonym set obtained by doping sensitive words according to a preset doping method, a second target set is acquired from the multiple second doped synonym sets, the doped synonyms in the second target set are used as doped synonyms of sensitive words under the preset doping method, and the second target set is a second doped synonym set in the multiple second doped synonym sets whose probability is greater than or equal to a second probability threshold.

[0105] The above describes the method of the embodiment of the present application, and the following describes the device of the embodiment of the present application. It should be understood that the device described below has any function of the computing device in the above method. Figures 1 to 8 The training method of the synonymous character generation model and the synonymous character generation method provided in the embodiment of the present application are described in detail. Fig. 9 and Fig.10 The following describes a training device for a synonymous character generation model and a synonymous character generation device provided according to an embodiment of the present application. It should be understood that the technical features described in the method embodiment are also applicable to the following device embodiment.

[0106] See also Fig. 9 The present application embodiment provides a training device for a synonymous character generation model, the device comprising:

[0107] A first training module 901 is used to pre-train a synonym generation model in an unsupervised training manner based on a sensitive word dataset, wherein the sensitive word dataset includes a plurality of sensitive words;

[0108] A corpus acquisition module 902 is used to acquire a labeled mixed corpus, wherein the labeled mixed corpus includes a plurality of question-answer pairs, wherein the questions in the question-answer pairs include a start word, a sensitive word, and a closing word, and the answers in the question-answer pairs are a set of mixed synonymous words of the sensitive word obtained based on the question;

[0109] The second training module 903 is used to train the pre-trained synonym generation model based on the labeled mixed corpus in a supervised training manner.

[0110] In some embodiments, the first training module 901 includes:

[0111] A first probability acquisition unit is used to input the sensitive word data set into the synonymous word generation model in each iteration process to obtain a first probability of each word in the sensitive word data set, where the first probability refers to the probability that the word is the next word of the first number of words in the sensitive word data set;

[0112] The first training unit is used to train a synonymous word generation model based on a first probability of each word in the sensitive word data set.

[0113] In some embodiments, the above-mentioned first probability acquisition unit is used to: input the sensitive word data set into the synonymous word generation model to obtain a first attention score and a second attention score, and based on the first attention score and the second attention score, obtain the first probability of each word in the sensitive word data set through the synonymous word generation model, the first attention score is used to indicate the attention score between the words in the sensitive word data set and other words, the second attention score is used to indicate the attention score between the words in the sensitive word data set and other words under different occlusion conditions, and the attention score is used to indicate the degree of association between words.

[0114] In some embodiments, the first training unit is used to: take the sum of the first probabilities of each word in the sensitive word data set as the first objective function value, adjust the parameters of the synonymous word generation model, so that the first objective function value is greater than or equal to the first threshold.

[0115] In some embodiments, the second training module 903 includes:

[0116] A second probability acquisition unit is used to input the question-answer pair combination of the sensitive word in the label-doped corpus into the synonym generation model for each sensitive word in each iteration process, and obtain the second probability of each question-answer pair combination, wherein the second probability refers to the probability that the output is the reference label of the sensitive word when the input is the question-answer pair combination, and the reference label of the sensitive word refers to the set of doped synonyms of the sensitive word;

[0117] The second training unit is used to train the synonymous word generation model based on the second probability of each question-answer pair combination.

[0118] In some embodiments, the second probability acquisition unit is used to: input the question-answer pair combination of sensitive words in the labeled corpus into the synonymous text generation model to obtain the third attention score and the fourth attention score of each question-answer pair in the question-answer pair combination; based on the third attention score and the fourth attention score, obtain the second probability of each question-answer pair combination through the synonymous text generation model, the third attention score is used to indicate the attention score of each word of the sensitive word in the question-answer pair, and the fourth attention score is used to indicate the attention score of each word of the answer in the question-answer pair.

[0119] In some embodiments, the second training unit is used to: take the sum of the second probabilities of each question-answer pair combination as the second objective function, and adjust the parameters of the synonymous word generation model so that the second objective function value is greater than or equal to a second threshold.

[0120] In some embodiments, the above-mentioned ending remarks are at least one of the following two ending remarks: the ending remarks of obtaining the sensitive words mixed with synonymous words, and the ending remarks of obtaining the sensitive words mixed with synonymous words under a preset mixing method.

[0121] It should be understood that the training device for the synonymous word generation model corresponds to the computing device in the above-mentioned method embodiment, and the modules in the device and the above-mentioned other operations and / or functions are respectively for implementing the various steps and methods implemented by the computing device in the method embodiment. The specific details can be found in the above-mentioned method embodiment. For the sake of brevity, they will not be repeated here.

[0122] See also Fig.10 , the embodiment of the present application provides a synonymous character generation device, the device comprising:

[0123] The first set probability acquisition module 1001 is used to acquire multiple first doped synonymous character sets of sensitive words and the probabilities of multiple first doped synonymous character sets, wherein the first doped synonymous character set refers to a doped synonymous character set obtained by doping sensitive words in multiple doping methods;

[0124] The first target set acquisition module 1002 is used to acquire a first target set from multiple first mixed synonymous word sets, and use the mixed synonymous words in the first target set as mixed synonymous words of sensitive words. The first target set is a first mixed synonymous word set in the multiple first mixed synonymous word sets whose probability is greater than or equal to a first probability threshold.

[0125] In some embodiments, the above apparatus further comprises:

[0126] A second set probability acquisition module, used to acquire multiple second doped synonymous character sets of sensitive words and the probabilities of multiple second doped synonymous character sets based on sensitive words and the doping methods of sensitive words, wherein the second doped synonymous character set refers to a doped synonymous character set obtained by doping sensitive words according to the doping methods;

[0127] The second target set acquisition module is used to acquire the second target set from multiple second doped synonymous word sets, and use the doped synonymous words in the second target set as the doped synonymous words of sensitive words in the doped mode. The second target set is a second doped synonymous word set in the multiple second doped synonymous word sets, whose probability is greater than or equal to the second probability threshold.

[0128] It should be understood that the synonymous word generation device corresponds to the computing device in the above-mentioned method embodiment, and the various modules in the device and the above-mentioned other operations and / or functions are respectively for implementing the various steps and methods implemented by the computing device in the method embodiment. The specific details can be found in the above-mentioned method embodiment. For the sake of brevity, they will not be repeated here.

[0129] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including a program code, and the program code can be executed by a processor in a computing device to complete the training method of the synonymous character generation model or the synonymous character generation method in the above embodiment. For example, the computer-readable storage medium is a non-temporary computer-readable storage medium, such as a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device.

[0130] An embodiment of the present application also provides a computer program product or a computer program, which includes a program code. The computer instructions are stored in a computer-readable storage medium. A processor in a computing device reads the program code from the computer-readable storage medium, and the processor executes the program code, so that the computing device executes the above-mentioned training method for a synonymous character generation model or the synonymous character generation method.

[0131] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, a component or a module, and the device may include a connected processor and a memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory, so that the chip executes the training method of the synonymous character generation model or the synonymous character generation method in the above-mentioned method embodiments.

[0132] Among them, the device, equipment, computer-readable storage medium, computer program product or chip provided in this embodiment is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0133] Through the description of the above implementation methods, those skilled in the art can understand that, for the convenience and simplicity of description, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the training method embodiment of the synonymous character generation model or the synonymous character generation method embodiment provided in the above embodiment belongs to the same concept, and its specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0134] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0135] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0136] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0137] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks or optical disks.

[0138] In the description of this application, unless otherwise specified, " / " means "or", for example, A / B can mean A or B. "And / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, "at least one" means one or more, and "plurality" means two or more. The words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not limit them to be different.

[0139] In this application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the words "exemplary" or "for example" is intended to present the related concepts in a concrete way.

[0140] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sensitive words involved in this application are obtained with full authorization.

[0141] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0142] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A training method for a synonym generation model, characterized in that: The method comprises: Based on a sensitive word dataset, a synonym generation model is pre-trained in an unsupervised training manner, wherein the sensitive word dataset includes a plurality of sensitive words; Acquire a labeled mixed corpus, wherein the labeled mixed corpus includes a plurality of question-answer pairs, wherein the questions in the question-answer pairs include a start word, a sensitive word, and a closing word, and the answers in the question-answer pairs are a set of mixed synonymous words of the sensitive word obtained based on the question; Based on the labeled mixed corpus, the pre-trained synonym generation model is trained in a supervised training manner.

2. The method according to claim 1, characterized in that: The pre-training of the synonym generation model based on the sensitive word dataset in an unsupervised training manner includes: In each iteration process, the sensitive word data set is input into the synonymous word generation model to obtain a first probability of each word in the sensitive word data set, wherein the first probability refers to a probability that the word is the next word of the first number of words in the sensitive word data set; The synonymous word generation model is trained based on the first probability of each word in the sensitive word data set.

3. The method according to claim 2, characterized in that The step of inputting the sensitive word data set into the synonymous word generation model to obtain a first probability of each word in the sensitive word data set includes: The sensitive word data set is input into the synonymous word generation model to obtain a first attention score and a second attention score. Based on the first attention score and the second attention score, a first probability of each word in the sensitive word data set is obtained through the synonymous word generation model. The first attention score is used to indicate the attention score between the words in the sensitive word data set and other words, and the second attention score is used to indicate the attention score between the words in the sensitive word data set and other words under different occlusion conditions. The attention scores are used to indicate the degree of association between words.

4. The method according to claim 2, characterized in that: The training of the synonymous character generation model based on the first probability of each character in the sensitive word data set includes: The sum of the first probabilities of each word in the sensitive word data set is used as a first objective function value, and the parameters of the synonymous word generation model are adjusted so that the first objective function value is greater than or equal to a first threshold.

5. The method according to any one of claims 1 to 4, characterized in that: The training of the pre-trained synonym generation model based on the labeled mixed corpus in a supervised training manner includes: In each iteration, for each sensitive word, the question-answer pair combination of the sensitive word in the labeled mixed corpus is input into the synonym generation model to obtain a second probability of each question-answer pair combination, wherein the second probability refers to the probability that the output is a reference label of the sensitive word when the input is the question-answer pair combination, and the reference label of the sensitive word refers to a set of mixed synonyms of the sensitive word; The synonymous word generation model is trained based on the second probability of each question-answer pair combination.

6. The method according to claim 5, characterized in that The step of inputting the question-answer pair combination of the sensitive words in the labeled mixed corpus into the synonymous word generation model to obtain the second probability of each question-answer pair combination includes: The question-answer pair combination of the sensitive words in the labeled mixed corpus is input into the synonymous character generation model to obtain a third attention score and a fourth attention score of each question-answer pair in the question-answer pair combination, and based on the third attention score and the fourth attention score, a second probability of each question-answer pair combination is obtained through the synonymous character generation model, the third attention score is used to indicate the attention score of each character of the sensitive word in the question-answer pair, and the fourth attention score is used to indicate the attention score of each character of the answer in the question-answer pair.

7. The method according to claim 5, characterized in that The training of the synonymous word generation model based on the second probability of each question-answer pair combination comprises: The sum of the second probabilities of each question-answer pair combination is used as the second objective function, and the parameters of the synonymous word generation model are adjusted so that the second objective function value is greater than or equal to a second threshold.

8. The method according to any one of claims 1 to 7, characterized in that: The ending remark is at least one of the following two ending remark: the ending remark of obtaining the sensitive word mixed with synonymous words, and the ending remark of obtaining the sensitive word mixed with synonymous words under a preset mixing method.

9. A method for generating synonymous characters, characterized in that: The method comprises: Acquire a plurality of first doped synonymous character sets of the sensitive word and the probabilities of the plurality of first doped synonymous character sets, wherein the first doped synonymous character set refers to a doped synonymous character set obtained by doping the sensitive word in a plurality of doping modes; A first target set is obtained from the multiple first mixed synonymous character sets, and the mixed synonymous characters in the first target set are used as mixed synonymous characters of the sensitive words. The first target set is a first mixed synonymous character set in the multiple first mixed synonymous character sets whose probability is greater than or equal to a first probability threshold.

10. The method according to claim 9, characterized in that The method further comprises: Based on the sensitive word and the doping method of the sensitive word, obtaining a plurality of second doped synonymous character sets of the sensitive word and the probabilities of the plurality of second doped synonymous character sets, wherein the second doped synonymous character set refers to a doped synonymous character set obtained by doping the sensitive word according to the doping method; A second target set is obtained from the multiple second doped synonymous word sets, and the doped synonymous words in the second target set are used as the doped synonymous words of the sensitive words in the doping mode, wherein the second target set is a second doped synonymous word set in the multiple second doped synonymous word sets whose probability is greater than or equal to a second probability threshold.

11. A training device for a synonym generation model, characterized in that: The device comprises: A first training module is used to pre-train a synonymous word generation model in an unsupervised training manner based on a sensitive word dataset, wherein the sensitive word dataset includes a plurality of sensitive words; A corpus acquisition module is used to acquire labeled mixed corpus, wherein the labeled mixed corpus includes a plurality of question-answer pairs, wherein the questions in the question-answer pairs include a start word, a sensitive word, and a closing word, and the answers in the question-answer pairs are a set of mixed synonymous words of the sensitive word obtained based on the question; The second training module is used to train the pre-trained synonym generation model based on the labeled mixed corpus in a supervised training manner.

12. A synonymous character generating device, characterized in that: The device comprises: A first set probability acquisition module, used to acquire a plurality of first doped synonymous character sets of the sensitive word and the probabilities of the plurality of first doped synonymous character sets, wherein the first doped synonymous character set refers to a doped synonymous character set obtained by doping the sensitive word in a plurality of doping modes; The first target set acquisition module is used to acquire a first target set from the multiple first mixed synonymous character sets, and use the mixed synonymous characters in the first target set as the mixed synonymous characters of the sensitive words. The first target set is the first mixed synonymous character set in the multiple first mixed synonymous character sets whose probability is greater than or equal to a first probability threshold.

13. A computing device, characterized in that: The computing device comprises a processor, and the processor is configured to execute program code so that the computing device performs the method according to any one of claims 1 to 10.

14. A computing device cluster, characterized in that: The method comprises a plurality of computing devices, including a processor, wherein the processor is configured to execute program code, so that the computing device cluster executes the method according to any one of claims 1 to 10.

15. A computer-readable storage medium, characterized in that: At least one program code is stored in the storage medium, and the at least one program code is read by the processor to enable the computing device to execute the method according to any one of claims 1 to 10.

Images