Similar text generation method, device, electronic device and readable storage medium
By converting the initial text into a word vector sequence and sampling hidden variables in the similar text generation model, it is directly fused into the generation process, and the posterior dissipation problem when the variational autoencoder generates similar text is solved, improving the target text and the initial The matching degree of text.
Patent Information
- Application Number
- CN202011421416.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-07
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2040-12-07
AI Technical Summary
When generating similar text, the existing variational autoencoder causes the generated target text to not match the initial text due to the posterior dissipation problem.
By converting the initial text into a word vector sequence, inputting the coding area of the similar text generation model, obtaining the implicit feature sequence and normal distribution, randomly sampling the hidden variables, and inputting them into the third feature map with the second implicit feature sequence. area, generate target text, so that hidden variables are directly fused into the generation process of each word.
It effectively improves the matching degree between the target text and the initial text, solving the problem of posterior dissipation.
Smart Images

Figure CN112528647B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of intelligent decision-making, and in particular to a similar text generation method, device, electronic device and readable storage medium. Background Art
[0002] With the development of science and technology, natural language processing is being used more and more widely in people's lives, such as using text generation models to generate similar texts.
[0003] Currently, VAE (Variational Auto-Encoder) is usually used to generate similar texts. However, the trained variational auto-encoder encounters the problem of posterior dissipation, that is, it depends on the distribution and number of samples, so that the generation process of the trained model completely ignores the VAE latent variables, so that the target text generated by the trained variational auto-encoder does not match the initial text well enough. Therefore, a similar text generation method is urgently needed to improve the matching degree between the target text and the initial text. Summary of the invention
[0004] In view of the above, it is necessary to provide a similar text generation method, which aims to improve the matching degree between the target text and the initial text.
[0005] The similar text generation method provided by the present invention comprises:
[0006] Parse the similar text generation request issued by the user based on the client, obtain the initial text carried in the similar text generation request, and convert the initial text into a word vector sequence;
[0007] Inputting the word vector sequence into the encoding area of the similar text generation model to obtain a first implicit feature sequence corresponding to the initial text, and obtaining a normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model;
[0008] Randomly sampling a latent variable from the normal distribution, inputting the latent variable into the second feature mapping area and the encoding area of the similar text generation model, and obtaining an initial latent feature corresponding to the latent variable;
[0009] Obtain a starting character preset for the similar text generation model, obtain a second latent feature sequence based on the starting character, the initial latent feature and the decoding area of the similar text generation model, input the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain a target text corresponding to the initial text.
[0010] Optionally, converting the initial text into a word vector sequence includes:
[0011] Performing word segmentation processing on the initial text to obtain a word sequence;
[0012] The word sequence is input into the word vector model to obtain a word vector sequence.
[0013] Optionally, the first feature mapping area includes a first MLP network and a second MLP network, and the first feature mapping area based on the first implicit feature sequence and the similar text generation model obtains a normal distribution corresponding to the implicit state of the initial text, including:
[0014] Inputting the first implicit feature sequence into a first MLP network in the first feature mapping area to obtain a mean value of the implicit state of the initial text;
[0015] Inputting the first latent feature sequence into the second MLP network of the first feature mapping area to obtain the variance of the latent state of the initial text;
[0016] A normal distribution corresponding to the implicit state of the initial text is determined based on the mean and the variance.
[0017] Optionally, the step of inputting the latent variable and the second latent feature sequence into a third feature mapping area of the similar text generation model to obtain a target text corresponding to the initial text comprises:
[0018] The i-th feature in the second implicit feature sequence and the latent variable are input into the third feature mapping area of the similar text generation model to generate the i-th word in the target text, and the words generated corresponding to each feature in the second implicit feature sequence are merged to obtain the target text corresponding to the initial text.
[0019] Optionally, the step of inputting the i-th feature in the second latent feature sequence and the latent variable into the third feature mapping area of the similar text generation model to generate the i-th word in the target text includes:
[0020] Inputting the i-th feature in the second latent feature sequence, the latent variable and the preset dictionary into the third feature mapping area of the similar text generation model to obtain a probability distribution table of each word in the preset dictionary at the i-th word position of the target text;
[0021] The word corresponding to the maximum probability value in the probability distribution table is used as the target word corresponding to the i-th word position in the target text.
[0022] Optionally, the probability distribution formula corresponding to the probability distribution table is:
[0023] p θ (x i ∣x <i,z)=Softmax(W z z+W H H i +b)
[0024] Among them, p θ (x i ∣x <i ,z) represents the probability distribution of each word in the preset dictionary at the i-th word position in the target text, x i represents the i-th word in the target text, x <i represents the word before the i-th word in the target text, and the parameter θ includes the decoding area parameter and bias parameters b, d z represents the variational parameter, d h represents the hidden state, |V| represents the number of words in the preset dictionary, z represents the hidden variable, H i Represents the i-th feature in the second implicit feature sequence.
[0025] In order to solve the above problem, the present invention also provides a similar text generation device, the device comprising:
[0026] A parsing module, used to parse a similar text generation request issued by a user based on a client, obtain an initial text carried in the similar text generation request, and convert the initial text into a word vector sequence;
[0027] An input module, used for inputting the word vector sequence into the encoding area of the similar text generation model to obtain a first implicit feature sequence corresponding to the initial text, and obtaining a normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model;
[0028] A sampling module, used for randomly sampling a latent variable from the normal distribution, inputting the latent variable into the second feature mapping area and the encoding area of the similar text generation model, and obtaining an initial latent feature corresponding to the latent variable;
[0029] A generation module is used to obtain a starting character preset for the similar text generation model, obtain a second latent feature sequence based on the starting character, the initial latent feature and the decoding area of the similar text generation model, and input the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain a target text corresponding to the initial text.
[0030] Optionally, the first feature mapping area includes a first MLP network and a second MLP network, and the first feature mapping area based on the first implicit feature sequence and the similar text generation model obtains a normal distribution corresponding to the implicit state of the initial text, including:
[0031] Inputting the first implicit feature sequence into a first MLP network in the first feature mapping area to obtain a mean value of the implicit state of the initial text;
[0032] Inputting the first latent feature sequence into the second MLP network of the first feature mapping area to obtain the variance of the latent state of the initial text;
[0033] A normal distribution corresponding to the implicit state of the initial text is determined based on the mean and the variance.
[0034] In order to solve the above problem, the present invention further provides an electronic device, the electronic device comprising:
[0035] at least one processor; and,
[0036] a memory communicatively connected to the at least one processor; wherein,
[0037] The memory stores a similar text generation program that can be executed by the at least one processor, and the similar text generation program is executed by the at least one processor so that the at least one processor can execute the above-mentioned similar text generation method.
[0038] In order to solve the above problem, the present invention also provides a computer-readable storage medium, on which a similar text generation program is stored. The similar text generation program can be executed by one or more processors to implement the above similar text generation method.
[0039] Compared with the prior art, the present invention first obtains the initial text carried by the similar text generation request, converts the initial text into a word vector sequence; inputs the word vector sequence into the encoding area of the similar text generation model, obtains the first implicit feature sequence corresponding to the initial text, and obtains the normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model; randomly samples a hidden variable from the normal distribution, inputs the hidden variable into the second feature mapping area and the encoding area of the similar text generation model, and obtains the initial implicit feature corresponding to the hidden variable; obtains the starting character preset for the similar text generation model, obtains the second implicit feature sequence based on the starting character, the initial implicit feature and the decoding area of the similar text generation model, and inputs the hidden variable and the second implicit feature sequence into the third feature mapping area of the similar text generation model to obtain the target text corresponding to the initial text. The present invention inputs the hidden variable and the second implicit feature sequence into the third feature mapping area of the similar text generation model together to obtain the target text corresponding to the initial text, so that the hidden variable is directly integrated into the process of generating each word in the target text, effectively solves the problem of posterior dissipation, and improves the matching degree between the initial text and the target text. Therefore, the present invention improves the matching degree between the target text and the initial text. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 A schematic diagram of a process for generating similar texts according to an embodiment of the present invention;
[0041] Figure 2 A schematic diagram of a process for generating a target text by a similar text generation model provided in one embodiment of the present invention;
[0042] Figure 3 A schematic diagram of modules of a similar text generating device provided by an embodiment of the present invention;
[0043] Figure 4 A schematic diagram of the structure of an electronic device for implementing a similar text generation method provided by an embodiment of the present invention;
[0044] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0045] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0046] It should be noted that the descriptions of "first", "second", etc. in the present invention are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0047] The present invention provides a similar text generation method. Figure 1 FIG. 1 is a flow chart of a similar text generation method provided by an embodiment of the present invention. The method can be executed by an electronic device, and the electronic device can be implemented by software and / or hardware.
[0048] In this embodiment, the similar text generation method includes:
[0049] S1. Parse a similar text generation request issued by a user based on a client, obtain an initial text carried in the similar text generation request, and convert the initial text into a word vector sequence.
[0050] The application scenario of this embodiment is: given an initial text, a target text similar to the initial text is generated, and the initial text can be a sentence, an article, or a poem.
[0051] The converting the initial text into a word vector sequence includes:
[0052] A11, performing word segmentation processing on the initial text to obtain a word sequence;
[0053] In this embodiment, a word list-based word segmentation algorithm (a forward maximum matching algorithm, a reverse maximum matching algorithm, a bidirectional maximum matching algorithm) or a word segmentation method based on an N-gram language model may be used to segment the initial text.
[0054] A12. Input the word sequence into a word vector model to obtain a word vector sequence.
[0055] The word vector model can be a word2vec model or a BERT model.
[0056] S2. Input the word vector sequence into the encoding area of the similar text generation model to obtain the first implicit feature sequence corresponding to the initial text, and obtain the normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model.
[0057] The similar text generation model includes an encoding area, a decoding area, and first, second, and third feature mapping areas. The encoding area and the decoding area are LSTM networks (Long Short-Term Memory Networks) or Transformer models, and the first, second, and third feature mapping areas are MLP networks (Multi-Layer Perceptrons).
[0058] Among them, the encoding area is used to convert the input word vector sequence into a smaller dense representation (the first latent feature sequence), the first feature mapping area is used to calculate the normal distribution corresponding to the implicit state of the initial text according to the first latent feature sequence, the second feature mapping area is used to map the latent variables sampled from the normal distribution into latent features, the decoding area is used to restore the input features, and the third feature mapping area is used to generate the target text according to the latent variables and the features output by the decoding area.
[0059] The similar text generation model is converted from a variational autoencoder. The difference from the existing variational autoencoder is that the similar text generation model adds a third feature mapping area (the existing variational autoencoder uses a fully connected layer to convert the features output by the decoding area into target text) and establishes a connection relationship between the output of the first feature mapping area and the input of the third feature mapping area.
[0060] For example, after the initial text x is segmented, the word sequence obtained has T words x=(x 1 ,x 2 ,…,x T ), the word vector sequence obtained after word vectorization is (e 1 ,e 2 ,…,e T ), input the word vector sequence into the encoding area of the model to obtain the first implicit feature sequence (h 1 ,h 2 ,…,h T )=Enc(e 1 ,e 2 ,…,e T ), where Enc() indicates the execution of encoding processing.
[0061] In this embodiment, the first feature mapping area includes a first MLP network and a second MLP network, and the first feature mapping area based on the first implicit feature sequence and the similar text generation model obtains a normal distribution corresponding to the implicit state of the initial text, including:
[0062] B11, inputting the first implicit feature sequence into the first MLP network of the first feature mapping area to obtain the mean value of the implicit state of the initial text;
[0063] B12, inputting the first implicit feature sequence into the second MLP network of the first feature mapping area to obtain the variance of the implicit state of the initial text;
[0064] B13. Determine a normal distribution corresponding to the implicit state of the initial text based on the mean and variance.
[0065] Among them, the process of determining the normal distribution corresponding to the implicit state according to the mean and variance is a prior art and will not be repeated here.
[0066] S3. Randomly sample a latent variable from the normal distribution, input the latent variable into the second feature mapping area and the encoding area of the similar text generation model, and obtain the initial latent feature corresponding to the latent variable.
[0067] For example, the latent variable randomly sampled from the normal distribution is z, and the initial latent feature obtained by z after passing through the second feature mapping area and the decoding area is H0 , the target text is based on H 0 Generated.
[0068] S4. Obtain a starting character preset for the similar text generation model, obtain a second latent feature sequence based on the starting character, the initial latent feature and the decoding area of the similar text generation model, input the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain a target text corresponding to the initial text.
[0069] For example, the predetermined starting character is X 0 (The starting character of the variational autoencoder is fixed each time it generates text. The starting character is a specific character and has nothing to do with the initial text or the hidden variable z.) 0 Perform word vectorization processing (convert through word vector model) to obtain E 0 , H 0 、E 0 Input the decoding area of the similar sample generation model to obtain H 1 The second implicit feature sequence is (H 1 ,H 2 ,…,H T ), currently generate the first feature H 1 , based on H 1 and hidden variable z to generate the first word X of the target text 1 , for X 1 Perform word vectorization processing to obtain E 1 , H 1 、E 1 Input the decoding area of the similar sample generation model to obtain H 2 , and so on, to obtain other features in the second implicit feature sequence.
[0070] In this embodiment, the step of inputting the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain the target text corresponding to the initial text includes:
[0071] The i-th feature in the second implicit feature sequence and the latent variable are input into the third feature mapping area of the similar text generation model to generate the i-th word in the target text, and the words generated corresponding to each feature in the second implicit feature sequence are merged to obtain the target text corresponding to the initial text.
[0072] In this embodiment, the step of inputting the i-th feature in the second latent feature sequence and the latent variable into the third feature mapping area of the similar text generation model to generate the i-th word in the target text includes:
[0073] C11, inputting the i-th feature in the second latent feature sequence, the latent variable and the preset dictionary into the third feature mapping area of the similar text generation model, and obtaining a probability distribution table of each word in the preset dictionary at the i-th word position of the target text;
[0074] C12. Use the word corresponding to the maximum probability value in the probability distribution table as the target word corresponding to the i-th word position in the target text.
[0075] The formula for the probability distribution corresponding to the probability distribution table is:
[0076] p θ (x i ∣x <i ,z)=Softmax(W z z+W H H i +b)
[0077] Among them, p θ (x i ∣x <i ,z) represents the probability distribution of each word in the preset dictionary at the i-th word position in the target text, x i represents the i-th word in the target text, x <i represents the word before the i-th word in the target text, and the parameter θ includes the decoding area parameter and bias parameters b, d z represents the variational parameter, d h represents the hidden state, |V| represents the number of words in the preset dictionary, z represents the hidden variable, H i Represents the i-th feature in the second implicit feature sequence.
[0078] In the prior art, when a variational autoencoder is applied, the second latent feature sequence is input into the fully connected layer to generate the target text, without considering the influence of the latent variable on the generation result. The present invention inputs the latent variable together with the second latent feature sequence into the third feature mapping area for mapping and merging, and then generates the target text, thereby realizing the direct integration of the latent variable into the process of generating each word of the target text, effectively solving the problem of posterior dissipation, and making the generated target text more matched with the initial text.
[0079] In this embodiment, the loss function The gradient function is:
[0080]
[0081] Among them, g(p θ (H,z)) is the loss function In p θThe above formula clearly shows that, unlike the existing variational autoencoders, the present invention retains W z To control the role of the hidden variable z. If there is no W z , when the text sequence is long, it is easy to reduce This is consistent with the fact that "variational autoencoders tend to remember the first word and the length of sentences". Through the improvement of the present invention, the problem of gradient dissipation can be effectively reduced.
[0082] As can be seen from the above embodiments, the similar text generation method proposed by the present invention first obtains the initial text carried by the similar text generation request, and converts the initial text into a word vector sequence; inputs the word vector sequence into the encoding area of the similar text generation model to obtain the first implicit feature sequence corresponding to the initial text, and obtains the normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model; randomly samples a hidden variable from the normal distribution, inputs the hidden variable into the second feature mapping area and the encoding area of the similar text generation model, and obtains the initial implicit feature corresponding to the hidden variable; obtains the starting character preset for the similar text generation model, obtains the second implicit feature sequence based on the starting character, the initial implicit feature and the decoding area of the similar text generation model, and inputs the hidden variable and the second implicit feature sequence into the third feature mapping area of the similar text generation model to obtain the target text corresponding to the initial text. The present invention inputs the hidden variable and the second implicit feature sequence into the third feature mapping area of the similar text generation model together to obtain the target text corresponding to the initial text, so that the hidden variable is directly integrated into the process of generating each word in the target text, effectively solves the problem of posterior dissipation, and improves the matching degree between the initial text and the target text. Therefore, the present invention improves the matching degree between the target text and the initial text.
[0083] like Figure 2 , which is a schematic diagram of a process of generating a target text using a similar text generation model provided in one embodiment of the present invention.
[0084] In this embodiment, there are T words x=(x 1 ,x 2 ,…,x T ), the word sequence (x 1 ,x 2 ,…,x T ) Input the word vector model, and the resulting word vector sequence is (e 1 ,e 2 ,…,e T ), input the word vector sequence into the encoding area of the similar text generation model, and the first implicit feature sequence obtained is (h 1 ,h 2 ,…,hT ).
[0085] The first latent feature sequence (h 1 ,h 2 ,…,h T ) inputs the first feature mapping area, obtains the normal distribution corresponding to the implicit state of the initial text, randomly samples a hidden variable z from the normal distribution, and z obtains the initial implicit feature H through the second feature mapping area and the decoding area of the similar text generation model 0 .
[0086] Get the starting character X corresponding to the similar text generation model 0 , for X 0 Perform word vectorization processing (convert through word vector model) to obtain E 0 , H 0 、E 0 Input the decoding area of the similar sample generation model to obtain H 1 , based on H 1 and hidden variable z to generate the first word X of the target text 1 , for X 1 Perform word vectorization processing to obtain E 1 , H 1 、E 1 Input the decoding area of the similar sample generation model to obtain H 2 , based on H 2 and hidden variable z to generate the first word X of the target text 2 , and so on, we get the second implicit feature sequence (H 1 ,H 2 ,…,H T ) and the target text X=(X 1 ,X 2 ,…,X T ) in the English translation.
[0087] like Figure 3 , which is a module diagram of a similar text generating device provided in one embodiment of the present invention.
[0088] The similar text generation device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the similar text generation device 100 can include a parsing module 110, an input module 120, a sampling module 130 and a generation module 140. The module of the present invention can also be called a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, which are stored in the memory of the electronic device.
[0089] In this embodiment, the functions of each module / unit are as follows:
[0090] The parsing module 110 is used to parse the similar text generation request issued by the user based on the client, obtain the initial text carried in the similar text generation request, and convert the initial text into a word vector sequence.
[0091] The application scenario of this embodiment is: given an initial text, a target text similar to the initial text is generated, and the initial text can be a sentence, an article, or a poem.
[0092] The converting the initial text into a word vector sequence includes:
[0093] A21, performing word segmentation processing on the initial text to obtain a word sequence;
[0094] In this embodiment, a word list-based word segmentation algorithm (a forward maximum matching algorithm, a reverse maximum matching algorithm, a bidirectional maximum matching algorithm) or a word segmentation method based on an N-gram language model may be used to segment the initial text.
[0095] A22. Input the word sequence into a word vector model to obtain a word vector sequence.
[0096] The word vector model can be a word2vec model or a BERT model.
[0097] The input module 120 is used to input the word vector sequence into the encoding area of the similar text generation model to obtain the first implicit feature sequence corresponding to the initial text, and obtain the normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model.
[0098] The similar text generation model includes an encoding area, a decoding area, and first, second, and third feature mapping areas. The encoding area and the decoding area are LSTM networks (Long Short-Term Memory Networks) or Transformer models, and the first, second, and third feature mapping areas are MLP networks (Multi-Layer Perceptrons).
[0099] Among them, the encoding area is used to convert the input word vector sequence into a smaller dense representation (the first latent feature sequence), the first feature mapping area is used to calculate the normal distribution corresponding to the implicit state of the initial text according to the first latent feature sequence, the second feature mapping area is used to map the latent variables sampled from the normal distribution into latent features, the decoding area is used to restore the input features, and the third feature mapping area is used to generate the target text according to the latent variables and the features output by the decoding area.
[0100] The similar text generation model is converted from a variational autoencoder. The difference from the existing variational autoencoder is that the similar text generation model adds a third feature mapping area (the existing variational autoencoder uses a fully connected layer to convert the features output by the decoding area into target text) and establishes a connection relationship between the output of the first feature mapping area and the input of the third feature mapping area.
[0101] In this embodiment, there are T words x=(x 1 ,x 2 ,…,x T ), the word vector sequence obtained after word vectorization is (e 1 ,e 2 ,…,e T ), input the word vector sequence into the encoding area of the model to obtain the first implicit feature sequence (h 1 ,h 2 ,…,h T )=Enc(e 1 ,e 2 ,…,e T ), where Enc() indicates the execution of encoding processing.
[0102] In this embodiment, the first feature mapping area includes a first MLP network and a second MLP network, and the first feature mapping area based on the first implicit feature sequence and the similar text generation model obtains a normal distribution corresponding to the implicit state of the initial text, including:
[0103] B21, inputting the first implicit feature sequence into the first MLP network of the first feature mapping area to obtain the mean value of the implicit state of the initial text;
[0104] B22, inputting the first implicit feature sequence into the second MLP network of the first feature mapping area to obtain the variance of the implicit state of the initial text;
[0105] B23. Determine a normal distribution corresponding to the implicit state of the initial text based on the mean and variance.
[0106] Among them, the process of determining the normal distribution corresponding to the implicit state according to the mean and variance is a prior art and will not be repeated here.
[0107] The sampling module 130 is used to randomly sample a latent variable from the normal distribution, input the latent variable into the second feature mapping area and the encoding area of the similar text generation model, and obtain the initial latent feature corresponding to the latent variable.
[0108] For example, the latent variable randomly sampled from the normal distribution is z, and the initial latent feature obtained by z after passing through the second feature mapping area and the decoding area is H 0 , the target text is based on H 0 Generated.
[0109] The generation module 140 is used to obtain the starting character preset for the similar text generation model, obtain a second latent feature sequence based on the starting character, the initial latent feature and the decoding area of the similar text generation model, and input the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain the target text corresponding to the initial text.
[0110] For example, the predetermined starting character is X 0 (The starting character of the variational autoencoder is fixed each time it generates text. The starting character is a specific character and has nothing to do with the initial text or the hidden variable z.) 0 Perform word vectorization processing (convert through word vector model) to obtain E 0 , H 0 、E 0 Input the decoding area of the similar sample generation model to obtain H 1 The second implicit feature sequence is (H 1 ,H 2 ,…,H T ), currently generate the first feature H 1 , based on H 1 and hidden variable z to generate the first word X of the target text 1 , for X 1 Perform word vectorization processing to obtain E 1 , H 1 、E 1 Input the decoding area of the similar sample generation model to obtain H 2 , and so on, to obtain other features in the second implicit feature sequence.
[0111] In this embodiment, the step of inputting the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain the target text corresponding to the initial text includes:
[0112] The i-th feature in the second implicit feature sequence and the latent variable are input into the third feature mapping area of the similar text generation model to generate the i-th word in the target text, and the words generated corresponding to each feature in the second implicit feature sequence are merged to obtain the target text corresponding to the initial text.
[0113] In this embodiment, the step of inputting the i-th feature in the second latent feature sequence and the latent variable into the third feature mapping area of the similar text generation model to generate the i-th word in the target text includes:
[0114] C21, inputting the i-th feature in the second latent feature sequence, the latent variable and the preset dictionary into the third feature mapping area of the similar text generation model, and obtaining a probability distribution table of each word in the preset dictionary at the i-th word position of the target text;
[0115] C22. Use the word corresponding to the maximum probability value in the probability distribution table as the target word corresponding to the i-th word position in the target text.
[0116] The formula for the probability distribution corresponding to the probability distribution table is:
[0117] p θ (x i ∣x <i ,z)=Softmax(W z z+W H H i +b)
[0118] Among them, p θ (x i ∣x <i ,z) represents the probability distribution of each word in the preset dictionary at the i-th word position in the target text, x i represents the i-th word in the target text, x <i represents the word before the i-th word in the target text, and the parameter θ includes the decoding area parameter and bias parameters b, d z represents the variational parameter, d h represents the hidden state, |V| represents the number of words in the preset dictionary, z represents the hidden variable, H i Represents the i-th feature in the second implicit feature sequence.
[0119] In the prior art, when a variational autoencoder is applied, the second latent feature sequence is input into the fully connected layer to generate the target text, without considering the influence of the latent variable on the generation result. The present invention inputs the latent variable together with the second latent feature sequence into the third feature mapping area for mapping and merging, and then generates the target text, thereby realizing the direct integration of the latent variable into the process of generating each word of the target text, effectively solving the problem of posterior dissipation, and making the generated target text more matched with the initial text.
[0120] In this embodiment, the loss function The gradient function is:
[0121]
[0122] Among them, g(p θ (H,z)) is the loss function In p θ The above formula clearly shows that, unlike the existing variational autoencoders, the present invention retains W z To control the role of the hidden variable z. If there is no W z , when the text sequence is long, it is easy to reduce This is consistent with the fact that "variational autoencoders tend to remember the first word and the length of sentences". Through the improvement of the present invention, the problem of gradient dissipation can be effectively reduced.
[0123] like Figure 4 , which is a schematic diagram of the structure of an electronic device for implementing a similar text generation method provided by an embodiment of the present invention.
[0124] The electronic device 1 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The electronic device 1 can be a computer, a single network server, a server group consisting of multiple network servers, or a cloud based on cloud computing consisting of a large number of hosts or network servers, wherein cloud computing is a type of distributed computing, a super virtual computer composed of a group of loosely coupled computer sets.
[0125] In this embodiment, the electronic device 1 includes, but is not limited to, a memory 11, a processor 12, and a network interface 13 that can be interconnected and communicated via a system bus. The memory 11 stores a similar text generation program 10, and the similar text generation program 10 can be executed by the processor 12. Figure 3 Only the electronic device 1 having components 11-13 and a similar text generation program 10 is shown, and those skilled in the art will understand that Figure 3 The structure shown does not constitute a limitation on the electronic device 1 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.
[0126] Among them, the memory 11 includes a memory and at least one type of readable storage medium. The memory provides a cache for the operation of the electronic device 1; the readable storage medium may be a non-volatile storage medium such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of the electronic device 1, such as a hard disk of the electronic device 1; in other embodiments, the non-volatile storage medium may also be an external storage device of the electronic device 1, such as a plug-in hard disk equipped on the electronic device 1, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. In this embodiment, the readable storage medium of the memory 11 is generally used to store an operating system and various application software installed on the electronic device 1, such as storing the code of the similar text generation program 10 in an embodiment of the present invention. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.
[0127] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication with other devices. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the similar text generation program 10.
[0128] The network interface 13 may include a wireless network interface or a wired network interface, and the network interface 13 is used to establish a communication connection between the electronic device 1 and a client (not shown in the figure).
[0129] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.
[0130] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.
[0131] The similar text generation program 10 stored in the memory 11 of the electronic device 1 is a combination of multiple instructions. When running in the processor 12, it can achieve:
[0132] Parse the similar text generation request issued by the user based on the client, obtain the initial text carried in the similar text generation request, and convert the initial text into a word vector sequence;
[0133] Inputting the word vector sequence into the encoding area of the similar text generation model to obtain a first implicit feature sequence corresponding to the initial text, and obtaining a normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model;
[0134] Randomly sampling a latent variable from the normal distribution, inputting the latent variable into the second feature mapping area and the encoding area of the similar text generation model, and obtaining an initial latent feature corresponding to the latent variable;
[0135] Obtain a starting character preset for the similar text generation model, obtain a second latent feature sequence based on the starting character, the initial latent feature and the decoding area of the similar text generation model, input the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain a target text corresponding to the initial text.
[0136] Specifically, the specific implementation method of the processor 12 for the similar text generation program 10 can refer to Figure 1 The description of the relevant steps in the corresponding embodiment is not repeated here. It should be emphasized that in order to further ensure the privacy and security of the above-mentioned initial text and target text, the above-mentioned initial text and target text can also be stored in a node of a blockchain.
[0137] Furthermore, if the module / unit integrated in the electronic device 1 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. The computer-readable medium can be non-volatile or non-volatile. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).
[0138] The computer-readable storage medium stores a similar text generation program 10, which can be executed by one or more processors. The specific implementation of the computer-readable storage medium of the present invention is basically the same as the above-mentioned similar text generation method embodiments, and will not be described in detail here.
[0139] In the several embodiments provided by the present invention, it should be understood that the disclosed devices, apparatuses and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.
[0140] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0141] In addition, each functional module in each embodiment of the present invention may be integrated into one processing unit, each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional modules.
[0142] It is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.
[0143] Therefore, no matter from which point of view, the embodiments should be regarded as illustrative and non-restrictive, and the scope of the present invention is limited by the appended claims rather than the above description, so it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention. Any attached figure mark in the claims should not be regarded as limiting the claims involved.
[0144] The blockchain referred to in this invention is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.
[0145] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in the system claim can also be implemented by one unit or device through software or hardware. The second and other words are used to indicate names, but not to indicate any particular order.
[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention.
Claims
1. A similar text generation method, characterized in that: The method comprises: Parse the similar text generation request issued by the user based on the client, obtain the initial text carried in the similar text generation request, and convert the initial text into a word vector sequence; Inputting the word vector sequence into the encoding area of the similar text generation model to obtain a first implicit feature sequence corresponding to the initial text, and obtaining a normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model; Randomly sampling a latent variable from the normal distribution, inputting the latent variable into the second feature mapping area and the encoding area of the similar text generation model, and obtaining an initial latent feature corresponding to the latent variable; Obtaining a starting character preset for the similar text generation model, obtaining a second latent feature sequence based on the starting character, the initial latent feature and a decoding area of the similar text generation model, and inputting the latent variable and the second latent feature sequence into a third feature mapping area of the similar text generation model to obtain a target text corresponding to the initial text; The similar text generation model includes a coding area, a decoding area, a first feature mapping area, a second feature mapping area and a third feature mapping area, the coding area and the decoding area are LSTM networks or Transformer models, and the first feature mapping area, the second feature mapping area and the third feature mapping area are MLP networks; The step of inputting the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain the target text corresponding to the initial text includes: inputting the i-th feature in the second latent feature sequence and the latent variable into the third feature mapping area of the similar text generation model to generate the i-th word in the target text, and merging the words generated corresponding to each feature in the second latent feature sequence to obtain the target text corresponding to the initial text.
2. The similar text generation method according to claim 1, characterized in that: The converting the initial text into a word vector sequence includes: Performing word segmentation processing on the initial text to obtain a word sequence; The word sequence is input into the word vector model to obtain a word vector sequence.
3. The similar text generation method according to claim 1, characterized in that: The first feature mapping area includes a first MLP network and a second MLP network, and the first feature mapping area based on the first implicit feature sequence and the similar text generation model obtains a normal distribution corresponding to the implicit state of the initial text, including: Inputting the first implicit feature sequence into a first MLP network in the first feature mapping area to obtain a mean value of the implicit state of the initial text; Inputting the first latent feature sequence into the second MLP network of the first feature mapping area to obtain the variance of the latent state of the initial text; A normal distribution corresponding to the implicit state of the initial text is determined based on the mean and the variance.
4. The similar text generation method according to claim 1, characterized in that: The step of inputting the i-th feature in the second latent feature sequence and the latent variable into the third feature mapping area of the similar text generation model to generate the i-th word in the target text includes: Inputting the i-th feature in the second latent feature sequence, the latent variable and the preset dictionary into the third feature mapping area of the similar text generation model to obtain a probability distribution table of each word in the preset dictionary at the i-th word position of the target text; The word corresponding to the maximum probability value in the probability distribution table is used as the target word corresponding to the i-th word position in the target text.
5. The similar text generation method according to claim 4, characterized in that: The probability distribution formula corresponding to the probability distribution table is: in, represents the probability distribution of each word in the preset dictionary at the i-th word position in the target text, represents the i-th word in the target text, represents the word before the i-th word in the target text, and the parameter θ includes the decoding area parameter , and the bias parameter b, represents the variational parameter, Indicates the implicit state, represents the number of words in the preset dictionary, z represents the latent variable, H i Represents the i-th feature in the second implicit feature sequence.
6. A similar text generation device, used to implement the similar text generation method according to any one of claims 1 to 5, characterized in that: The device comprises: A parsing module, used to parse a similar text generation request issued by a user based on a client, obtain an initial text carried in the similar text generation request, and convert the initial text into a word vector sequence; An input module, used for inputting the word vector sequence into the encoding area of the similar text generation model to obtain a first implicit feature sequence corresponding to the initial text, and obtaining a normal distribution corresponding to the implicit state of the initial text based on the first implicit feature sequence and the first feature mapping area of the similar text generation model; A sampling module, used for randomly sampling a latent variable from the normal distribution, inputting the latent variable into the second feature mapping area and the encoding area of the similar text generation model, and obtaining an initial latent feature corresponding to the latent variable; A generation module is used to obtain a starting character preset for the similar text generation model, obtain a second latent feature sequence based on the starting character, the initial latent feature and the decoding area of the similar text generation model, and input the latent variable and the second latent feature sequence into the third feature mapping area of the similar text generation model to obtain a target text corresponding to the initial text.
7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a similar text generation program that can be executed by the at least one processor, and the similar text generation program is executed by the at least one processor so that the at least one processor can execute the similar text generation method according to any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a similar text generation program, and the similar text generation program can be executed by one or more processors to implement the similar text generation method as described in any one of claims 1 to 5.
