A method, apparatus, device and medium for generating text corpus

Generating generalized text through pre-trained text generalization model and decoding networks, the problem of poor generalized text quality in the existing technology is solved, the training effect of the NLP model and the quality of generalized text is improved, and it is suitable for personalized fields.

CN114692642BActive Publication Date: 2024-11-15BEIJING ORION STAR TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011627584.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-31
Publication Date
2024-11-15
Estimated Expiration
2040-12-31

AI Technical Summary

Technical Problem

When generating text corpus, the preset generalization rules lead to poor quality of generalized text, and the semantic information cannot be fully utilized, which affects the training effect of the NLP model.

Method used

By using pre-trained text generalization models and decoding networks, target generalization text is generated, taking into account context semantics and word order information, ensuring the accuracy and quality of generalized text.

Benefits of technology

The generated generalized text is closer to natural language, improves the accuracy of NLP model training and the quality of generalized text, and is suitable for text corpus generation in the personalized field.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114692642B_ABST
    Figure CN114692642B_ABST
Patent Text Reader

Abstract

The present invention discloses a method, device, equipment and medium for generating text corpus, which are used to solve the problem that the quality of the generalized text generated based on the existing preset generalization rules is poor, which is not conducive to the subsequent model training based on the text corpus. Since the embodiment of the present invention is based on at least one pre-trained text generalization model, a decoding network and the text corpus when generating the target generalized text corresponding to the text corpus, the obtained generalized text is closer to the natural language, and the decoding network is not only based on the first probability sequence, but also based on the decoded first character when decoding each first probability sequence, so that the second character decoded from the first probability sequence can consider the semantics of the context, word order and other information, and determine the generalized text. Subsequently, according to the generalized text obtained by decoding, it is more accurate to determine the target generalized text corresponding to the text corpus.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer processing technology, and in particular to a text corpus generation method, device, equipment and medium. Background Art

[0002] With the development of computer processing technology, more and more fields have begun to use natural language processing (NLP) models to intelligently recognize the natural language input by users, so that corresponding operations can be performed according to the user's expectations. Among them, the accuracy of the model plays a vital role in whether the user's natural language can be accurately recognized, and the accuracy of the model is generally affected by the amount of text corpus used for model training. For some more personalized fields, when training the NLP model applied to the field, since there are relatively few text corpora about the field, how to obtain a large amount of text corpora in the field to train the NLP model in the field is an urgent problem to be solved.

[0003] At present, in order to quickly generate a large amount of text corpus in a certain field, the text corpus currently saved in the field can be generalized according to the preset generalization rules. For example, the nouns, verbs and other parts of speech in any text corpus are replaced by the generalization rule of synonym replacement to obtain generalized text; between each word segment in the text corpus, the generalization rule of randomly inserting any preset word is adopted to obtain generalized text; the order of each word segment in the text corpus is randomly exchanged to obtain generalized text; each word segment in the text corpus is randomly deleted to obtain generalized text. Since this method only processes the text corpus according to the preset generalization rules and does not fully use the semantic information of the text corpus, the obtained generalized text generally has problems such as not conforming to the speaking habits of people's natural language, changing the semantics of the text corpus, and missing semantics, which makes the quality of the generalized text poor and is not conducive to the subsequent training of the NLP model in this field. Summary of the invention

[0004] The embodiments of the present invention provide a text corpus generation method, apparatus, device and medium to solve the problem that the generalized text generated based on the existing preset generalization rules is of poor quality, which is not conducive to subsequent model training based on the text corpus.

[0005] An embodiment of the present invention provides a method for generating a text corpus, the method comprising:

[0006] Obtaining, by means of at least one text generalization model, at least one first probability sequence corresponding to the text corpus to be generalized, wherein the first probability sequence includes a probability value of each pre-configured character corresponding to a character in a generalized text corresponding to the text corpus;

[0007] For each of the first probability sequences, the first probability sequence is decoded in turn through a decoding network and the decoded first character; if it is determined that the second character decoded from the first probability sequence meets a preset end condition, the decoded character sequence is determined as a generalized text;

[0008] According to the generalized text obtained by decoding, a target generalized text corresponding to the text corpus is determined.

[0009] An embodiment of the present invention provides a text corpus generation device, the device comprising:

[0010] An acquisition unit, configured to acquire, through at least one text generalization model, at least one first probability sequence corresponding to the text corpus to be generalized, wherein the first probability sequence includes a probability value of each pre-configured character corresponding to a character in a generalized text corresponding to the text corpus;

[0011] A decoding unit, configured to decode each of the first probability sequences in turn through a decoding network and the decoded first characters; if it is determined that the second character decoded from the first probability sequence satisfies a preset end condition, determine the decoded character sequence as a generalized text;

[0012] The determination unit is used to determine the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding.

[0013] An embodiment of the present invention provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of the text corpus generation method as described above when executing a computer program stored in the memory.

[0014] An embodiment of the present invention provides a computer-readable storage medium storing a computer program, which implements the steps of the text corpus generation method as described above when the computer program is executed by a processor.

[0015] Since the embodiment of the present invention generates a target generalized text corresponding to a text corpus based on at least one pre-trained text generalization model, a decoding network, and the text corpus, the acquired generalized text is closer to natural language, and when the decoding network decodes each first probability sequence, it determines the generalized text not only based on the first probability sequence but also based on the decoded first character, so that the second character decoded from the first probability sequence can consider the semantics, word order and other information of the context, and subsequently determines the target generalized text corresponding to the text corpus based on the decoded generalized text, which will be more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 A schematic diagram of a text corpus generation process provided by an embodiment of the present invention;

[0018] Figure 2 A schematic diagram of a specific text corpus generation process provided by an embodiment of the present invention;

[0019] Figure 3 A schematic diagram of the training process of each specific text generalization model provided in an embodiment of the present invention;

[0020] Figure 4 A schematic diagram of the structure of a text corpus generation device provided by an embodiment of the present invention;

[0021] Figure 5 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to improve the quality of generated generalized text, embodiments of the present invention provide a text corpus generation method, apparatus, device and medium.

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] Embodiment 1: Figure 1 A schematic diagram of a text corpus generation process provided by an embodiment of the present invention, the process includes:

[0025] S101: Obtain at least one first probability sequence corresponding to a text corpus to be generalized through at least one text generalization model, wherein the first probability sequence includes a pre-configured probability value of each character corresponding to a character in a generalized text corresponding to the text corpus.

[0026] The text corpus generation method provided by the embodiment of the present invention is applied to an electronic device, which may be an intelligent device, such as a robot, or a server.

[0027] In the embodiment of the present invention, if a text corpus in a certain field is to be generalized, it is necessary to collect some standard text corpora in the field in advance, such as text corpora that are commonly used in the field, some text corpora that can reflect the characteristics of the field, etc., and determine these standard text corpora as the text corpora to be generalized. Based on the acquired text corpora, corresponding processing is performed to obtain the target generalized text corresponding to the text corpus.

[0028] In the specific implementation process, in order to facilitate the subsequent processing of the text corpus, any acquired text corpus to be generalized is preprocessed, that is, the digital sequence corresponding to the text corpus is determined. The digital sequence corresponding to the text corpus can be determined based on the pre-configured correspondence between characters and numbers and each character contained in the text corpus.

[0029] It should be noted that in order to distinguish each character contained in the text corpus, the numbers corresponding to different characters are different.

[0030] After the processed text corpus is obtained based on the above embodiment, the processed text corpus is subsequently processed based on the text corpus generation method provided by the embodiment of the present invention.

[0031] In order to generate a target generalized text corresponding to a text corpus, in an embodiment of the present invention, at least one text generalization model is pre-trained. After the processed text corpus is obtained, that is, the digital sequence corresponding to the text corpus is obtained, the digital sequence is input into at least one pre-trained text generalization model. Through at least one pre-trained text generalization model, based on the input processed text corpus, that is, based on the input digital sequence, at least one first probability sequence corresponding to the text corpus can be obtained.

[0032] At least one first probability sequence corresponding to the text corpus is output in sequence. The first probability sequence includes a pre-configured probability value of each character corresponding to a character in the generalized text corresponding to the text corpus.

[0033] In an embodiment of the present invention, any text generalization model can be any one of a transformer model, an LSTM-copy model, and a convolutional model CNN.

[0034] S102: For each first probability sequence, decode the first probability sequence in turn through a decoding network and the decoded first character; if it is determined that the second character decoded from the first probability sequence meets a preset end condition, determine the decoded character sequence as a generalized text.

[0035] In an embodiment of the present invention, at least one first text generalization model is pre-trained to obtain at least one first probability sequence corresponding to the text corpus, and each first probability sequence is processed accordingly in turn to determine the characters contained in the generalized text corresponding to the text corpus.

[0036] In which, whenever a first probability sequence corresponding to the text corpus is obtained through at least one first text generalization model that has been pre-trained, the first probability sequence is processed accordingly. Alternatively, after at least one first probability sequence corresponding to the text corpus is obtained through at least one first text generalization model that has been pre-trained, each first probability sequence is processed accordingly in sequence according to the output order of each first probability sequence.

[0037] Specifically, the following processing is performed on each first probability sequence in turn:

[0038] The first probability sequence is decoded by a decoding network, such as a beam search, and a decoded character (referred to as the first character for the convenience of description), thereby obtaining a character decoded from the first probability sequence (referred to as the second character for the convenience of description). The decoded first character is a character decoded from each first probability sequence outputted sequentially before the first probability sequence by at least one text generalization model.

[0039] In order to determine whether the complete generalized text corresponding to the text corpus has been obtained, an end condition is preset. After decoding any first probability sequence based on the above embodiment, it is determined whether the second character decoded from the first probability sequence meets the preset end condition. If it is determined that the decoded second character meets the preset end condition, it is determined that the complete generalized text corresponding to the text corpus has been obtained; otherwise, it means that the complete generalized text corresponding to the text corpus has not been obtained, and the above processing is performed on the next first probability sequence.

[0040] It should be noted that the above operations are performed sequentially for the first probability sequence obtained in the above embodiment, so as to obtain the generalized text corresponding to the text corpus.

[0041] In a possible implementation, when the generalized text obtained by decoding satisfies a preset condition (referred to as the first condition for convenience of explanation), it is determined that each generalized text corresponding to the text corpus has been obtained. The generalized text obtained by decoding satisfies the preset first condition, including that the number of generalized texts decoded reaches a set number, then it is determined that the generalized text obtained by decoding satisfies the preset first condition, and / or, the first probability sequence output by the text generalization model for the last time has been decoded, then it is determined that the generalized text obtained by decoding satisfies the preset first condition.

[0042] S103: Determine a target generalized text corresponding to the text corpus according to the generalized text obtained by decoding.

[0043] After it is determined based on the above embodiment that the generalized text corresponding to the text corpus is obtained, based on the text corpus generation method provided by the embodiment of the present invention, the target generalized text corresponding to the text corpus is determined.

[0044] In an embodiment of the present invention, each time a generalized text corresponding to a text corpus is obtained, the generalized text may be processed accordingly to determine a target generalized text corresponding to the text corpus; or after a set number of generalized texts corresponding to the text corpus are obtained, corresponding processing may be performed based on the obtained set number of generalized texts to determine a target generalized text corresponding to the text corpus.

[0045] Specifically, when setting the set number, different values ​​can be set according to different scenarios. If you want to obtain as many generalized texts corresponding to the text corpus as possible, you can set the set number to be larger; if you have strict requirements on the quality of the generalized text corresponding to the generated text corpus, you can set the set number to be smaller. In the specific implementation process, it can be flexibly set according to actual needs, and no specific limitation is made here.

[0046] In a possible implementation, the acquired generalized text may be directly determined as the target generalized text corresponding to the text corpus.

[0047] Since the embodiment of the present invention generates a target generalized text corresponding to a text corpus based on at least one pre-trained text generalization model, a decoding network, and the text corpus, the acquired generalized text is closer to natural language, and when the decoding network decodes each first probability sequence, it determines the generalized text not only based on the first probability sequence but also based on the decoded first character, so that the second character decoded from the first probability sequence can consider the semantics, word order, and other information of the context, and subsequently determines the target generalized text corresponding to the text corpus based on the decoded generalized text, which will be more accurate.

[0048] Embodiment 2: In order to obtain the target generalized text corresponding to the text corpus, based on the above embodiment, in the embodiment of the present invention, the text corpus to be generalized is determined in the following manner:

[0049] Based on the log data of the intelligent question-answering system, the text corpus to be generalized is determined, wherein the log data includes an input question input into the intelligent question-answering system and a standard question determined by the intelligent question-answering system to match the input question; and / or, the text corpus to be generalized is determined based on the text sentence input into the semantic recognition model for semantic recognition.

[0050] In an embodiment of the present invention, in order to generalize the text corpus in a certain field, it is necessary to collect the text corpus in the field in advance as the text corpus to be generalized, so that corresponding processing can be performed based on the text corpus to be generalized, and the target generalized text corresponding to the text corpus can be determined, thereby obtaining a large amount of text corpus in the field.

[0051] Specifically, the text corpus to be generalized can be determined in the following way:

[0052] Method 1: In actual application scenarios, the log data of the intelligent question-answering system includes an input question input into the intelligent question-answering system and a standard question determined by the intelligent question-answering system to match the input question. A large amount of log data (referred to as first log data for convenience of explanation) can be obtained through the intelligent question-answering system, so that the text corpus to be generalized can be determined based on the large amount of log data.

[0053] In a possible implementation, determining the text corpus to be generalized based on the acquired first log data of the intelligent question answering system includes: determining the input question and / or standard question included in the first log data as the text corpus to be generalized.

[0054] In this method, the first log data includes input questions and standard questions, and the standard questions included in the first log data can be determined as the text corpus to be generalized, or the input questions included in the first log data can be determined as the text corpus to be generalized. Of course, both the standard questions and the input questions included in the first log data can also be determined as the text corpus to be generalized.

[0055] Method 2: In actual application scenarios, the semantic recognition model can be used to perform semantic recognition on the input text sentences, and the text corpus to be generalized can also be determined based on the text sentences input into the semantic recognition model for semantic recognition, that is, the text sentences input into the semantic recognition model for semantic recognition can be directly determined as the text corpus to be generalized.

[0056] Embodiment 3: In order to accurately obtain the target generalized text corresponding to the text corpus, based on the above embodiments, in an embodiment of the present invention, if the number of text generalization models is greater than 1, the first probability sequence is decoded through a decoding network and the decoded first character, including:

[0057] For each first probability sequence output by each text generalization model each time, determine the vector sum of the first probability sequence output by the text generalization model at the same time; and decode the vector sum through a decoding network and the decoded first character.

[0058] On the basis of the above-mentioned embodiment, any text generalization model that has been pre-trained can output at least one first probability sequence corresponding to the text corpus, and any text generalization model outputs each first probability sequence in sequence. Among them, for the first probability sequence output by each text generalization model each time, the first probability sequence output by each text generalization model at the same time is used to predict the same corresponding character in the generalized text corresponding to the text corpus. Therefore, in an embodiment of the present invention, in order to accurately determine the target generalized text corresponding to the text corpus, the first probability sequence output by each text generalization model at the same time can be combined to predict the corresponding characters in the generalized text corresponding to the text corpus. Specifically, for the first probability sequence output by each text generalization model each time, the first probability sequence output by each text generalization model at the same time is added to obtain the vector sum.

[0059] For example, the two pre-trained text generalization models are text generalization model A and text generalization model B. Through text generalization model A, the three first probability sequences corresponding to the text corpus are obtained in sequence, which are a, b, and c respectively. Through text generalization model B, the three first probability sequences corresponding to the text corpus are obtained in sequence, which are e, f, and g respectively. Then, for the first probability sequence output by each text generalization model each time, the first probability sequences output by text generalization model A and text generalization model B for the first time are determined to be a and e respectively, and the vectors and are determined based on a and e; the first probability sequences output by text generalization model A and text generalization model B for the first time are determined to be b and f respectively, and the vectors and are determined based on b and f; the first probability sequences output by text generalization model A and text generalization model B for the third time are determined to be c and g respectively, and the vectors and are determined based on c and g.

[0060] After any vector sum is obtained based on the above embodiment, the vector sum is decoded according to the decoding network and the decoded first character to obtain the second character corresponding to the vector sum. After the second character corresponding to the vector sum is decoded, it is determined whether the decoded second character meets the preset end condition, so as to determine whether a generalized text corresponding to the text corpus is obtained. Specifically, if it is determined that the decoded second character meets the preset end condition, it is determined that the complete generalized text corresponding to the text corpus has been obtained; otherwise, it means that the complete generalized text corresponding to the text corpus has not been obtained, and the above-mentioned processing is performed on the first probability sequence output by each text generalization model next time.

[0061] In a possible implementation, determining whether the second character decoded from the first probability sequence satisfies a preset end condition includes:

[0062] If the second character is a pre-configured end identifier, it is determined that the second character meets the preset end condition; and / or if the number of decoded characters reaches a set threshold, it is determined that the second character meets the preset end condition.

[0063] In the embodiment of the present invention, in order to determine whether the complete generalized text corresponding to the text corpus has been decoded, an end marker is pre-configured, for example, <eos>, &&&, etc., which are not the same as any character string or special symbol that may be contained in any generalized text. After the second character is decoded, it is determined whether the decoded second character is a pre-configured end mark. If the decoded second character is a pre-configured end mark, it means that the complete generalized text corresponding to the text corpus has been decoded, then it is determined that the decoded second character meets the preset end condition, and the text containing the decoded second character and the decoded first character is determined, and the text is determined as the generalized text corresponding to the text corpus.

[0064] In the actual application process, when the generated generalized text contains too many characters, the quality of the generalized text will generally be affected, for example, the semantics of the second half of the generalized text will deviate from the semantics of the text corpus. Based on this, in an embodiment of the present invention, a set threshold is pre-configured, and the set threshold is used to identify the maximum number of characters contained in the generalized text corresponding to the generated text corpus, and based on the set threshold, the preset end condition is determined to be that the number of characters currently decoded has reached the set threshold. In the specific implementation process, after the second character is decoded, the number of characters decoded is determined according to the second character and the decoded first character. Then the number of characters decoded is compared with the set threshold to determine whether the second character currently decoded meets the preset end condition. Specifically, if it is determined that the number of characters decoded has reached the set threshold, it means that the complete generalized text corresponding to the text corpus has been decoded, then it is determined that the decoded second character meets the preset end condition, and the text containing the decoded second character and the decoded first character is determined, and the text is determined as the generalized text corresponding to the text corpus.

[0065] In the specific implementation process, the above two preset end conditions may exist at the same time. When it is determined that the decoded second character meets any of the above preset end conditions, that is, the decoded second character is a pre-configured end identifier, or the number of decoded characters reaches the set threshold, it is determined that the decoded second character meets the preset end condition.

[0066] Based on the method in the above embodiment, a set number of generalized texts corresponding to the text corpus can be obtained. However, due to the complexity and diversity of natural language, and the possibility that the pre-configured set number is too large, among the set number of generalized texts corresponding to the obtained text corpus, there may be generalized texts that are too different from the semantics of the text corpus, which is not conducive to the subsequent determination of the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding. In general, the target generalized text corresponding to the text corpus is similar in semantics to the text corpus, and the target generalized text with similar semantics to the text corpus will contain the longer character string contained in the text corpus, while the target generalized text with dissimilar semantics to the text corpus generally does not contain the longer character string contained in the text corpus, or contains the shorter character string contained in the text corpus. Based on this, in an embodiment of the present invention, the target generalized text corresponding to the text corpus is determined based on the generalized text obtained by decoding, including:

[0067] The longest common substring between the generalized text and the text corpus is determined; if the length of the longest common substring is greater than a pre-configured length threshold, the generalized text is determined to be the target generalized text.

[0068] Specifically, the longest common substring between the generalized text corresponding to the text corpus and the text corpus is obtained, and corresponding processing is performed based on the length of the longest common substring of the generalized text to determine the target generalized text corresponding to the text corpus. The longest common substring between any generalized text corresponding to the text corpus and the text corpus represents the number of characters contained in the longest character string that exists in both the generalized text and the text corpus.

[0069] It should be noted that obtaining the longest common substring between any generalized text and the text corpus belongs to the prior art and will not be elaborated here.

[0070] In an embodiment of the present invention, after obtaining any generalized text corresponding to the text corpus, the longest common substring between the generalized text and the text corpus can be determined, and then corresponding processing can be performed based on the longest common substring of the generalized text to determine the target generalized text corresponding to the text corpus; or after obtaining a set number of generalized texts corresponding to the text corpus, the longest common substring between each generalized text and the text corpus can be determined, and then corresponding processing can be performed based on the longest common substring of each generalized text to determine the target generalized text corresponding to the text corpus.

[0071] In order to facilitate the determination of the target generalized text corresponding to the text corpus, a length threshold is pre-configured. After the longest common substring of the generalized text is determined based on the above embodiment, it is determined whether the length of the longest common substring of the generalized text is greater than the pre-configured length threshold. Specifically, if it is determined that the length of the longest common substring of the generalized text is not greater than the pre-configured length threshold, it means that the generalized text is very likely to be semantically dissimilar to the text corpus, then the generalized text is determined to be an invalid generalized text, and the invalid generalized text is deleted; if it is determined that the length of the longest common substring of the generalized text is greater than the pre-configured length threshold, it means that the generalized text is very likely to be semantically similar to the text corpus, then the generalized text is determined to be the target generalized text corresponding to the text corpus.

[0072] In actual application, the obtained generalized text may contain some non-standard characters, such as special symbols, multiple repeated symbols, etc., which do not exist in normal natural languages, and the non-standard characters contained in the generalized text will affect the quality of the obtained generalized text. Based on this, in an embodiment of the present invention, before determining the target generalized text corresponding to the text corpus based on the decoded generalized text, the method also includes: using a regular algorithm to delete characters matching any pre-configured non-standard characters from the generalized text.

[0073] In the embodiment of the present invention, in order to improve the instruction of the target generalized text corresponding to the acquired text corpus, non-standard characters are pre-configured, such as special symbols, multiple repeated symbols, etc. Then, a regular algorithm is used to delete characters matching any pre-configured non-standard characters from the acquired generalized text.

[0074] Similarly, in an embodiment of the present invention, after obtaining any generalized text, a regular algorithm is used to delete characters matching any pre-configured non-standard characters from the generalized text, or after obtaining a set number of generalized texts, a regular algorithm is used for each generalized text to delete characters matching any pre-configured non-standard characters from the generalized text. Of course, in order to reduce the workload, after determining the generalized text whose longest common substring length is greater than a pre-configured length threshold through the above embodiment, a regular algorithm is used to delete characters matching any pre-configured non-standard characters from the generalized text.

[0075] It should be noted that the specific method of using a regular algorithm to delete characters that match any pre-configured non-standard characters from the generalized text belongs to the prior art and will not be described in detail here.

[0076] Embodiment 4: In order to make the obtained generalized text more universal, based on the above embodiments, in the embodiment of the present invention, before determining the target generalized text corresponding to the text corpus according to the decoded generalized text, the method further includes:

[0077] Determine a target participle in the generalized text that matches a keyword included in the text corpus; and replace the target participle in the generalized text with the name of the slot type corresponding to the keyword matching the target participle.

[0078] In actual application scenarios, the text formats of text corpora in certain fields are roughly the same, that is, the slot types corresponding to the segmentations contained in the text corpora in certain fields are roughly the same. For example, the text corpora in the navigation field generally contain segmentations with the slot type of location, and the text corpora in the music field generally contain segmentations with the slot type of singer, etc. For these fields, the text that generally contains a certain slot type required by the field is very likely to be a generalized text of any text corpus in the field. Based on this, in order to ensure that the obtained generalized text is more universal, in an embodiment of the present invention, the keywords contained in the text corpus and the name of the slot type corresponding to each keyword are pre-set. After the generalized text is obtained, the generalized text can be processed accordingly according to the keywords contained in the text corpus and the name of the slot type corresponding to each keyword.

[0079] In the specific implementation process, each segmentation included in the generalized text is determined. For each segmentation, it is determined whether the segmentation matches any keyword included in the text corpus. If it is determined that the segmentation matches any keyword included in the text corpus, the segmentation is determined to be the target segmentation; the name of the slot type corresponding to the keyword matching the target segmentation is replaced with the target segmentation in the generalized text.

[0080] In an embodiment of the present invention, after each target participle is determined, each target participle may be replaced separately, or after any target participle is determined, the name of the slot type corresponding to the keyword matching the target participle may replace the target participle in the generalized text.

[0081] For example, for the obtained text corpus "I want to listen to Jay Chou's "Dao Xiang", the keywords in the text corpus can be set to "Jay Chou" and "Dao Xiang", and the name of the slot type corresponding to the keyword "Jay Chou" is set to {<"slot_type":"artist">}, and the name of the slot type corresponding to the keyword "Dao Xiang" is set to {<"slot_type":"song">}. After the generalized text is obtained later, if the generalized text contains the participle of "Jay Chou" or "Dao Xiang", the participle "Jay Chou" contained in the generalized text is replaced with {<"artist">}, and the participle "Dao Xiang" in the generalized text is replaced with <"song">.

[0082] For another example, the obtained text corpus is {"text":"I want to listen to Jay Chou's Nunchaku"}, and the slot information slots corresponding to the text corpus is [{"slot_type":" <song>","slot_value":"Nunchucks"},{"slot_type":" <artist>","slot_value":"Jay Chou"}. If the generalized text corresponding to the obtained text corpus is "I want to listen to nunchakus", it is determined that "nunchakus" in the generalized text matches the preset keyword "nunchakus", then in the generalized text, "nunchakus" is replaced with "song", that is, the generalized text after replacement is "I want to listen to <song>”.

[0083] In the embodiment of the present invention, the step of replacing the target word in the generalized text may be performed after any generalized text is obtained, or the step of replacing the target word in the generalized text may be performed after a set number of generalized texts are obtained. Of course, in order to reduce the workload, the step of replacing the target word in the generalized text may be performed after the regular algorithm is used to delete the characters matching any pre-configured non-standard characters from the generalized text.

[0084] Embodiment 5: In order to make the obtained generalized text closer to normal natural language and further improve the quality of the generalized text, based on the above embodiments, in the embodiment of the present invention, before determining the target generalized text corresponding to the text corpus according to the decoded generalized text, the method further includes:

[0085] Add a start character before the first character of each generalized text, and add an end character after the end character; for each pre-configured sliding window, use the sliding window to start from the start character of each generalized text, and slide with a preset sliding step until the end character of the generalized text is reached; determine the number of times the characters in the sliding window appear in each generalized text; determine the dictionary corresponding to the sliding window according to the content of each character and the number of times each character content corresponds to; starting from the first character in each generalized text except the start character, according to the dictionary corresponding to each sliding window, use a path search algorithm to obtain the optimal generalized text with a preset optimal path number and determine it as the target generalized text.

[0086] In the embodiment of the present invention, a start character, such as α, γ, etc., and an end character, such as β, θ, etc., are pre-set. After the generalized text is obtained based on the above embodiment, the start character is added before the first character of each generalized text, and the end character is added after the last character, so as to facilitate the distinction of each generalized text.

[0087] In the embodiment of the present invention, after obtaining any generalized text, the start character may be added before the first character of the generalized text, and the end character may be added after the last character. Alternatively, after obtaining a set number of generalized texts, the start character may be added before the first character of each generalized text, and the end character may be added after the last character. Of course, after replacing each target word contained in each generalized text, the start character may be added before the first character of the replaced generalized text, and the end character may be added after the last character.

[0088] In the embodiments of the present invention, different sliding windows are pre-configured, and the lengths of each sliding window are not equal, such as 1, 2, 3, etc., so as to generate dictionaries (n-gram dictionaries) corresponding to different sliding windows through different sliding windows and each generalized text. Specifically, for each pre-configured sliding window, the sliding window is used to slide from the start character of each generalized text respectively with a preset sliding step length. For example, the sliding step length is 1 until the end character of the generalized text; it is determined that when the sliding window slides each time, the character content determined by the characters in the sliding window on the generalized text is recorded, and the number of times the character content appears in each generalized text is determined; according to each recorded character content and the number of times corresponding to each character content, the dictionary corresponding to the sliding window is determined.

[0089] For example, the pre-configured sliding window is a sliding window with a length of 1, and the obtained generalized texts are "α I want to go to KFC β", "α Where is KFC β", and "α I will go to KFC β". The sliding window with a length of 1 is used to slide from the start character of each generalized text respectively with a preset sliding step length of 1 until the end character of each generalized text; it is determined that when the sliding window slides each time, the character content determined by the characters in the sliding window on the generalized text is recorded, which are "α", "I", "want", "to", "go", "to", "KFC", "β", "where", "is", "will", respectively. Among them, the number of times each character content appears in each generalized text is that "α" is 3, "I" is 2, "want" is 1, "to" is 2, "go" is 2, "to" is 3, "KFC" is 3, "β" is 3, "where" is 1, "is" is 1, "will" is 1. Then, according to "α" = 3, "I" = 2, "want" = 1, "to" = 2, "go" = 2, "to" = 3, "KFC" = 3, "β" = 3, "where" = 1, "is" = 1, "will" = 1, the dictionary of the sliding window with a length of 1 is determined.

[0090] When the dictionaries corresponding to each sliding window are obtained based on the method in the above embodiments, the first character except the start character in each generalized text is used as the start node of the path, that is, starting from the first character except the start character in each generalized text, such as "I", "KFC" above, according to the dictionaries corresponding to each sliding window respectively, and using the path search algorithm, the optimal generalized text with the preset optimal path number is obtained and determined as the target generalized text.

[0091] Specifically, according to the path search algorithm and each character content recorded in the dictionary corresponding to each sliding window, the node after the start node, that is, the character after the first character (for the convenience of description, recorded as the target character). Then, for each target character corresponding to any start node, according to the number of times each character content recorded in the dictionary corresponding to each sliding window corresponds, the number of times the start node corresponds, the number of times the character content containing the start node and the target character corresponds, and the conditional probability that the target character is the node after the start node is determined. Any target character whose conditional probability meets the preset condition is determined as the node after the start node, that is, the character after the first character, and then the node of the target character that meets the preset condition (for the convenience of description, recorded as the second condition) is determined as the start node, and then the above steps are performed until the target character that meets the preset second condition is determined to be the preset end character, and the optimal generalized text is determined according to the text containing each determined node.

[0092] Among them, satisfying the preset second condition may be that any target character whose conditional probability is greater than a preset probability threshold satisfies the preset second condition, or that among the conditional probabilities corresponding to each target character, the target character with the largest conditional probability satisfies the preset second condition.

[0093] In order to ensure the diversity of the optimal generalized text, the number of optimal paths may be pre-configured. Based on the method in the above embodiment, the optimal generalized text with the preset number of optimal paths is determined as the target generalized text.

[0094] In a possible implementation, after each optimal generalized text is determined by the above-mentioned embodiment, optimal generalized texts with a preset optimal path number are randomly determined as target generalized texts from each optimal generalized text.

[0095] In another possible implementation, for the first non-starting character in each generalized text, the number of times the first non-starting character corresponds to the character is determined according to the dictionary corresponding to each sliding window. The characters of each first non-starting character are sorted in descending order of the number of times, and only the first n characters are determined as the starting nodes. For each starting node, according to the steps in the above embodiment, the optimal generalized text containing the starting node is determined. The n optimal generalized texts obtained are all determined as target generalized texts. Among them, the value of n is equal to the preset number of optimal paths.

[0096] It should be noted that when setting the number of optimal paths, different values ​​can be set according to different scenarios. If you want to obtain as many optimal generalized texts as possible, you can set the number of optimal paths to be larger; if you have strict requirements on the quality of the obtained optimal generalized texts, you can set the number of optimal paths to be smaller. In the specific implementation process, it can be flexibly set according to actual needs, and no specific limitation is made here.

[0097] Embodiment 6: In order to filter the duplicate texts in each acquired generalized text, based on the above embodiments, in the embodiment of the present invention, before determining the target generalized text corresponding to the text corpus according to the decoded generalized text, the method further includes:

[0098] For each generalized text, the generalized text is matched with other generalized texts. If there are other generalized texts that match, the generalized text and the other generalized texts that match are deduplicated.

[0099] After the generalized text is obtained based on the above-mentioned embodiment, there may be repeated generalized texts in the currently obtained generalized text. If each generalized text is directly saved, a large amount of storage space will be wasted. Therefore, in order to save space for storing generalized texts, the repeated texts in each obtained generalized text can be deleted. In the specific implementation process, for each generalized text, the generalized text is matched with other generalized texts. If there are other generalized texts that match, it means that the generalized text is a repeated text, and the generalized text is deleted. If there are no other generalized texts that match, it means that the generalized text is not a repeated text, and the generalized text is not deleted, and the next generalized text is obtained.

[0100] Among them, the process of matching the generalized text with other generalized texts belongs to the existing technology, which can be matched through a model, or through an algorithm, of course, it can also be matched in other ways. During the specific implementation process, it can be flexibly set according to needs, and no specific limitation is made here.

[0101] In an embodiment of the present invention, after directly obtaining a set number of generalized texts, duplicate texts existing in each obtained generalized text may be deleted. Alternatively, in order to save workload, after obtaining the optimal generalized texts of a preset optimal path number through the above embodiment, duplicate texts existing in each obtained optimal generalized text may be deleted.

[0102] In order to make the generalized text corresponding to the acquired text corpus closer to normal natural language, before determining the target generalized text corresponding to the text corpus based on the decoded generalized text, the method also includes: removing repeated phrases continuously included in the generalized text.

[0103] In actual application, multiple consecutively repeated phrases generally do not appear in a sentence in normal natural language. Therefore, in the embodiment of the present invention, the content in any generalized text can be deduplicated, that is, the repeated phrases continuously included in the generalized text can be deduplicated.

[0104] The repeated phrase contains at least one character.

[0105] In a possible implementation, the generalized text is segmented according to a preset segmentation method to determine each segmentation contained in the generalized text. For each segmentation, the number of characters contained in the segmentation is determined (for the convenience of explanation, it is recorded as the first number), and it is determined whether the phrase containing the first number of characters before the segmentation in the generalized text is consistent with the segmentation, and whether the phrase containing the first number of characters after the segmentation in the generalized text is consistent with the segmentation. If it is determined that the phrase containing the first number of characters before the segmentation in the generalized text is not consistent with the segmentation, and it is determined that the phrase containing the first number of characters after the segmentation in the generalized text is not consistent with the segmentation, then it is determined that the segmentation is not a repeated phrase, and the next segmentation is obtained; if it is determined that the phrase containing the first number of characters before the segmentation in the generalized text is consistent with the segmentation, or it is determined that the phrase containing the first number of characters after the segmentation in the generalized text is consistent with the segmentation, then it is determined that the segmentation is a repeated phrase, and the segmentation is deleted from the generalized text, and the next segmentation is obtained.

[0106] In another possible implementation, the generalized text is segmented according to a preset segmentation method to determine each segmentation contained in the generalized text. For each segmentation, each segmentation adjacent to the segmentation in the generalized text is determined (for the convenience of description, it is recorded as an adjacent segmentation), and it is determined whether the segmentation is consistent with any adjacent segmentation. If it is determined that the segmentation is consistent with any adjacent segmentation, the segmentation is determined to be a repeated phrase, the segmentation is deleted from the generalized text, and the next segmentation is obtained; if it is determined that the segmentation is inconsistent with each adjacent segmentation, it is determined that the segmentation is not a repeated phrase, and the next segmentation is obtained.

[0107] After determining the continuous repeated phrases in the generalized text in the above manner, the continuous repeated phrases can be deleted from the generalized text to one phrase, so that the target generalized text corresponding to the acquired text corpus is closer to normal natural language.

[0108] For example, the generalized text is "Today I went to see a concert". Through the above method, it can be determined that the continuous repeated phrases in the generalized text are "today", and the two continuous repeated phrases "today" in the generalized text are deleted to one phrase "today", thereby obtaining the target generalized text as "Today I went to see a concert".

[0109] In one possible implementation, in order to further improve the accuracy of the target generalized text corresponding to the text corpus, in an embodiment of the present invention, after determining the continuous repeated phrases existing in the generalized text, the staff can be notified to check the continuous repeated phrases existing in the generalized text, so as to determine whether the continuous repeated phrases existing in the generalized text need to be processed based on the inspection results of the staff, so that the generalized text corresponding to the acquired text corpus is closer to normal natural language.

[0110] Among them, the staff can also directly process the continuous repeated phrases in the generalized text according to their own inspection results.

[0111] In the embodiment of the present invention, the staff can be notified by means of communication such as text messages and emails, or the display of the smart device can be controlled to display a generalized text with continuous repeated phrases to notify the staff. The specific notification method can be flexibly set according to actual needs and is not specifically limited here.

[0112] In addition, in order to further accurately determine the target generalized text corresponding to the text corpus, the target generalized text corresponding to the text corpus is determined according to the generalized text obtained by decoding, including:

[0113] If the evaluation score corresponding to the generalized text is not obtained, or the evaluation score corresponding to the generalized text obtained is greater than the preset score, the generalized text is determined as the target generalized text.

[0114] In an embodiment of the present invention, the staff can perform a quality check on the generalized text corresponding to the currently acquired text corpus, such as determining whether the generalized text is semantically similar to the text corpus, whether the word order of the generalized text is normal, whether the generalized text is a text in the target field, etc., and score the generalized text corresponding to the text corpus according to the quality check results. If the staff inputs an evaluation score for the generalized text corresponding to a text corpus, the text corpus, the generalized text, and the evaluation score are correspondingly saved.

[0115] In order to further accurately determine the target generalized text corresponding to the text corpus, in the embodiment of the present invention, a score is preset. After the generalized text is obtained based on the method in the above embodiment, it is determined whether the evaluation score corresponding to the generalized text is currently saved. If there is an evaluation score, it means that the staff has conducted a quality inspection on the generalized text, and then it is determined whether the evaluation score corresponding to the generalized text is not greater than the preset score. If it is determined that the evaluation score corresponding to the generalized text is not greater than the preset score, it means that the quality inspection result of the generalized text is not good, and the generalized text is not determined as the target generalized text corresponding to the text corpus; if it is determined that the evaluation score corresponding to the generalized text is greater than the preset score, it means that the generalized text is qualified, and the generalized text is determined as the target generalized text corresponding to the text corpus.

[0116] In another possible situation, if it is determined that the evaluation score corresponding to the generalized text is not currently saved, it means that the staff has not performed quality inspection on the generalized text. Then you can continue to wait until the evaluation score corresponding to the generalized text is obtained, and then perform subsequent steps to determine whether the evaluation score is not greater than the preset score. You can also directly determine the generalized text as the target generalized text corresponding to the text corpus.

[0117] In the specific implementation process, after obtaining the generalized text corresponding to a certain text corpus, according to the text corpus and the generalized text, it is queried whether there is an evaluation score of the generalized text of the text corpus currently saved. If there is an evaluation score, and the evaluation score corresponding to the generalized text is not greater than the preset score, it means that the staff has conducted a quality inspection on the generalized text, but the quality inspection result of the generalized text is not good, and the generalized text is not determined as the target generalized text corresponding to the text corpus. If the evaluation score is not obtained, or the evaluation score corresponding to the generalized text obtained is greater than the preset score, the generalized text is determined as the target generalized text corresponding to the text corpus.

[0118] Embodiment 7: In order to obtain the generalized text corresponding to the text corpus through at least one text generalization model, based on the above embodiments, in this embodiment of the present invention, if the number of text generalization models is greater than 1, each text generalization model is trained in the following manner:

[0119] Obtain any sample corpus in the sample set, the sample corpus corresponds to a standard probability sequence corresponding to each character in the sample generalized text, the standard probability sequence corresponding to any character includes a pre-configured probability value corresponding to the character and probability values ​​corresponding to other characters except the character; obtain at least one second probability sequence corresponding to the sample generalized text through each original text generalization model, and the parameter values ​​of the parameters of different original text generalization models are different; for each second probability sequence, calculate a sub-loss value according to the second probability sequence and the corresponding standard probability sequence; and train each original text generalization model according to the sum of each sub-loss value.

[0120] In order to obtain the generalized text corresponding to the text corpus through at least one text generalization model, each text generalization model needs to be trained based on the sample corpus in the pre-acquired sample set, wherein the sample corpus in the sample set corresponds to the standard probability sequence corresponding to each character in the sample generalized text.

[0121] It should be noted that the standard probability sequence corresponding to any character in the sample generalized text includes the probability value corresponding to each character in the pre-configured characters and the probability values ​​corresponding to other characters except the character.

[0122] In a specific implementation, any sample corpus in the sample set is obtained, and the sample corpus is preprocessed to obtain a digital sequence corresponding to the sample corpus. The digital sequence is simultaneously input into each original text generalization model. Through each original text generalization model, based on the input digital sequence, at least one probability sequence (for the convenience of explanation, recorded as a second probability sequence) corresponding to the sample generalized text is obtained. Among them, any original text generalization model outputs each second probability sequence in sequence. The parameter values ​​of some parameters of different original text generalization models are configured differently, or the parameter values ​​of all parameters are configured differently.

[0123] For each second probability sequence output by each original text generalization model each time, determine the vector sum of the second probability sequence output by each original text generalization model at the same time (for the convenience of explanation, recorded as the sample vector sum). According to the sample vector sum and the corresponding standard probability sequence, calculate the sub-loss value.

[0124] After obtaining the sub-loss value corresponding to each character contained in the sample generalized text, each original text generalization model is trained according to the sum of each sub-loss value to adjust the parameter values ​​of each parameter in each original text generalization model respectively.

[0125] The sample set used for training each original text generalization model contains a large number of sample corpora, and the above operations are performed on each sample corpus. When the preset convergence condition is met, the training of each text generalization model is completed. Among them, the preset convergence condition can be met for the sum of each sub-loss value obtained in the current training to be less than the preset loss threshold, or the number of iterations for training each original text generalization model reaches the set maximum number of iterations, etc. It can be set flexibly in the specific implementation, and is not specifically limited here.

[0126] As a possible implementation method, when training each original text generalization model, the sample corpus in the sample set can be divided into training samples and test samples. Each original text generalization model is first trained based on the training samples, and then the reliability of each trained text generalization model is verified based on the test samples.

[0127] It should be noted that the electronic device for training the generalization model of each original text in the embodiment of the present invention can be the same as the electronic device for generating the text corpus in the above embodiment, or it can be different. In the specific implementation process, it can be flexibly set according to actual needs and is not specifically limited here.

[0128] In another possible implementation, when there is one text generalization model, the text generalization model is trained in the following manner:

[0129] Obtain any sample corpus in the sample set, the sample corpus corresponds to a standard probability sequence corresponding to each character in the sample generalized text, and the standard probability sequence corresponding to any character includes a pre-configured probability value corresponding to each character and probability values ​​corresponding to other characters except the character; obtain at least one second probability sequence corresponding to the sample generalized text through the original text generalization model; for each second probability sequence, calculate a sub-loss value according to the second probability sequence and the corresponding standard probability sequence; and train each original text generalization model respectively according to the sum of each sub-loss value.

[0130] Specifically, the training process of a text generalization model is similar to that described in the above embodiment, and the repeated parts are not repeated here.

[0131] In order to facilitate the training of the text generalization model, it is necessary to collect sample corpus and sample generalized text corresponding to the sample corpus in advance. In an embodiment of the present invention, the sample corpus and the sample generalized text corresponding to the sample corpus are determined by the following method:

[0132] Determine a sample corpus and a sample generalized text corresponding to the sample corpus based on log data of the intelligent question-answering system, wherein the log data includes an input question sentence input into the intelligent question-answering system and a standard question sentence matched with the input question sentence determined by the intelligent question-answering system; and / or

[0133] Based on the text sentences input into the semantic recognition model for semantic recognition, and the semantic recognition results corresponding to the text sentences, the sample corpus and the sample generalized text corresponding to the sample corpus are determined. The semantic recognition results include the domain information, slot information and intent information of the text sentences.

[0134] Specifically, the sample corpus and the sample generalized text corresponding to the sample corpus can be determined in the following way:

[0135] Method 1: In an embodiment of the present invention, log data generated by an intelligent question-answering system during actual use can be obtained. The log data includes input questions input into the intelligent question-answering system and standard questions matched with the input questions determined by the intelligent question-answering system. The sample corpus included in the sample set for training a text generalization model and the sample generalized text corresponding to the sample corpus can be obtained through the input questions included in the log data and the standard questions matched with the input questions.

[0136] In a possible implementation, determining a sample corpus and a sample generalized text corresponding to the sample corpus based on log data of the intelligent question answering system includes:

[0137] Determine log data containing the same standard question; and determine a sample corpus and a sample generalized text corresponding to the sample corpus according to the input question and the standard question included in the determined log data.

[0138] In the actual application process, the input question included in the log data and the standard question matching the input question are generally semantically similar sentences. Similarly, the input question and the standard question included in the log data containing the same standard question are generally semantically similar sentences, and semantically similar sentences can be generalized texts of each other. Based on this, in an embodiment of the present invention, the input question and the standard question included in the log data containing the same standard question can be used to determine the sample corpus and the sample generalized text corresponding to the sample corpus. Specifically, after obtaining the log data of the intelligent question-answering system, the log data containing the same standard question is determined (for the convenience of explanation, it is recorded as the target data), and the sample corpus and the sample generalized text corresponding to the sample corpus are determined according to the input question and the standard question included in the determined target data.

[0139] When determining the sample corpus and the sample generalized text corresponding to the sample corpus based on the input questions and standard questions included in the determined target data, any target question among all the questions (including input questions and standard questions) included in the target data is determined as the sample corpus, that is, any input question can be determined as the sample corpus, and the standard question can also be determined as the sample corpus, and other questions included in the target data except the target question are determined as the generalized text corresponding to the target question, that is, while ensuring that the sample corpus and the sample generalized text corresponding to the sample corpus are different, any input question can be determined as the sample text corpus corresponding to the sample corpus, and any standard question can also be determined as the sample text corpus corresponding to the sample corpus.

[0140] In a possible implementation, in order to ensure the accuracy of the determined sample corpus and the sample generalized text corresponding to the sample corpus, in an embodiment of the present invention, the log data is obtained by determining the log data whose input questions in the log data and the standard questions in the log data are semantically similar through a scoring model of an intelligent question-answering system, or by determining the log data whose input questions in the log data and the standard questions in the log data are semantically similar through manual quality inspection. In the above manner, it can be ensured that the determined sample corpus and the sample generalized text corresponding to the sample corpus are semantically similar.

[0141] Method 2: In an embodiment of the present invention, the text sentence input into the semantic recognition model for semantic recognition during actual use, and the semantic recognition result corresponding to the text sentence determined by the semantic recognition model can be obtained, so as to obtain the sample corpus included in the sample set for training the text generalization model and the sample generalized text corresponding to the sample corpus through the text sentence input into the semantic recognition model for semantic recognition and the semantic recognition result corresponding to the text sentence.

[0142] Among them, the domain information, slot information and intent information of the text sentence can be obtained through the semantic recognition model.

[0143] In a possible implementation, determining a sample corpus and a sample generalized text corresponding to the sample corpus according to a text sentence input into a semantic recognition model for semantic recognition and a semantic recognition result corresponding to the text sentence includes:

[0144] Determine text sentences with the same or similar semantic recognition results; and determine sample corpus and sample generalized text corresponding to the sample corpus based on the determined text sentences.

[0145] In the actual application process, the sample corpus and the sample generalized text corresponding to the sample corpus are two texts with similar semantics. The semantic recognition model can directly obtain the semantic recognition results corresponding to each input text sentence, and by comparing the semantic recognition results corresponding to each text sentence, the text sentences with the same or similar semantics are determined. Based on this, in an embodiment of the present invention, the sample corpus and the sample generalized text corresponding to the sample corpus can be determined according to the text sentences with the same or similar semantics. Specifically, after obtaining each text sentence input into the semantic recognition model for semantic recognition, and the semantic recognition results corresponding to each text sentence, the text sentences with the same or similar semantic recognition results are determined (for the convenience of description, they are recorded as target text sentences). According to the determined target text sentence, the sample corpus and the sample generalized text corresponding to the sample corpus are determined.

[0146] In a possible implementation, determining text sentences having the same semantic recognition result includes:

[0147] Get any text sentence; match the semantic recognition result corresponding to the text sentence with the semantic recognition results corresponding to other text sentences respectively. That is, match the domain information of the text sentence with the domain information corresponding to other text sentences, the slot information of the text sentence with the slot information corresponding to other text sentences, and the intention information of the text sentence with the intention information corresponding to other text sentences respectively. Among them, the matching here means that the semantic recognition result of the text sentence is consistent with the semantic recognition result corresponding to other text sentences, that is, the domain information of the text sentence is consistent with the domain information corresponding to other text sentences, the slot information of the text sentence is consistent with the slot information corresponding to other text sentences, and the intention information of the text sentence is consistent with the intention information corresponding to other text sentences. The text sentence and other text sentences that match the semantic recognition results corresponding to the text sentence are determined as target text sentences.

[0148] In another possible implementation, determining text sentences having similar semantic recognition results includes:

[0149] Get any text sentence; determine the similar results corresponding to the pre-configured semantic recognition results corresponding to the text sentence, and determine whether the semantic recognition results corresponding to other text sentences match any pre-configured similar results. That is, match the domain information of the other text sentence with the domain information in the pre-configured similar results, the slot information of the other text sentence with the slot information in the pre-configured similar results, and the intent information of the other text sentence with the intent information in the pre-configured similar results. Among them, the matching here means that the semantic recognition results corresponding to the other text sentences are consistent with the pre-configured similar results, that is, the domain information of the other text sentences is consistent with the domain information in the pre-configured similar results, the slot information of the other text sentences is consistent with the slot information in the pre-configured similar results, and the intent information of the other text sentences is consistent with the intent information in the pre-configured similar results. Determine the text sentence and other text sentences that match the similar results as the target text sentences.

[0150] After the target text sentences are determined, any two target text sentences can be determined as sample corpora and sample generalized texts corresponding to the sample corpora.

[0151] Embodiment 8: Figure 2 A specific text corpus generation process diagram provided in an embodiment of the present invention, wherein the implementation process of the method includes two parts: model training and text corpus generation. The following description is made using an electronic device for generating text corpus as a server:

[0152] Part 1: Model training.

[0153] S201: The first server trains each original text generalization model based on the sample corpus in the sample set and the sample generalization text corresponding to the sample corpus, and obtains each trained text generalization model.

[0154] Figure 3 A schematic diagram of a specific training process of each text generalization model provided in an embodiment of the present invention, the process includes:

[0155] S301: Obtain each log data of the intelligent question-answering system, determine the target data containing the same standard question sentence, and continue to execute S302.

[0156] S302: Determine a sample corpus and a sample generalized text corresponding to the sample corpus according to the input question and the standard question included in the determined target data, and then proceed to S305.

[0157] S303: Determine the target text sentences with the same or similar semantic recognition results, and proceed to S304.

[0158] S304: According to the determined target text sentence, determine the sample corpus and the sample generalized text corresponding to the sample corpus, and continue to execute S305.

[0159] Among them, the execution order between S301~S302 and S303~S304 is not particular, S301~S302 can be executed first and then S303~S304, S303~S304 can be executed first and then S301~S302, or S301~S302 and S303~S304 can be executed at the same time.

[0160] S305: Based on each sample corpus and the sample generalized text corresponding to each sample corpus, determine a sample set for text generalization model training, and continue to execute S306.

[0161] Among them, the sample corpus corresponds to the standard probability sequence corresponding to each character in the sample generalized text.

[0162] S306: Obtain any sample corpus in the sample set and the standard probability sequence corresponding to each character in the sample generalized text corresponding to the sample corpus, and continue to execute S307.

[0163] S307: Obtain at least one second probability sequence corresponding to the sample generalized text through each original text generalization model. The parameter values ​​of the parameters of different original text generalization models are different, and then continue to execute S308.

[0164] S308: For each second probability sequence, calculate the sub-loss value according to the second probability sequence and the corresponding standard probability sequence, and continue to execute S309.

[0165] S309: According to the sum of each sub-loss value, each original text generalization model is trained respectively to adjust the parameter values ​​of the parameters in each original text generalization model respectively.

[0166] For each sample corpus in the sample set, the above steps S306 to S309 are executed until a preset convergence condition is met, and it is determined that a trained text generalization model is obtained.

[0167] In the process of model training, an offline method is generally adopted, and the sample corpus in the sample set is trained in advance by the first server to obtain each text generalization model that has been trained.

[0168] The second part is text corpus generation. Based on each text generalization model trained by the first server, each text generalization model trained by the first server is saved to the second server, and the generalized text corresponding to the text corpus is determined by the second server. The specific implementation includes the following steps:

[0169] S202: The second server preprocesses the acquired text corpus.

[0170] S203: The second server obtains the generalized text corresponding to the text corpus based on each pre-trained text generalization model, the decoding network, and the pre-processed text corpus.

[0171] Specifically, at least one first probability sequence corresponding to the text corpus to be generalized is obtained through at least one text generalization model; for each first probability sequence output by each text generalization model, the vector sum of the first probability sequence output by the text generalization model at the same time is determined; if it is determined that the second character decoded from the first probability sequence meets the preset end condition, the decoded character sequence is determined as the generalized text. Wherein, determining that the decoded second character meets the preset end condition includes: if the second character is a pre-configured end mark, determining that the second character meets the preset end condition; and / or if the number of decoded characters reaches a set threshold, determining that the second character meets the preset end condition.

[0172] S204: The second server determines the longest common substring between the generalized text and the text corpus.

[0173] S205: If the second server determines that the length of the longest common substring is greater than a pre-configured length threshold.

[0174] S206: The second server uses a regular algorithm to delete characters matching any pre-configured non-standard characters from the generalized text.

[0175] S207: The second server determines whether the acquired text corpus corresponds to a name of a slot type, and if so, executes S208, otherwise, executes S209.

[0176] S208: The second server determines a target participle in the generalized text that matches the keyword included in the text corpus, replaces the target participle in the generalized text with the name of the slot type corresponding to the keyword matching the target participle, and then executes S209.

[0177] S209: The second server adds a start character before the first character of each generalized text, and adds an end character after the last character.

[0178] S210: The second server uses each pre-configured sliding window to slide from the start character of each generalized text with a preset sliding step until it reaches the end character of the generalized text; determines the number of times the characters in the sliding window appear in each generalized text; and determines the dictionary corresponding to the sliding window based on the content of each character and the number of times each character content corresponds to.

[0179] S211: The second server starts from the first character in each generalized text except the start character, and uses a path search algorithm according to the dictionary corresponding to each sliding window to obtain an optimal generalized text with a preset optimal path number.

[0180] S212: The second server matches each optimal generalized text with other optimal generalized texts, and if there are other matching optimal generalized texts, performs deduplication processing on the optimal generalized text and the other matching optimal generalized texts.

[0181] S213: The second server removes duplicate phrases continuously included in each optimal generalized text.

[0182] S214: the second server, for each optimal generalized text, if the evaluation score corresponding to the optimal generalized text is not obtained, or the evaluation score corresponding to the optimal generalized text obtained is greater than a preset score.

[0183] S215: Determine the optimal generalized text as the target generalized text corresponding to the text corpus.

[0184] Embodiment 9: The embodiment of the present invention provides a text corpus generation device, Figure 4 A schematic diagram of the structure of a text corpus generation device provided by an embodiment of the present invention, the device comprising:

[0185] An acquisition unit 41 is used to acquire at least one first probability sequence corresponding to the text corpus to be generalized through at least one text generalization model, wherein the first probability sequence includes a probability value of each character that is pre-configured to be a corresponding character in the generalized text corresponding to the text corpus;

[0186] A decoding unit 42 is used to decode each first probability sequence in turn through a decoding network and the decoded first character; if it is determined that the second character decoded from the first probability sequence meets a preset end condition, the decoded character sequence is determined as a generalized text;

[0187] The determination unit 43 is used to determine the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding.

[0188] In a possible implementation, the determination unit 43 is further used to determine the text corpus to be generalized in the following manner: determining the text corpus to be generalized based on log data of the intelligent question and answer system, wherein the log data includes an input question input into the intelligent question and answer system, and a standard question determined by the intelligent question and answer system to match the input question; and / or determining the text corpus to be generalized based on a text sentence input into a semantic recognition model for semantic recognition.

[0189] In a possible implementation manner, the determining unit 43 is specifically configured to: determine the input question sentence and / or the standard question sentence included in the first log data as the text corpus to be generalized.

[0190] In a possible implementation manner, the acquiring unit 41 is specifically configured to:

[0191] If the number of text generalization models is greater than 1, for the first probability sequence output by each text generalization model each time, determine the vector sum of the first probability sequence output by the text generalization model at the same time; and decode the vector sum through the decoding network and the decoded first character.

[0192] In one possible implementation, the decoding unit 42 is specifically used to: if the second character is a preconfigured end marker, determine that the second character satisfies a preset end condition; and / or, if the number of decoded characters reaches a set threshold, determine that the second character satisfies a preset end condition.

[0193] In a possible implementation manner, the determining unit 43 is specifically configured to:

[0194] The longest common substring between the generalized text and the text corpus is determined; if the length of the longest common substring is greater than a pre-configured length threshold, the generalized text is determined to be the target generalized text.

[0195] In a possible implementation, the device further includes: a first processing unit;

[0196] The first processing unit is used to use a regular algorithm to delete characters matching any pre-configured non-standard characters from the generalized text before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding.

[0197] In a possible implementation, the first processing unit is further used to: determine a target participle in the generalized text that matches the keyword contained in the text corpus before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding; and replace the target participle in the generalized text with the name of the slot type corresponding to the keyword matching the target participle.

[0198] In a possible implementation, the first processing unit is further used to: determine, based on the decoded generalized text, before determining the target generalized text corresponding to the text corpus, add a start character before the first character of each generalized text, and add an end character after the end character; for each pre-configured sliding window, use the sliding window to start from the start character of each generalized text, and slide with a preset sliding step until the end character of the generalized text is reached; determine the number of times the characters in the sliding window appear in each generalized text; determine the dictionary corresponding to the sliding window based on the content of each character and the number of times each character content corresponds to; starting from the first character in each generalized text except the start character, according to the dictionary corresponding to each sliding window, use a path search algorithm to obtain the optimal generalized text with a preset optimal path number and determine it as the target generalized text.

[0199] In a possible implementation, the device further includes: a second processing unit;

[0200] The second processing unit is used to match each generalized text with other generalized texts before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding. If there are other matching generalized texts, the generalized text and the other matching generalized texts are deduplicated.

[0201] In a possible implementation, the second processing unit is further configured to perform a deduplication process on repeated phrases continuously included in the generalized text before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding.

[0202] In a possible implementation, the second processing unit is further configured to determine the generalized text as the target generalized text if the evaluation score corresponding to the generalized text is not obtained, or the evaluation score corresponding to the generalized text obtained is greater than a preset score.

[0203] In one possible implementation, if the number of text generalization models is greater than 1, each text generalization model is trained in the following manner:

[0204] Obtain any sample corpus in the sample set, the sample corpus corresponds to a standard probability sequence corresponding to each character in the sample generalized text, the standard probability sequence corresponding to any character includes a pre-configured probability value corresponding to the character and probability values ​​corresponding to other characters except the character; obtain at least one second probability sequence corresponding to the sample generalized text through each original text generalization model, and the parameter values ​​of the parameters of different original text generalization models are different; for each second probability sequence, calculate a sub-loss value according to the second probability sequence and the corresponding standard probability sequence; and train each original text generalization model according to the sum of each sub-loss value.

[0205] Embodiment 10: Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention, the electronic device comprises: a processor 51, a communication interface 52, a memory 53 and a communication bus 54, wherein the processor 51, the communication interface 52, and the memory 53 communicate with each other through the communication bus 54;

[0206] The memory 53 stores a computer program. When the program is executed by the processor 51, the processor 51 performs the following steps:

[0207] At least one first probability sequence corresponding to the text corpus to be generalized is obtained through at least one text generalization model, wherein each character in the first probability sequence is pre-configured as a probability value of a corresponding character in the generalized text corresponding to the text corpus; for each first probability sequence, the first probability sequence is decoded in turn through a decoding network and the decoded first character; if it is determined that the second character decoded from the first probability sequence meets a preset end condition, the decoded character sequence is determined as the generalized text; and according to the decoded generalized text, a target generalized text corresponding to the text corpus is determined.

[0208] Since the principle of solving the problem by the above electronic device is similar to the text corpus generation method in the above embodiment, the implementation of the above electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0209] Based on any of the above embodiments, the communication bus mentioned in the above electronic device can be a peripheral component interconnect standard (PCI) bus or an extended industrial standard architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 52 is used for communication between the above electronic device and other devices. The memory can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk storage.

[0210] Optionally, the memory may also be at least one storage device located far away from the aforementioned processor.

[0211] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital instruction processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0212] Embodiment 11: Based on the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0213] At least one first probability sequence corresponding to the text corpus to be generalized is obtained through at least one text generalization model, wherein each character in the first probability sequence is pre-configured as a probability value of a corresponding character in the generalized text corresponding to the text corpus; for each first probability sequence, the first probability sequence is decoded in turn through a decoding network and the decoded first character; if it is determined that the second character decoded from the first probability sequence meets a preset end condition, the decoded character sequence is determined as the generalized text; and according to the decoded generalized text, a target generalized text corresponding to the text corpus is determined.

[0214] Since the principle of solving the problem by the above-mentioned computer-readable storage medium is similar to the text corpus generation method in the above-mentioned embodiment, the implementation of the above-mentioned computer-readable storage medium can refer to the implementation of the method, and the repeated parts will not be repeated.

[0215] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0216] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0217] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0218] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0219] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.< / song> < / artist> < / song> < / eos>

Claims

1. A text corpus generation method, characterized in that: The method comprises: Obtaining, by means of at least one text generalization model, at least one first probability sequence corresponding to the text corpus to be generalized, wherein the first probability sequence includes a probability value of each pre-configured character corresponding to a character in a generalized text corresponding to the text corpus; For each of the first probability sequences, the first probability sequence is decoded by using a beam search algorithm in a decoding network and the decoded first character; if it is determined that the second character decoded from the first probability sequence satisfies a preset end condition, the decoded character sequence is determined as a generalized text; Determining a target generalized text corresponding to the text corpus according to the generalized text obtained by decoding; Wherein, if the number of the text generalization models is greater than 1, decoding the first probability sequence by using a beam search algorithm in a decoding network and the decoded first character includes: For each first probability sequence outputted by the text generalization model each time, determining the vector sum of the first probability sequences outputted by the text generalization model at the same time; The vectors and are decoded by using a beam search algorithm in the decoding network and the decoded first character.

2. The method according to claim 1, characterized in that The text corpus to be generalized is determined in the following way: Determining a text corpus to be generalized based on log data of an intelligent question-answering system, wherein the log data includes an input question sentence input into the intelligent question-answering system, and a standard question sentence matched with the input question sentence determined by the intelligent question-answering system; and / or According to the text sentences input into the semantic recognition model for semantic recognition, the text corpus to be generalized is determined.

3. The method according to claim 2, characterized in that Determining the text corpus to be generalized based on the acquired first log data of the intelligent question answering system includes: The input question sentence and / or the standard question sentence included in the first log data is determined as the text corpus to be generalized.

4. The method according to claim 1, characterized in that The step of determining whether the second character decoded from the first probability sequence satisfies a preset end condition includes: If the second character is a pre-configured end mark, determining that the second character satisfies a preset end condition; and / or If the number of characters obtained by decoding reaches a set threshold, it is determined that the second character meets a preset end condition.

5. The method according to claim 1, characterized in that The step of determining a target generalized text corresponding to the text corpus according to the generalized text obtained by decoding includes: Determining the longest common substring between the generalized text and the text corpus; If the length of the longest common substring is greater than a pre-configured length threshold, the generalized text is determined to be the target generalized text.

6. The method according to claim 1, characterized in that Before determining the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding, the method further includes: A regular algorithm is used to delete characters matching any pre-configured non-standard characters from the generalized text.

7. The method according to claim 1, characterized in that Before determining the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding, the method further includes: Determining a target word segment in the generalized text that matches a keyword included in the text corpus; The target participle in the generalized text is replaced by the name of the slot type corresponding to the keyword matching the target participle.

8. The method according to claim 1, characterized in that The step of determining the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding further includes: Adding a start character before the first character of each generalized text and adding an end character after the last character; For each pre-configured sliding window, the sliding window is used to slide from the start character of each generalized text at a preset sliding step length until the end character of the generalized text is reached; the number of times the character in the sliding window appears in each generalized text is determined; and the dictionary corresponding to the sliding window is determined according to the content of each character and the number of times corresponding to each character content; Starting from the first character except the start character in each generalized text, according to the dictionary corresponding to each sliding window, a path search algorithm is adopted to obtain the optimal generalized text with a preset optimal path number to determine it as the target generalized text.

9. The method according to any one of claims 1 to 8, characterized in that: Before determining the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding, the method further includes: For each of the generalized texts, the generalized text is matched with other generalized texts. If there are other generalized texts that match, the generalized text and the other generalized texts that match are deduplicated.

10. The method according to any one of claims 1 to 8, characterized in that: Before determining the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding, the method further includes: Repeated phrases continuously included in the generalized text are deduplicated.

11. The method according to any one of claims 1 to 8, characterized in that: The step of determining a target generalized text corresponding to the text corpus according to the generalized text obtained by decoding includes: If the evaluation score corresponding to the generalized text is not obtained, or the evaluation score corresponding to the generalized text obtained is greater than a preset score, the generalized text is determined as the target generalized text.

12. The method according to any one of claims 1 to 8, characterized in that: If the number of the text generalization models is greater than 1, each of the text generalization models is trained in the following manner: Obtain any sample corpus in the sample set, wherein the sample corpus corresponds to a standard probability sequence corresponding to each character in the sample generalized text, and the standard probability sequence corresponding to any character includes a probability value corresponding to the character in each pre-configured character and probability values ​​corresponding to other characters except the character; Obtaining at least one second probability sequence corresponding to the sample generalized text through each original text generalization model, wherein the parameter values ​​of the parameters of different original text generalization models are different; For each of the second probability sequences, calculating a sub-loss value according to the second probability sequence and a corresponding standard probability sequence; According to the sum of each sub-loss value, each original text generalization model is trained respectively.

13. The method according to claim 12, characterized in that The sample corpus and the sample generalized text corresponding to the sample corpus are determined in the following manner: Determining the sample corpus and the sample generalized text corresponding to the sample corpus based on log data of the intelligent question-answering system, wherein the log data includes an input question sentence input into the intelligent question-answering system and a standard question sentence matched with the input question sentence determined by the intelligent question-answering system; and / or The sample corpus and the sample generalized text corresponding to the sample corpus are determined based on the text sentence input into the semantic recognition model for semantic recognition, and the semantic recognition result corresponding to the text sentence. The semantic recognition result includes the domain information, slot information and intent information of the text sentence.

14. The method according to claim 13, characterized in that The determining of the sample corpus and the sample generalized text corresponding to the sample corpus based on the log data of the intelligent question answering system includes: Determine the log data containing the same standard questions; The sample corpus and the sample generalized text corresponding to the sample corpus are determined according to the input question and the standard question included in the determined log data.

15. The method according to claim 13, characterized in that The determining of the sample corpus and the sample generalized text corresponding to the sample corpus according to the text sentence input into the semantic recognition model for semantic recognition and the semantic recognition result corresponding to the text sentence includes: determining text sentences having the same or similar semantic recognition results; According to the determined text sentence, a sample corpus and a sample generalized text corresponding to the sample corpus are determined.

16. A text corpus generation device, characterized in that: The device comprises: An acquisition unit, configured to acquire, through at least one text generalization model, at least one first probability sequence corresponding to the text corpus to be generalized, wherein the first probability sequence includes a probability value of each pre-configured character corresponding to a character in a generalized text corresponding to the text corpus; A decoding unit is used to decode each of the first probability sequences in turn by using a beam search algorithm in a decoding network and the decoded first character; if it is determined that the second character decoded from the first probability sequence meets a preset end condition, the decoded character sequence is determined as a generalized text; A determination unit, configured to determine a target generalized text corresponding to the text corpus according to the generalized text obtained by decoding; The acquisition unit is specifically used for: If the number of the text generalization models is greater than 1, for each first probability sequence output by the text generalization model each time, determine the vector sum of the first probability sequence output by the text generalization model at the same time; and decode the vector sum by using the beam search algorithm in the decoding network and the decoded first character.

17. The device according to claim 16, characterized in that The determining unit is further configured to determine the text corpus to be generalized in the following manner: Based on the log data of the intelligent question-answering system, the text corpus to be generalized is determined, wherein the log data includes an input question input into the intelligent question-answering system, and a standard question determined by the intelligent question-answering system to match the input question; and / or, the text corpus to be generalized is determined based on the text sentence input into the semantic recognition model for semantic recognition.

18. The device according to claim 17, characterized in that The determining unit is specifically used for: The input question sentence and / or the standard question sentence included in the first log data is determined as the text corpus to be generalized.

19. The device according to claim 16, characterized in that The decoding unit is specifically used for: If the second character is a pre-configured end identifier, it is determined that the second character meets the preset end condition; and / or if the number of decoded characters reaches a set threshold, it is determined that the second character meets the preset end condition.

20. The device according to claim 16, characterized in that The determining unit is specifically used for: The longest common substring between the generalized text and the text corpus is determined; if the length of the longest common substring is greater than a pre-configured length threshold, the generalized text is determined to be the target generalized text.

21. The device according to claim 16, characterized in that The device further comprises: a first processing unit; The first processing unit is used to use a regular algorithm to delete characters matching any pre-configured non-standard characters from the generalized text before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding.

22. The device according to claim 21, characterized in that The first processing unit is also used to: determine the target participle in the generalized text that matches the keywords contained in the text corpus before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding; and replace the target participle in the generalized text with the name of the slot type corresponding to the keyword matching the target participle.

23. The device according to claim 21, characterized in that The first processing unit is further configured to: add a start character before the first character of each generalized text and add an end character after the last character before determining the target generalized text corresponding to the text corpus according to the generalized text obtained by decoding; for each pre-configured sliding window, use the sliding window to start from the start character of each generalized text and perform sliding processing with a preset sliding step length until reaching the end character of the generalized text; Determine the number of times the character in the sliding window appears in each of the generalized texts; Determine a dictionary corresponding to the sliding window according to each character content and the number of times each character content corresponds to; Starting from the first character except the start character in each generalized text, according to the dictionary corresponding to each sliding window, a path search algorithm is adopted to obtain the optimal generalized text with a preset optimal path number to determine it as the target generalized text.

24. The device according to any one of claims 16 to 23, characterized in that: The device further comprises: a second processing unit; The second processing unit is used to match each generalized text with other generalized texts before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding, and if there are other matching generalized texts, deduplicate the generalized text and the other matching generalized texts.

25. The device according to claim 24, characterized in that The second processing unit is further configured to perform a deduplication process on repeated phrases continuously included in the generalized text before determining the target generalized text corresponding to the text corpus based on the generalized text obtained by decoding.

26. The device according to claim 24, characterized in that The second processing unit is further configured to determine the generalized text as the target generalized text if the evaluation score corresponding to the generalized text is not obtained, or if the evaluation score corresponding to the generalized text obtained is greater than a preset score.

27. An electronic device, characterized in that: The electronic device comprises at least a processor and a memory, and the processor is used to implement the steps of the text corpus generation method as claimed in any one of claims 1 to 15 when executing a computer program stored in the memory.

28. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the text corpus generation method as described in any one of claims 1 to 15.