Corpus screening method and apparatus, electronic device, computer readable medium

Abstract

The disclosure provides a corpus screening method and device, and relates to the technical fields of artificial intelligence such as knowledge graph, natural language processing and deep learning. The specific implementation scheme is: selecting a text sample set of a target field from an obtained candidate corpus set; based on the text sample set and a pre-constructed language network, a language model is trained, the language model is used to predict a probability distribution of a next text unit adjacent to input content in the current input text being a text in a preset word table according to the input content; based on the language model, the probability of each candidate corpus in the candidate corpus set belonging to the target field is determined; and based on the probability of each candidate corpus belonging to the target field, the candidate corpus set is screened to obtain a first screening result. The embodiment improves the accuracy of target field corpus screening.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to knowledge graphs, natural language processing, deep learning and other technical fields, and in particular to a corpus screening method and apparatus, electronic devices, computer-readable media and computer program products. Background Technology

[0002] For corpora in the target domain, filtering is generally done through manual rules or by checking the frequency of occurrence of keywords in the target domain.

[0003] Existing screening methods require the creation of different manual rules for different fields, and separate screening rules need to be built for different batches of data in different fields, resulting in high labor costs and poor screening effects. Summary of the Invention

[0004] A corpus screening method and apparatus, electronic device, computer-readable medium, and computer program product are provided.

[0005] According to the first aspect, a corpus screening method is provided, which includes: selecting a text sample set of a target domain from an acquired candidate corpus; training a language model based on the text sample set and a pre-constructed language network, wherein the language model is used to predict the probability distribution of the next text unit adjacent to the input content in the current input text as a text in a preset vocabulary; determining the probability that each candidate corpus in the candidate corpus belongs to the target domain based on the language model; and screening the candidate corpus to obtain a first screening result based on the probability that each candidate corpus belongs to the target domain.

[0006] According to a second aspect, a corpus filtering apparatus is provided, comprising: a set selection unit configured to select a set of text samples of a target domain from an acquired candidate corpus; a language training unit configured to train a language model based on the text sample set and a pre-constructed language network, wherein the language model is used to predict the probability distribution of the next text unit adjacent to the input content being a text in a preset vocabulary based on the input content in the current input text; a determination unit configured to determine the probability that each candidate corpus in the candidate corpus belongs to the target domain based on the language model; and a filtering unit configured to filter the candidate corpus to obtain a first filtering result based on the probability that each candidate corpus belongs to the target domain.

[0007] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0008] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the method described in any implementation of the first aspect.

[0009] According to a fifth aspect, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0010] The corpus filtering method and apparatus provided in the embodiments of this disclosure first select a text sample set of the target domain from the acquired candidate corpus set; second, based on the text sample set and a pre-constructed language network, a language model is trained, which is used to predict the probability distribution of the next text unit adjacent to the input content in the current input text as a text in a preset vocabulary; third, based on the language model, the probability of each candidate corpus in the candidate corpus belonging to the target domain is determined; finally, based on the probability of each candidate corpus belonging to the target domain, the candidate corpus is filtered to obtain a first filtering result. Thus, by using the text sample set of the target domain to train the language model, the language model becomes familiar with the language style of the target domain. By using the text probability distribution determined when the language model predicts text, the probability of each candidate corpus in the candidate corpus belonging to the target domain is determined, thereby fully utilizing the recognition performance of the language model, filtering out more corpus from the target domain, and improving the accuracy of target domain corpus filtering.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a flowchart of an embodiment of the corpus screening method according to the present disclosure;

[0014] Figure 2 This is a schematic diagram of a corpus screening process according to an embodiment of the present disclosure;

[0015] Figure 3 This is a flowchart of another embodiment of the corpus screening method according to the present disclosure;

[0016] Figure 4 This is another structural diagram of the corpus screening process in an embodiment of this disclosure;

[0017] Figure 5 This is a schematic diagram of the structure of an embodiment of the corpus screening device according to the present disclosure;

[0018] Figure 6 This is a block diagram of an electronic device used to implement the corpus screening method of the embodiments of this disclosure. Detailed Implementation

[0019] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0020] In this embodiment, "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0021] Figure 1 A flowchart 100 is shown as an embodiment of the corpus screening method according to the present disclosure, the corpus screening method including the following steps:

[0022] Step 101: Select a set of text samples for the target domain from the acquired candidate corpus.

[0023] In this embodiment, the candidate corpus set is a batch of massive corpora obtained by using search engines or other methods. The candidate corpus set includes corpora from the target domain and corpora from non-target domains. The target domain is the domain corresponding to the screening target, such as the medical field; the non-target domain is the domain outside the target domain.

[0024] In this embodiment, the text sample set is a collection of text samples from the target domain obtained to achieve the screening objective. These text samples are a small number of samples from the screening objective. For example, if the screening objective is to obtain millions of text samples related to "cancer screening", then a small number of text samples, such as 1,000, are first obtained from the candidate text set.

[0025] In this embodiment, the execution entity running on the corpus selection method needs to acquire a candidate corpus set in order to train the language model, and the execution entity of the corpus selection method can acquire the candidate corpus set in various ways. For example, the execution entity can acquire the candidate corpus set stored in the database server through a wired or wireless connection. Alternatively, the execution entity can receive a candidate corpus set collected in real time by a terminal or other device.

[0026] In this embodiment, the candidate corpus set includes at least one candidate corpus, and each candidate corpus may include at least one candidate unit arranged in sequence. Depending on the unit processed by the language model, the candidate unit may be a candidate character, candidate word, etc. The text sample set includes at least one text sample, and each text sample may include at least one unit sample arranged in sequence. Depending on the unit processed by the language model, the unit sample may be a character sample, word sample, etc.

[0027] In this embodiment, selecting a text sample set of the target domain from the candidate corpus includes: selecting a text sample set from the candidate corpus based on pre-set target domain text rules.

[0028] Optionally, selecting a text sample set of the target domain from the candidate corpus may further include: matching candidate units in the candidate corpus of the candidate corpus with entity names in the knowledge graph based on a pre-constructed knowledge graph of the target domain; and in response to at least one candidate unit matching an entity name in the knowledge graph, using the candidate corpus with at least one candidate unit as a text sample in the text sample set.

[0029] Step 102: Based on the text sample set and the pre-built language network, a language model is trained to obtain the language model.

[0030] In this embodiment, the language model is used to predict the probability distribution of the next text unit adjacent to the input content as a text in a preset vocabulary, based on the input content in the current input text.

[0031] In this embodiment, the input text is the text input to the language model. The text input to the language model can be any text applicable to the language model, and the input text can include multiple sequentially adjacent input contents. The input contents are the units that the language model can predict. For example, the input text is wt, where w and t are both input contents. Inputting wt into the language model yields the language model prediction result w't'. The prediction result for w is w', where w' is the probability distribution of w belonging to the text in the preset vocabulary. The prediction result for t is t', where t' is the probability distribution of t belonging to the text in the preset vocabulary.

[0032] In this embodiment, the probability distribution includes multiple probability values. Each probability value is the probability that each input content obtained by the language model is a text in the preset vocabulary. The number of probability values ​​in the probability distribution is the same as the number of texts in the preset vocabulary (the texts in the vocabulary can be the basic units of the vocabulary). For example, if the preset vocabulary has 100 texts, then the probability distribution of each input content includes 100 probability values.

[0033] In this embodiment, the language network is an initial text generation network. After training, the text generation network obtains a speech model. The speech model is used to recognize each unit sample in the text sample set and obtain the recognition result of the corresponding unit sample.

[0034] In this embodiment, as Figure 2 As shown, a language model is trained using a text sample set. During training, a pre-defined vocabulary is first specified for the language model. This vocabulary includes at least one text. Based on this vocabulary, the language network predicts the probability distribution of the next text unit adjacent to the input content as a text from the pre-defined vocabulary when predicting the next text unit. For example, if the pre-defined vocabulary includes three texts: c, d, and e, and the input text "ad" contains the character "a", the language model predicts the next text unit adjacent to "a". The predicted probability is 20% for "c", 80% for "d", and 10% for "e". Therefore, "c" - 20%, "d" - 80%, and "e" - 10% represent the probability distribution of "d" as a text from the pre-defined vocabulary. In this case, the 80% probability of "d" from the pre-defined vocabulary is selected as the probability value of "d" in the input text.

[0035] In this embodiment, to better train the language model for the target domain, the preset vocabulary can be replaced with a text sample set plus commonly used words, so that the language model output results all have a corresponding probability distribution in the text sample set. The probability of each word appearing in each qualified sentence in the prediction results output by the language model can be used to represent the perplexity of the model, and the perplexity can also characterize the association or dependency relationship between each word in the sentences selected by the language model. The language model provided in this embodiment can distinguish the domain of the corpus through perplexity scoring.

[0036] In this embodiment, the network structure of the language network can adopt structures such as RNN (Recurrent Neural Network), LSTM (Long Short-Term Memory), Transformer, or Transformer-XL.

[0037] In this embodiment, the training steps of the language model include: Step 1, selecting text samples from the text sample set; Step 2, inputting the selected text samples into the language network, enabling the language network to decode the selected text samples and predict the prediction results for the corresponding text samples; Step 3, calculating the loss value of the language network based on the prediction results of the language network and the ground truth values ​​in the selected text samples; Step 4, if the language network does not meet the training completion conditions, adjusting the parameters of the language network and continuing to execute steps 1 to 4 until the language network meets the training completion conditions, and then using the language network as the language model. In this embodiment, the training completion conditions include: the loss of the language network reaching a certain loss threshold or the number of training iterations of the language network reaching a preset number, wherein the number of training iterations refers to the number of times steps 1 to 4 are executed.

[0038] Taking a transformer-structured language model as an example, each text sample in the text sample set is input into the model character by character, allowing the language model to learn what the next character of the text is (selecting the most likely character from a fixed set of tens of thousands of words). Through this training, the language model can become familiar with the text style of the target domain. Once the language model is trained, it can be used to filter the massive candidate corpus set in step 101.

[0039] Step 103: Based on the language model, determine the probability that each candidate corpus in the candidate corpus belongs to the target domain.

[0040] In this embodiment, based on the smallest text unit (candidate unit) processed by the language model, each candidate corpus includes at least one candidate text, and each candidate text is the smallest text unit of a language model. The language model is used to predict the candidate texts of each candidate corpus in the candidate corpus set. Each candidate text has a probability distribution. The probability corresponding to the candidate text is selected from the probability distribution to obtain the probability of each candidate text. The probabilities of all candidate texts in each candidate corpus are combined to obtain the probability of each candidate corpus. Since the language model is trained on the text sample set, it has the characteristics of the target domain in which the text sample set is located. Therefore, the higher the probability of each candidate corpus, the more the language model tends to think that the candidate corpus is familiar to the language model, and the more the language model thinks that the candidate corpus belongs to the target domain. Therefore, the probability of each candidate corpus is the probability that each candidate corpus belongs to the target domain. Figure 2 The candidate corpora in the candidate corpus set are input into the language model to obtain the probability distribution of the candidate corpora in the language model. Based on the probability distribution, the probability of each candidate corpus belonging to the target domain is determined.

[0041] Step 104: Based on the probability that each candidate corpus belongs to the target domain, the candidate corpus set is filtered to obtain the first filtering result.

[0042] In this embodiment, the candidate corpus in the first screening result is the corpus of the target domain. Based on the requirements of the screening target, the number of candidate corpus in the first screening result can be the number required by the screening target, such as several million or more.

[0043] In this embodiment, when the target domain is different, the domains of each candidate corpus in the first screening result are different. For example, when the target domain is the medical field, the first screening result is a corpus of millions of "cancer examination" related corpora.

[0044] In this embodiment, each candidate corpus in the first screening result can be either a sentence or a paragraph. By selecting candidate corpora with a higher probability of belonging to the target domain, corpora belonging to the target domain can be filtered out from a massive candidate corpus set. The probability of a candidate corpus belonging to the target domain reflects the likelihood of whether a candidate corpus belongs to the target domain; therefore, the higher the probability, the more likely the candidate corpus belongs to the first screening result. The above-mentioned method of obtaining the first screening result by screening the candidate corpus set based on the probability of each candidate corpus belonging to the target domain includes: determining the number of candidates to be screened; selecting candidate corpora with a probability greater than a preset value from the selected number of candidate corpora as candidate corpora of the target domain, and using the candidate corpora of the target domain as the first screening result.

[0045] The corpus filtering method provided in this embodiment is consistent across text corpora in any target domain. For candidate corpora from different domains and batches, no manual modification of the filtering strategy is required; only the text sample set for the target domain needs to be changed. The trained language model can then filter the corpora for that target domain, saving labor costs. The language model's filtering of corpora is adaptive, accurately matching the characteristics of the corpora, and is more refined than manual rules, resulting in fewer false deletions and higher-quality corpora. Because no manual operation is required, large volumes of corpora can be processed in a distributed manner across multiple machines. It boasts fast operation speed and good scalability.

[0046] The corpus filtering method provided in this disclosure first selects a text sample set of the target domain from the acquired candidate corpus; second, based on the text sample set and a pre-constructed language network, a language model is trained, which is used to predict the probability distribution of the next text unit adjacent to the input content in the current input text as a text in a preset vocabulary; third, based on the language model, the probability of each candidate corpus in the candidate corpus belonging to the target domain is determined; finally, based on the probability of each candidate corpus belonging to the target domain, the candidate corpus is filtered to obtain a first filtering result. Thus, by using a text sample set of the target domain to train the language model, the language model becomes familiar with the language style of the target domain. The probability distribution of the text determined when the language model predicts the text is used to determine the probability of each candidate corpus in the candidate corpus belonging to the target domain, thereby fully utilizing the recognition performance of the language model to filter corpus in the target domain and improving the accuracy of target domain corpus filtering.

[0047] In some embodiments of this disclosure, the language network is a decoder. The language model is trained based on the text sample set and the pre-constructed language network, including: selecting text samples from the text sample set, wherein each text sample includes at least one unit sample; inputting the start symbol, the selected text sample, and the end symbol into the decoder in sequence to obtain the probability distribution corresponding to each unit sample in the selected text sample output by the decoder, wherein the probability distribution is the prediction probability of the text unit in the text sample set being the prediction result; and obtaining the language model in response to the decoder satisfying the training completion condition.

[0048] In this optional implementation, the unit sample is determined by the smallest working unit of the language network. For example, if the selected text sample is a piece of text, then the unit sample is the character or word that makes up the piece of text.

[0049] In this optional implementation, the decoder can be a Transformer decoder. The decoder decodes only one word at a time. The start symbol and end symbol are the start and end of the decoding work recognized by the decoder. After the start symbol is input into the decoder, the decoder predicts the first unit sample of the selected text sample to obtain the first prediction result. The start symbol and the first unit sample are input into the decoder to obtain the second prediction result. After the end symbol and the selected text sample are input into the decoder, the decoder determines that the prediction work is completed. The decoder loss is calculated by using all prediction results and the selected text sample. In response to the decoder loss being less than or equal to the loss threshold, the decoder is determined to meet the training completion condition.

[0050] The optional implementation provides a method for training a language model by sequentially inputting a start symbol, a selected text sample, and an end symbol into a decoder to obtain the probability distribution of each unit sample in the selected text sample output by the decoder. In response to the decoder meeting the training completion condition, a language model is obtained. Thus, the language network is trained using a text sample set, allowing the language network to become familiar with the language style of the text sample set, thereby improving the reliability of language model training.

[0051] In some optional implementations of this embodiment, the above-mentioned determination of the probability that each candidate corpus in the candidate corpus belongs to the target domain based on the language model includes: for each candidate corpus in the candidate corpus, inputting the start symbol, the candidate corpus, and the end symbol into the language model in sequence to obtain the probability distribution of each candidate character corresponding to the candidate corpus; obtaining the probability of each candidate character based on the probability distribution of each candidate character; and calculating the probability that the candidate corpus belongs to the target domain based on the probability of all candidate characters in the candidate corpus.

[0052] In this optional implementation, the language network can adopt a transformer model structure. The transformer model consists of two main parts: an encoder and a decoder. The encoder is responsible for mapping the natural language sequence into hidden layers, which are the encoded output values. The encoded output values ​​include the mathematical expression of the natural language sequence. Then, the decoder maps the encoded output values ​​back into the natural language sequence, so that the language network can solve various problems, such as sentiment classification, named entity recognition, semantic relation extraction, summary generation, machine translation, etc.

[0053] The language model in this embodiment can be used for text generation. Based on the probability distribution of the language model's output, the probability value of the input text is determined. The process of obtaining the probability of each candidate character based on its corresponding probability distribution includes: after each candidate character in each candidate corpus is input into the language model, the language model predicts the candidate corpus to obtain the probability distribution of each candidate character in the candidate corpus. This probability distribution includes all characters in the prediction vocabulary and the probability corresponding to each character in the prediction vocabulary. Characters that are the same as or match each candidate character in the probability distribution are selected as target characters, and the probability corresponding to each target character is used as the probability of each candidate character.

[0054] The method for determining the probability of each candidate corpus in the target domain provided by this optional implementation first inputs each candidate character of each candidate corpus into the language model in turn to obtain the probability distribution corresponding to each candidate character; determines the probability of the candidate character based on the probability distribution of the candidate character, and calculates the probability of each candidate corpus in the target domain based on the probability of the candidate character. Thus, through the understanding of the language style of the target domain by the language model, the result of whether all candidate corpora in the candidate corpus set belong to the target domain is obtained, thereby improving the reliability of the target domain corpus obtained.

[0055] In some optional implementations of this embodiment, calculating the probability that the candidate corpus belongs to the target domain based on the probabilities of all candidate characters of the candidate corpus includes: multiplying the probabilities of all candidate characters of the candidate corpus to obtain the probability that the candidate corpus belongs to the target domain.

[0056] In this optional implementation, a candidate corpus is input to the language model character by character, and the language model is made to predict the probability distribution of the next character as a character in the preset word list. For example, if the candidate corpus is "I love the grassland", first input the start symbol to the language model to make the language model predict the probability distribution of the next character as a character in the preset word list, and select the probability of "[I]" from the probability distribution; input "I" to the language model to make the language model predict the probability distribution of the next character as a character in the preset word list, and select the probability of "love" from the probability distribution; then input "I love" to make the language model predict the probability distribution of the next character as a character in the preset word list, and select the probability of "grass" from the probability distribution; then input "I love grass" to make the language model predict the probability distribution of the next character as a character in the preset word list, and select the probability of "land" from the probability distribution. In this way, the prediction probabilities of each character of the candidate corpus by the language model can be obtained. Multiply the probabilities of all characters to obtain the probability that the candidate corpus belongs to the target domain.

[0057] The method for calculating the probability of a candidate corpus in the target domain provided by this optional implementation obtains the probability of the candidate corpus in the target domain by multiplying the probabilities of all candidate characters of the candidate corpus, providing an implementation for the probability of the candidate corpus.

[0058] Optionally, calculating the probability that the candidate corpus belongs to the target domain based on the probabilities of all candidate characters of the candidate corpus includes: adding the logarithms of the probabilities of all candidate characters of the candidate corpus to obtain the probability that the candidate corpus belongs to the target domain.

[0059] In some optional implementations of this embodiment, the above-mentioned filtering of candidate corpora based on the probability that each candidate corpus belongs to the target domain to obtain a first filtering result includes: in response to the probability that the candidate corpus in the candidate corpus belongs to the target domain being greater than or equal to a probability threshold, using the candidate corpus as the candidate corpus in the first filtering result.

[0060] In this optional implementation method, such as Figure 2 As shown, the probability threshold can be a threshold determined in the filtering target, or it can be a value set according to the filtering requirements. The probability threshold can also vary depending on the different forms of the candidate corpus in the target domain.

[0061] In this optional implementation, the first screening result includes multiple candidate corpora, all of which belong to the target domain.

[0062] The method for obtaining the first screening result provided by this optional implementation compares the probability of candidate corpora in the target domain with a probability threshold, which can maximize the screening of corpora in the target domain among the candidate corpora and improve the reliability of the first screening result.

[0063] Optionally, the above-mentioned screening of candidate corpora based on the probability that each candidate corpus belongs to the target domain to obtain the first screening result includes: sorting the probability of all candidate corpora in the target domain in descending order, and selecting a predetermined number of candidate corpora as the first screening result.

[0064] In this embodiment, the number can be set based on the sample screening requirements of the target domain, for example, the number can be set to 500,000.

[0065] In some embodiments of this disclosure, the above-mentioned corpus screening method further includes: training a similarity comparison model based on a text sample set and a text set in a non-target domain, the similarity comparison model being used to determine the similarity between two input texts; selecting text samples from the text sample set; inputting the selected text samples and each candidate corpus in the first screening result into the similarity comparison model to obtain the similarity between each candidate corpus and the text sample; and obtaining a second screening result based on the similarity.

[0066] In this embodiment, a similarity comparison model is trained using a text sample set and a text set from the non-target domain. This allows the similarity comparison model to become familiar with both types of text, including those from the target domain and those from the non-target domain, thereby distinguishing candidate corpora from the target domain and non-target domain in the first screening results.

[0067] In this embodiment, the greater the similarity between the candidate corpus and the text sample, the closer the candidate corpus and the text sample are, and the more likely the candidate corpus belongs to the target domain. By filtering based on similarity, candidate corpus with the same features as the corpus in the target domain can be selected from the perspective of corpus features, thereby further filtering the first filtering result.

[0068] The corpus screening method provided in the embodiments of this disclosure uses a language model to screen the candidate corpus set to obtain a first screening result. Then, through a trained similarity comparison model, it compares the candidate corpus in the first screening result of the text sample set and obtains a second screening result based on the similarity between the two. This further filters the first screening result from the perspective of similarity, thereby improving the accuracy of the candidate corpus in the second screening result belonging to the target domain.

[0069] In some optional implementations of this embodiment, the above-mentioned training of a similarity comparison model based on a text sample set and a text set from a non-target domain includes: selecting any two text samples from the text sample set and concatenating them to obtain a positive sample; randomly selecting text from the text set from the non-target domain and concatenating one text sample from the text sample set with the selected text to obtain a negative sample; training a similarity comparison network based on the positive and negative samples, wherein the input of the similarity comparison network is the two types of text and the output is the similarity value between the two types of text; and obtaining a similarity comparison model in response to the similarity comparison network satisfying the training completion condition.

[0070] The method for training the similarity comparison model provided by this optional implementation can maximize the diversity and quantity of positive and negative samples by concatenating text samples in the text sample set and concatenating text samples with text from non-target domains, thereby improving the reliability of similarity comparison model training.

[0071] Optionally, the above-mentioned training of a similarity comparison model based on a text sample set and a text set in a non-target domain includes: selecting text samples from the text sample set as positive samples; randomly selecting texts from the text set in the non-target domain as negative samples; training a similarity comparison network based on the positive and negative samples, wherein the input of the similarity comparison network is the two types of text, and the output is the similarity value between the two types of text; and obtaining the similarity comparison model in response to the similarity comparison network satisfying the training completion condition.

[0072] Figure 3 A flowchart 300 shows another embodiment of the corpus screening method of this disclosure, which includes the following steps:

[0073] Step 301: Select a set of text samples from the acquired candidate corpus for the target domain.

[0074] Step 302: Based on the text sample set and the pre-built language network, a language model is trained to obtain the language model.

[0075] Step 303: Based on the language model, determine the probability that each candidate corpus in the candidate corpus belongs to the target domain.

[0076] Step 304: Based on the probability that each candidate corpus belongs to the target domain, the candidate corpus set is filtered to obtain the first filtering result.

[0077] It should be understood that the operations and features in steps 301-304 correspond to the operations and features in steps 101-104, respectively. Therefore, the descriptions of the operations and features in steps 101-104 also apply to steps 301-304, and will not be repeated here.

[0078] Step 305: Train a similarity comparison model based on the text sample set and the text set of the non-target domain.

[0079] Step 306: Input the text samples in the text sample set and each candidate corpus in the first screening result into the similarity comparison model to obtain the similarity between each candidate corpus and the text sample.

[0080] Step 307: Based on similarity, obtain the second screening result.

[0081] It should be understood that the operations and features in steps 305-307 above have been described in the above embodiments and will not be repeated here.

[0082] Step 308: Perform named entity recognition on the candidate corpora in the second screening results to obtain the named entity sequences in each candidate corpus.

[0083] like Figure 4 As shown, entity recognition is performed on all candidate corpora in the second screening results to identify all entities in all candidate corpora.

[0084] Step 309: Based on the pre-constructed knowledge graph of the target domain, calculate the proportion of graph entities in the named entity sequence of each candidate corpus.

[0085] like Figure 4 As shown, the knowledge graph includes graph entities with different entity names in the target domain, as well as the relationships between these graph entities. The graph entities in the knowledge graph are matched with all entities in the candidate corpus by name, and the percentage of graph entities in the candidate corpus is calculated. For example, if a candidate corpus contains 100 characters, and 20 of these 20 entities match a graph entity, then the percentage of graph entities in the candidate corpus is 20 / 100 = 20%.

[0086] Step 310: Based on the percentage value, filter the second screening results to obtain the third screening results.

[0087] In this embodiment, the proportion of graph entities in each candidate corpus in the second screening result is determined. Based on the proportion of graph entities, all candidate corpora in the second screening result are sorted in reverse order from largest to smallest to obtain a candidate corpus sequence. The first predetermined number of candidate corpora in the candidate corpus sequence are selected as the third screening result. This predetermined number can be determined based on the screening objective, for example, 10,000.

[0088] For example, the graph entities of the knowledge graph include three graph entities: u, v, and i; the second screening result includes: first candidate corpus (x1, x2, x3), second candidate corpus (y1, v, y2, i), and third candidate corpus (u, v, i); the graph entity ratio of the first candidate corpus in the second screening result is 0 / 3; the graph entity ratio of the second candidate corpus in the second screening result is 2 / 4; the graph entity ratio of the third candidate corpus in the second screening result is 3 / 3; based on the ratio values ​​of all candidate corpora in the second screening result, the candidate corpus sequence obtained by sorting all candidate corpora from largest to smallest is: third candidate corpus, second candidate corpus, and first candidate corpus. When the number is set to 2, the third candidate corpus and the second candidate corpus in the candidate corpus sequence are selected as the third screening result.

[0089] The corpus screening method provided in the embodiments of this disclosure, based on the first screening result obtained by screening the candidate corpus set using a language model, compares the candidate corpus in the first screening result of the text sample set with the candidate corpus obtained by the trained similarity comparison model, and obtains the second screening result based on the similarity between the two; further, based on the pre-constructed knowledge graph of the target domain, the second screening result is screened to obtain the third screening result, thereby further screening the second screening result from the perspective of the target domain expertise, improving the accuracy of the candidate corpus in the third screening result belonging to the target domain.

[0090] In some other embodiments of this disclosure, the above-mentioned corpus screening method includes: training a corpus screening model based on a text sample set and a text set of non-target domains, the corpus screening model being used to determine the probability that the input text belongs to the target domain; inputting each candidate corpus from the first screening result into the corpus screening model to obtain the probability that each candidate corpus belongs to the target domain; and obtaining a fourth screening result based on the probability that each candidate corpus belongs to the target domain.

[0091] In this optional implementation, the corpus selection model is a model trained using text sample sets from the target domain and text sets from non-target domains. This corpus selection model can distinguish between texts from the target domain and non-target domains from the perspective of text features.

[0092] The corpus screening method provided in this embodiment uses a text sample set and a text set from a non-target domain to train a corpus screening model. The corpus screening model distinguishes candidate corpora in the first screening result, thereby improving the accuracy of candidate corpora in the first screening result belonging to the target domain.

[0093] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a corpus screening device, which is similar to... Figure 1 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0094] like Figure 5 As shown, the corpus screening device 500 provided in this embodiment includes: a set selection unit 501, a language training unit 502, a determination unit 503, and a screening unit 504. The set selection unit 501 can be configured to select a set of text samples from a acquired candidate corpus. The language training unit 502 can be configured to train a language model based on the text sample set and a pre-constructed language network. The language model is used to predict the probability distribution of the next text unit adjacent to the input content being a text in a preset vocabulary, based on the input content in the current input text. The determination unit 503 can be configured to determine the probability that each candidate corpus in the candidate corpus belongs to the target domain based on the language model. The screening unit 504 can be configured to screen the candidate corpus to obtain a first screening result based on the probability that each candidate corpus belongs to the target domain.

[0095] In this embodiment, the specific processing and technical effects of the corpus screening device 500, including the set selection unit 501, language training unit 502, determination unit 503, and screening unit 504, can be found in reference [reference needed]. Figure 1 The relevant descriptions of steps 101, 102, 103, and 104 in the corresponding embodiments will not be repeated here.

[0096] In some optional implementations of this embodiment, the language network is a decoder, and the language training unit 502 is configured to: select text samples from the text sample set, the text samples including at least one unit sample; input the start symbol plus the selected text sample and the end symbol into the decoder in sequence to obtain the probability distribution corresponding to each unit sample in the selected text samples output by the decoder, the probability distribution being the prediction probability of the text unit in the text sample set being the prediction result; and obtain the language model in response to the decoder satisfying the training completion condition.

[0097] In some optional implementations of this embodiment, the determining unit 503 can be configured to: input the start symbol, the candidate corpus, and the end symbol into the language model sequentially for each candidate corpus in the candidate corpus set to obtain the probability distribution of each candidate character corresponding to the candidate corpus; obtain the probability of each candidate character based on the probability distribution of each candidate character; and calculate the probability that the candidate corpus belongs to the target domain based on the probability of all candidate characters in the candidate corpus.

[0098] In some optional implementations of this embodiment, the determining unit 503 is further configured to multiply the probabilities of all candidate characters in the candidate corpus to obtain the probability that the candidate corpus belongs to the target domain.

[0099] In some optional implementations of this embodiment, the filtering unit 504 is configured to: in response to the probability that the candidate corpus in the candidate corpus set belongs to the target domain being greater than or equal to a probability threshold, use the candidate corpus as the candidate corpus in the first filtering result.

[0100] In some optional implementations of this embodiment, the apparatus 500 further includes: a similarity training unit (not shown in the figure), a sample selection unit (not shown in the figure), a similarity acquisition unit (not shown in the figure), and a result acquisition unit (not shown in the figure). The similarity training unit can be configured to train a similarity comparison model based on a text sample set and a text set outside the target domain. The similarity comparison model is used to determine the similarity between two input texts. The sample selection unit can be configured to select text samples from the text sample set. The similarity acquisition unit can be configured to input the selected text samples and each candidate corpus from the first screening result into the similarity comparison model to obtain the similarity between each candidate corpus and the text sample. The result acquisition unit can be configured to obtain a second screening result based on the similarity.

[0101] In some optional implementations of this embodiment, the aforementioned similarity training unit is configured to: select any two text samples from the text sample set and concatenate them to obtain a positive sample; randomly select text from the text set outside the target domain, and concatenate one text sample from the text sample set with the selected text to obtain a negative sample; train a similarity comparison network based on the positive and negative samples, wherein the input of the similarity comparison network is the two types of text, and the output is the similarity value between the two types of text; and obtain a similarity comparison model in response to the similarity comparison network meeting the training completion condition.

[0102] In some optional implementations of this embodiment, the apparatus 500 further includes: an entity recognition unit (not shown in the figure), a calculation unit (not shown in the figure), and a proportion obtaining unit (not shown in the figure). The entity recognition unit can be configured to perform named entity recognition on the candidate corpora in the second filtering result to obtain named entity sequences in each candidate corpus. The calculation unit can be configured to calculate the proportion value of graph entities in the named entity sequences of each candidate corpus based on a pre-constructed knowledge graph of the target domain. The proportion obtaining unit can be configured to filter the second filtering result based on the proportion value to obtain a third filtering result.

[0103] In some optional implementations of this embodiment, the apparatus 500 further includes: a corpus training unit (not shown in the figure), a probability acquisition unit (not shown in the figure), and a filtering acquisition unit (not shown in the figure). The corpus training unit can be configured to train a corpus filtering model based on a text sample set and a text set outside the target domain. The corpus filtering model is used to determine the probability that the input text belongs to the target domain. The probability acquisition unit can be configured to input each candidate corpus from the first filtering result into the corpus filtering model to obtain the probability that each candidate corpus belongs to the target domain. The filtering acquisition unit can be configured to obtain a fourth filtering result based on the probability that each candidate corpus belongs to the target domain.

[0104] The corpus filtering apparatus provided in the embodiments of this disclosure firstly involves a set selection unit 501 selecting a text sample set of the target domain from an acquired candidate corpus set; secondly, a language training unit 502 training a language model based on the text sample set and a pre-constructed language network, the language model being used to predict the probability distribution of the next text unit adjacent to the input content in the current input text as a text from a preset vocabulary; thirdly, a determination unit 503 determining the probability that each candidate corpus in the candidate corpus belongs to the target domain based on the language model; and finally, a filtering unit 504 filtering the candidate corpus set based on the probability that each candidate corpus belongs to the target domain to obtain a first filtering result. Thus, by training the language model using a text sample set of the target domain, the language model becomes familiar with the language style of the target domain. By using the text probability distribution determined when the language model predicts text, the probability of each candidate corpus in the candidate corpus belonging to the target domain is determined, thereby fully utilizing the recognition performance of the language model to filter corpus in the target domain and improving the accuracy of target domain corpus filtering.

[0105] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0106] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0107] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0108] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0109] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0110] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as corpus screening methods. For example, in some embodiments, the corpus screening method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform corpus screening methods by any other suitable means (e.g., by means of firmware).

[0111] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0112] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable corpus screening device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0113] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0114] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0115] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0116] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0117] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0118] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A corpus screening method, the method comprising: selecting a text sample set of a target domain from a candidate corpus set obtained; training a language model based on the text sample set of the target domain obtained from the candidate corpus set and a pre-constructed language network, the language model being configured to predict a probability distribution of a next text unit adjacent to an input content in a current input text being a preset vocabulary text according to the input content; determining a probability of each candidate corpus in the candidate corpus set belonging to the target domain based on the language model; screening the candidate corpus set based on the probability of each candidate corpus belonging to the target domain to obtain a first screening result.

2. The method of claim 1, wherein, The language network is a decoder, and the training of the language model based on the text sample set and the pre-constructed language network comprises: selecting a text sample from the text sample set, the text sample comprising at least one unit sample; inputting a start symbol, the selected text sample and an end symbol into the decoder in sequence to obtain a probability distribution corresponding to each unit sample in the selected text sample output by the decoder, the probability distribution being a prediction probability of a text unit in the text sample set being a prediction result; obtaining the language model in response to the decoder satisfying a training completion condition.

3. The method of claim 1, wherein, The determination of the probability of each candidate corpus in the candidate corpus set belonging to the target domain based on the language model comprises: for each candidate corpus in the candidate corpus set, inputting a start symbol, the candidate corpus and an end symbol into the language model in sequence to obtain a probability distribution of each candidate word corresponding to the candidate corpus; obtaining a probability of each candidate word based on the probability distribution corresponding to each candidate word; calculating the probability of the candidate corpus belonging to the target domain based on the probabilities of all candidate words of the candidate corpus.

4. The method of claim 3, wherein, The calculation of the probability of the candidate corpus belonging to the target domain based on the probabilities of all candidate words of the candidate corpus comprises: multiplying the probabilities of all candidate words of the candidate corpus to obtain the probability of the candidate corpus belonging to the target domain.

5. The method according to one of claims 1 to 4, wherein The screening of the candidate corpus set based on the probability of each candidate corpus belonging to the target domain to obtain the first screening result comprises: in response to the probability of a candidate corpus in the candidate corpus set belonging to the target domain being greater than or equal to a probability threshold, regarding the candidate corpus as a candidate corpus in the first screening result.

6. The method of claim 1, further comprising: training a similarity comparison model based on the text sample set and a text set of a non-target domain, the similarity comparison model being configured to judge a similarity between two input texts; selecting a text sample from the text sample set; inputting the selected text sample and each candidate corpus in the first screening result into the similarity comparison model to obtain a similarity between each candidate corpus and the text sample; obtaining a second screening result based on the similarity.

7. The method of claim 6, wherein, The training of the similarity comparison model based on the text sample set and the text set of the non-target domain comprises: splicing any two text samples selected from the text sample set to obtain a positive sample. randomly select a text from the text set of the non-target field, splice one text sample in the text sample set and the selected text to obtain a negative sample; train a similarity comparison network based on the positive sample and the negative sample, an input of the similarity comparison network being two texts and an output being a similarity value between the two texts; obtain a similarity comparison model in response to the similarity comparison network satisfying a training completion condition.

8. The method of claim 6 or 7, further comprising: performing named entity recognition on the candidate corpus in the second screening result to obtain a named entity sequence in each candidate corpus; calculating a graph entity proportion value in the named entity sequence of each candidate corpus based on a pre-constructed knowledge graph of the target field; screening the second screening result based on the proportion value to obtain a third screening result.

9. The method of claim 1, comprising: training a corpus screening model based on the text sample set and the text set of the non-target field, the corpus screening model being used to judge a probability that an input text belongs to the target field; inputting each candidate corpus in the first screening result into the corpus screening model to obtain a probability that each candidate corpus belongs to the target field; obtaining a fourth screening result based on the probability that each candidate corpus belongs to the target field.

10. A corpus screening device, comprising: a set selection unit configured to select a text sample set of a target field from an obtained candidate corpus set; a language training unit configured to train a language model based on the text sample set of the target field obtained from the candidate corpus set and a pre-constructed language network, the language model being used to predict a probability distribution of a next text unit adjacent to input content in a current input text being a preset vocabulary text according to the input content; a determination unit configured to determine a probability that each candidate corpus in the candidate corpus set belongs to the target field based on the language model; a screening unit configured to screen the candidate corpus set to obtain a first screening result based on the probability that each candidate corpus belongs to the target field.

11. The apparatus of claim 10, wherein, The language network is a decoder, and the language training unit is configured to: select a text sample from the text sample set, the text sample comprising at least one unit sample; input a start symbol, the selected text sample and an end symbol into the decoder in sequence to obtain a probability distribution corresponding to each unit sample in the selected text sample output by the decoder, the probability distribution being a prediction probability that a text unit in the text sample set is a prediction result; and obtain a language model in response to the decoder satisfying a training completion condition.

12. The apparatus of claim 10, wherein, The determination unit is configured to: for each candidate corpus in the candidate corpus set, input a start symbol, the candidate corpus and an end symbol into the language model in sequence to obtain a probability distribution corresponding to each candidate word of the candidate corpus; obtain a probability of each candidate word based on the probability distribution corresponding to each candidate word; and calculate a probability that the candidate corpus belongs to the target field based on the probabilities of all candidate words of the candidate corpus.

13. The apparatus of claim 12, wherein, The determining unit is further configured to multiply the probabilities of all candidate words in the candidate corpus to obtain a probability that the candidate corpus belongs to the target domain.

14. The apparatus of one of claims 10-13, wherein, The screening unit is configured to, in response to a probability that a candidate corpus in the candidate corpus set belongs to the target domain being greater than or equal to a probability threshold, take the candidate corpus as a candidate corpus in a first screening result.

15. The apparatus of claim 10, further comprising: a similarity training unit configured to train a similarity comparison model based on the text sample set and a text set of a non-target domain, the similarity comparison model being used to determine a similarity between two input texts; a sample selection unit configured to select a text sample from the text sample set; a similarity obtaining unit configured to input the selected text sample and each candidate corpus in the first screening result into the similarity comparison model to obtain a similarity between each candidate corpus and the text sample; a result obtaining unit configured to obtain a second screening result based on the similarities.

16. The apparatus of claim 15, wherein, The similarity training unit is configured to: splice any two text samples in the text sample set to obtain a positive sample; randomly select a text from the text set of the non-target domain, and splice the selected text and a text sample in the text sample set to obtain a negative sample; train a similarity comparison network based on the positive sample and the negative sample, the similarity comparison network having an input of two texts and an output of a similarity value between the two texts; and in response to the similarity comparison network satisfying a training completion condition, obtain a similarity comparison model.

17. The apparatus of claim 15 or 16, further comprising: an entity recognition unit configured to perform named entity recognition on the candidate corpus in the second screening result to obtain a named entity sequence in each candidate corpus; a calculation unit configured to calculate a graph entity proportion value in the named entity sequence of each candidate corpus based on a pre-constructed knowledge graph of the target domain; a proportion obtaining unit configured to screen the second screening result based on the proportion values to obtain a third screening result.

18. The apparatus of claim 10, further comprising: a corpus training unit configured to train a corpus screening model based on the text sample set and a text set of a non-target domain, the corpus screening model being used to determine a probability that an input text belongs to a target domain; a probability obtaining unit configured to input each candidate corpus in the first screening result into the corpus screening model to obtain a probability that each candidate corpus belongs to the target domain; a screening obtaining unit configured to obtain a fourth screening result based on the probability that each candidate corpus belongs to the target domain.

19. An electronic device, comprising: comprising: at least one processor; and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

20. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are for causing the computer to perform the method of any one of claims 1-9.

21. A computer program product comprising computer instructions which, when executed by a processor, implement the method of any one of claims 1-9.

Images