A method, device and terminal device for extracting vertical domain corpus

By calculating the vertical domain score, the target sentences are divided into the vertical domain corpus database, which solves the problem of low efficiency in vertical domain corpus collection, and realizes efficient extraction of vertical domain corpus similar to the target domain characteristics, meeting the accuracy requirements of vertical domain translation services. .

CN115034234BActive Publication Date: 2024-11-22SICHUAN LAN-BRIDGE INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210693426.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-17
Publication Date
2024-11-22
Estimated Expiration
2042-06-17

AI Technical Summary

Technical Problem

The existing methods of collecting vertical corpus are inefficient and cannot effectively extract a large number of potential vertical corpus.

Method used

By obtaining vertical corpus and common corpus, calculate the vertical corpus score, and divide the target sentences into the vertical corpus database or non-vertical corpus database based on the score, evaluate the vertical corpus score using word frequency characteristics and language model characteristics to achieve vertical corpus mates efficient extraction.

Benefits of technology

It can efficiently extract vertical corpus with similar characteristics to the target field, improve the efficiency of new corpus acquisition of vertical corpus databases, and meet the growing demand for vertical translation business.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115034234B_ABST
    Figure CN115034234B_ABST
Patent Text Reader

Abstract

The present invention is applicable to the fields of computational linguistics and translation technology, and provides a vertical domain corpus extraction method, device and terminal device, the method comprising obtaining vertical domain corpus and general corpus; calculating vertical domain scores based on the vertical domain corpus and general corpus; and dividing the target sentence into a vertical domain corpus database or a non-vertical domain corpus database according to the vertical domain score. Through the present invention, vertical domain corpus with similar characteristics to the target domain can be extracted from a large amount of general corpus as new corpus for the vertical domain corpus database.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the fields of computational linguistics and translation technology, and in particular to a vertical domain corpus extraction method, device and terminal equipment. Background Art

[0002] Machine translation can provide enterprises and individuals with the ability to quickly translate between different languages. For machine translation, the use of error correction models can optimize translation results. Language models play a vital role in error correction. Among them, a large factor affecting the effect of language models is data, especially statistical language models. By adjusting the distribution of data sets, such as using corpora in specific vertical domains for training, the final effect can be effectively improved.

[0003] It can be seen that the demand for vertical domain engines for machine translation is increasing, and the accuracy of vertical domain engines is also increasing to meet the growing various vertical domain translation business scenarios. However, the current means of collecting vertical domain corpora are limited to manual acquisition, and it is impossible to extract the corresponding vertical domain corpora from a large number of potential existing corpora. Summary of the invention

[0004] The main purpose of the present invention is to propose a method, device and terminal device for extracting vertical domain corpus to solve the problem of low efficiency of existing vertical domain corpus collection methods.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present invention provides a method for extracting vertical domain corpus, comprising:

[0006] Obtain vertical domain corpus and general corpus;

[0007] Calculating a vertical domain score based on the vertical domain corpus and the general corpus;

[0008] The target sentence is divided into a vertical domain corpus database or a non-vertical domain corpus database according to the vertical domain score.

[0009] In combination with the first aspect of the present invention, in a first implementation of the present invention, calculating the vertical domain score based on the vertical domain corpus and the general corpus includes:

[0010] Performing word segmentation processing on the vertical domain corpus and the general corpus;

[0011] The segmented vertical domain corpus is stored in the vertical domain set, and the segmented universal corpus is stored in the universal set; wherein the vertical domain set includes J vertical domain words, and the universal set includes K universal words, where J and K are positive integers;

[0012] Based on the vertical domain set and the general set, obtaining word scores of J vertical domain words;

[0013] According to the word scores of the J vertical domain words, the vertical domain weight vector in the target sentence is calculated to obtain the vertical domain score of the target sentence.

[0014] In combination with the first embodiment of the first aspect of the present invention, in the second embodiment of the present invention, based on the vertical domain set and the general set, obtaining word scores of J vertical domain words includes:

[0015] The jth vertical domain word in the vertical domain set is represented by the first count value, and the kth general word in the general set is represented by the second count value; wherein the first count value = the total count of the jth vertical domain word / the total count of the vertical domain set words, the second count value = the total count of the kth general word / the total count of the general set words, j is a positive integer less than or equal to J, and k is a positive integer less than or equal to K;

[0016] Divide the first count value corresponding to the j-th vertical domain word by the second count value corresponding to the k-th common word to obtain the word score of the j-th vertical domain word.

[0017] In combination with the second embodiment of the first aspect of the present invention, in the third embodiment of the present invention, according to the word scores of the J vertical domain words, the vertical domain weight vector in the target sentence is calculated to obtain the vertical domain score of the target sentence, including:

[0018] Performing word segmentation processing on the target sentence to obtain a word segmentation result;

[0019] According to the word scores of the J vertical domain words, calculate the vertical domain weight vector of each word in the word segmentation result;

[0020] The vertical domain weight vectors of all words in the word segmentation result are counted to obtain the vertical domain score of the target sentence.

[0021] Among them, the vertical domain weight vectors of all words in the word segmentation result are counted to obtain the vertical domain score of the target sentence. The formula is:

[0022] s_score=w_score(w) / word_sum;

[0023] Among them, s_score represents the vertical domain score of the target sentence, w_score(w) represents the sum of the vertical domain weight vectors of all words in the word segmentation result, and word_sum represents the total number of words in the word segmentation result.

[0024] In combination with the first aspect of the present invention, in a fourth implementation of the present invention, calculating the vertical domain score based on the vertical domain corpus and the general corpus includes:

[0025] Performing word segmentation processing on the vertical domain corpus and the general corpus, and retaining the word association relationship between the vertical domain corpus and the general corpus;

[0026] The segmented vertical domain corpus is stored in a vertical domain set, and the segmented general corpus is stored in a general set; wherein the vertical domain set includes H vertical domain sentences, each vertical domain sentence includes a plurality of vertical domain words and word associations between the vertical domain words, and the general set includes L general sentences, each general sentence includes a plurality of general words and word associations between the general words, and H and L are positive integers;

[0027] Based on the vertical domain set and the universal set, obtaining a vertical domain Ngram language model and a universal Ngram language model;

[0028] The vertical domain score of the target sentence is calculated by using the vertical domain Ngram language model and the general Ngram language model.

[0029] In combination with the fourth implementation manner of the first aspect of the present invention, in a fifth implementation manner of the present invention, based on the vertical domain set and the universal set, obtaining a vertical domain Ngram language model and a universal Ngram language model includes:

[0030] Use kenlm to perform Ngram language model training on the vertical domain set and the universal set to obtain a vertical domain Ngram language model and a universal Ngram language model.

[0031] In combination with the fifth implementation of the first aspect of the present invention, in a sixth implementation of the present invention, the vertical domain score of the target sentence is calculated by using the vertical domain Ngram language model and the general Ngram language model, including:

[0032] Input the target sentence into the vertical domain Ngram language model and the universal Ngram language model to obtain a vertical domain perplexity score and a universal perplexity score of the target sentence;

[0033] The vertical domain score of the target sentence is obtained through the vertical domain perplexity score and the universal perplexity score.

[0034] Among them, the vertical domain perplexity score and the universal perplexity score of the target sentence are obtained, and the formula is:

[0035]

[0036]

[0037] Where N represents N in Ngram language model training, M represents the length of the target sentence, w represents the target sentence, PP_g(w) is the perplexity of w sentence in the general Ngram model, and PP_d(w) is the perplexity of the sentence in the vertical domain Ngram model;

[0038] The vertical domain score of the target sentence is obtained by using the vertical domain perplexity score and the universal perplexity score, and the formula is:

[0039] s_score=ln(PP_g(w))-ln(PP_d(w));

[0040] Among them, s_score represents the vertical domain score of the target sentence.

[0041] A second aspect of the present invention provides a vertical domain corpus extraction device, comprising:

[0042] Corpus acquisition module, used to acquire vertical domain corpus and general corpus;

[0043] A vertical domain score calculation module, used to calculate a vertical domain score based on the vertical domain corpus and the general corpus;

[0044] A corpus partitioning module is used to partition the target sentence into a vertical domain corpus database or a non-vertical domain corpus database according to the vertical domain score.

[0045] The third aspect of an embodiment of the present invention provides a terminal device, including a memory, a processor, and a computer program stored in the above memory and executable on the above processor, and when the above processor executes the above computer program, the steps of the method provided in the first aspect are implemented.

[0046] A fourth aspect of an embodiment of the invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect above are implemented.

[0047] The embodiment of the present invention proposes a vertical domain corpus extraction method, which analyzes the vertical domain score of the target sentence to determine whether the target sentence can be classified into the vertical domain corpus database as a new corpus for the vertical domain corpus database. When a large number of target sentences are extracted from the general corpus, the vertical domain corpus extraction method provided by the embodiment of the present invention can efficiently extract vertical domain corpora with similar characteristics to the target domain as new corpora for the vertical domain corpus database. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 A schematic diagram of the implementation process of the vertical domain corpus extraction method provided in an embodiment of the present invention;

[0049] Figure 2A schematic diagram of a score distribution density curve of a vertical domain corpus and a general corpus provided in an embodiment of the present invention;

[0050] Figure 3 A schematic diagram of another implementation flow of the vertical domain corpus extraction method provided in an embodiment of the present invention;

[0051] Figure 4 A schematic diagram of another implementation flow of the vertical domain corpus extraction method provided in an embodiment of the present invention;

[0052] Figure 5 A schematic diagram of the composition structure of a vertical domain corpus extraction device provided in an embodiment of the present invention.

[0053] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0054] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0055] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0056] In this document, suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present invention, and have no specific meanings. Therefore, "module" and "component" can be used interchangeably.

[0057] like Figure 1 As shown, an embodiment of the present invention provides a vertical domain corpus extraction method, which can extract vertical domain corpus with similar characteristics to the target domain from a large amount of general corpus to serve as new corpus of the vertical domain corpus database, and the method includes:

[0058] S101. Obtain vertical domain corpus and general corpus.

[0059] In the embodiment of the present invention, the vertical domain corpus is the corpus data belonging to a predefined vertical field, and the general corpus is the corpus data belonging to any field. The vertical domain corpus and the general corpus have overlapping fields.

[0060] In one embodiment, the vertical domain corpus may be at least 10,000 data points, and the general corpus may be at least 100,000 data points.

[0061] In the above step S101, the vertical domain corpus and the general corpus can be in Chinese, English or other languages. S102: Calculate a vertical domain score based on the vertical domain corpus and the general corpus.

[0062] The embodiment of the present invention provides two vertical domain score calculation schemes, one using word frequency features to evaluate the vertical domain score value, and the other using language model features to evaluate the vertical domain score value.

[0063] S103: Divide the target sentence into a vertical domain corpus database or a non-vertical domain corpus database according to the vertical domain score.

[0064] In the above step S103, when the vertical domain score is greater than or equal to a preset value, the target sentence is classified into the vertical domain corpus database; when the vertical domain score is less than the preset value, the target sentence is classified into the non-vertical domain corpus database.

[0065] In specific applications, the preset value can be set by drawing the score distribution density curve of the vertical domain corpus and the score distribution density curve of the general corpus, finding the obvious gap between the two and setting the threshold, such as Figure 2 In the figure, the x-axis represents the score, the y-axis represents the score distribution density, curve y1 is the score distribution density curve of the vertical domain corpus, and y2 is the score distribution density curve of the general corpus. According to y1 and y2, 0.2 can be used as the threshold.

[0066] like Figure 3 As shown, in an embodiment of the present invention, one implementation of step S102 is:

[0067] S10211, performing word segmentation processing on the vertical domain corpus and the general corpus;

[0068] S10212, storing the segmented vertical domain corpus into the vertical domain set, and storing the segmented general corpus into the general set;

[0069] Wherein, the vertical domain set includes J vertical domain words, the universal set includes K universal words, and J and K are positive integers;

[0070] S10213. Based on the vertical domain set and the general set, obtain word scores of J vertical domain words;

[0071] S10214. Calculate the vertical domain weight vector in the target sentence according to the word scores of the J vertical domain words to obtain the vertical domain score of the target sentence.

[0072] In the above steps S10211 to S10214, the word score is used to represent the word frequency feature of the vertical domain word. When the word scores of J vertical domain words are obtained, the word score is used to obtain the word frequency feature. At this time, the word score is actually the frequency of occurrence of the vertical domain word based on the common words.

[0073] Among them, one implementation method of obtaining the word scores of the J vertical domain words may be:

[0074] The jth vertical domain word in the vertical domain set is represented by the first count value, and the kth general word in the general set is represented by the second count value; wherein the first count value = the total count of the jth vertical domain word / the total count of the vertical domain set words, the second count value = the total count of the kth general word / the total count of the general set words, j is a positive integer less than or equal to J, and k is a positive integer less than or equal to K;

[0075] Divide the first count value corresponding to the j-th vertical domain word by the second count value corresponding to the k-th common word to obtain the word score of the j-th vertical domain word.

[0076] It should be noted that the count of the jth vertical domain word represents a vertical domain word in the vertical domain set. For example, if the vertical domain set includes the words abcdeddd, and abcd is a vertical domain word, then the count of the 4th vertical domain word is 1, and the total count of the 4th vertical domain word is 4.

[0077] It should be noted that in the above steps, in order to avoid the division error caused by the second count value being 0, and also to smooth the error caused by missing corpus, when dividing the first count value corresponding to the j-th vertical domain word by the second count value corresponding to the k-th common word, corresponding smoothing values ​​are added to the numerator and denominator respectively, such as 1 / si and 1 / sg, and finally the word score of the j-th vertical domain word is calculated.

[0078] It should be noted that the j-th vertical domain word in the vertical domain set has a corresponding relationship with the k-th general word in the general set, and the corresponding relationship means that the two words are the same or have similar semantics.

[0079] Based on this, the detailed implementation steps of the above step S10214 include:

[0080] Performing word segmentation processing on the target sentence to obtain a word segmentation result;

[0081] According to the word scores of the J vertical domain words, calculate the vertical domain weight vector of each word in the word segmentation result;

[0082] The vertical domain weight vectors of all words in the word segmentation result are counted to obtain the vertical domain score of the target sentence.

[0083] It should be noted that if there are words belonging to the vertical domain set in the target sentence, the vertical domain weight vector in the target sentence is calculated based on the word scores of the J vertical domain words; if there are no words belonging to the vertical domain set in the target sentence, the weight of this word is set to 0.

[0084] Among them, the vertical domain weight vectors of all words in the word segmentation result are counted to obtain the vertical domain score of the target sentence. The formula is:

[0085] s_score=w_score(w) / word_sum;

[0086] Among them, s_score represents the vertical domain score of the target sentence, w_score(w) represents the sum of the vertical domain weight vectors of all words in the word segmentation result, and word_sum represents the total number of words in the word segmentation result.

[0087] like Figure 4 As shown, in an embodiment of the present invention, another implementation of the above step S102 is:

[0088] S10221, performing word segmentation processing on the vertical domain corpus and the general corpus, and retaining the word association relationship between the vertical domain corpus and the general corpus;

[0089] S10222. Store the segmented vertical domain corpus in the vertical domain set, and store the segmented general corpus in the general set;

[0090] Wherein, the vertical domain set includes H vertical domain sentences, each vertical domain sentence includes multiple vertical domain words and word association relationships between the vertical domain words, the universal set includes L universal sentences, each universal sentence includes multiple universal words and word association relationships between the universal words, and H and L are positive integers;

[0091] S10223. Based on the vertical domain set and the universal set, obtain a vertical domain Ngram language model and a universal Ngram language model;

[0092] In an embodiment of the present invention, kenlm is used to perform Ngram language model training on the vertical domain set and the universal set to obtain a vertical domain Ngram language model and a universal Ngram language model.

[0093] S10224. Calculate the vertical domain score of the target sentence using the vertical domain Ngram language model and the general Ngram language model.

[0094] In the above steps S10221 to S10224, the outputs of the vertical domain Ngram language model and the universal Ngram language model are used to represent the language model features of the target sentence, which are the vertical domain features and universal features of each word in the target sentence, and the vertical domain features and universal features of the entire target sentence obtained by processing the target sentence in combination with the word association relationship between each vertical domain word in the vertical domain set and the word association relationship between each universal word in the universal set. After the target sentence is input into the vertical domain Ngram language model and the universal Ngram language model, the outputs of the vertical domain Ngram language model and the universal Ngram language model are combined to calculate the vertical domain score in the target sentence, which is actually to calculate the possibility that the entire target sentence is different from the universal sentence.

[0095] Based on this, the implementation of the above step S10224 may be:

[0096] Input the target sentence into the vertical domain Ngram language model and the universal Ngram language model to obtain a vertical domain perplexity score and a universal perplexity score of the target sentence;

[0097] The vertical domain score of the target sentence is obtained through the vertical domain perplexity score and the universal perplexity score.

[0098] In the embodiment of the present invention, the calculation formula of the vertical domain perplexity score and the general perplexity score is the same, which is:

[0099]

[0100]

[0101] Among them, N represents N in Ngram language model training, M represents the length of the target sentence, w represents the target sentence, PP_g(w) is the perplexity of sentence w in the general Ngram model, and PP_d(w) is the perplexity of the sentence in the vertical domain Ngram model.

[0102] It should be noted that the N in the Ngram language model training is consistent with the Ngram selected by the user, such as Unigram represents N=1, Bigram represents N=2, Trigram represents N=3, and so on.

[0103] Among them, the vertical domain score of the target sentence is obtained by the vertical domain perplexity score and the universal perplexity score, and the formula is:

[0104] s_score=ln(PP_g(w))-ln(PP_d(w));

[0105] Among them, s_score represents the vertical domain score of the target sentence.

[0106] like Figure 5 As shown, the embodiment of the present invention further provides a vertical domain corpus extraction device 50, comprising:

[0107] Corpus acquisition module 51, used to acquire vertical domain corpus and general corpus;

[0108] A vertical domain score calculation module 52, configured to calculate a vertical domain score based on the vertical domain corpus and the general corpus;

[0109] The corpus division module 53 is used to divide the target sentence into a vertical domain corpus database or a non-vertical domain corpus database according to the vertical domain score.

[0110] An embodiment of the present invention also provides a terminal device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the vertical domain corpus extraction method described in the above embodiment is implemented.

[0111] An embodiment of the present invention further provides a storage medium, which is a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, each step in the vertical domain corpus extraction method described in the above embodiment is implemented.

[0112] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the foregoing embodiments describe the present invention in detail, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or replace some of the technical features therein by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A vertical domain corpus extraction method, characterized in that: include: Obtain vertical domain corpus and general corpus; Calculating a vertical domain score based on the vertical domain corpus and the general corpus; Dividing the target sentence into a vertical domain corpus database or a non-vertical domain corpus database according to the vertical domain score; Calculating a vertical domain score based on the vertical domain corpus and the general corpus includes: Performing word segmentation processing on the vertical domain corpus and the general corpus, and retaining the word association relationship between the vertical domain corpus and the general corpus; The segmented vertical domain corpus is stored in a vertical domain set, and the segmented general corpus is stored in a general set; wherein the vertical domain set includes H vertical domain sentences, each vertical domain sentence includes a plurality of vertical domain words and word associations between the vertical domain words, and the general set includes L general sentences, each general sentence includes a plurality of general words and word associations between the general words, and H and L are positive integers; Based on the vertical domain set and the universal set, obtaining a vertical domain Ngram language model and a universal Ngram language model; Calculate the vertical domain score of the target sentence by using the vertical domain Ngram language model and the general Ngram language model; Based on the vertical domain set and the universal set, a vertical domain Ngram language model and a universal Ngram language model are obtained, including: Using kenlm, perform Ngram language model training on the vertical domain set and the universal set to obtain a vertical domain Ngram language model and a universal Ngram language model; The vertical domain score of the target sentence is calculated by using the vertical domain Ngram language model and the general Ngram language model, including: Input the target sentence into the vertical domain Ngram language model and the universal Ngram language model to obtain a vertical domain perplexity score and a universal perplexity score of the target sentence; Obtaining a vertical domain score of the target sentence through the vertical domain perplexity score and the universal perplexity score; Among them, the vertical domain perplexity score and the universal perplexity score of the target sentence are obtained, and the formula is: Where N represents N in Ngram language model training, M represents the length of the target sentence, W represents the target sentence, PP_g(W) is the perplexity of W sentence in the general Ngram language model, and PP_d(W) is the perplexity of the sentence in the vertical domain Ngram language model; The vertical domain score of the target sentence is obtained by using the vertical domain perplexity score and the universal perplexity score, and the formula is: s_score=ln(PP_g(W))-ln(PP_d(W)); Among them, s_score represents the vertical domain score of the target sentence.

2. The vertical domain corpus extraction method according to claim 1, characterized in that: Calculating a vertical domain score based on the vertical domain corpus and the general corpus includes: Performing word segmentation processing on the vertical domain corpus and the general corpus; The segmented vertical domain corpus is stored in the vertical domain set, and the segmented universal corpus is stored in the universal set; wherein the vertical domain set includes J vertical domain words, and the universal set includes K universal words, where J and K are positive integers; Based on the vertical domain set and the general set, obtaining word scores of J vertical domain words; According to the word scores of the J vertical domain words, the vertical domain weight vector in the target sentence is calculated to obtain the vertical domain score of the target sentence.

3. The vertical domain corpus extraction method according to claim 2, characterized in that: Based on the vertical domain set and the general set, word scores of J vertical domain words are obtained, including: The jth vertical domain word in the vertical domain set is represented by the first count value, and the kth general word in the general set is represented by the second count value; wherein the first count value = the total count of the jth vertical domain word / the total count of the vertical domain set words, the second count value = the total count of the kth general word / the total count of the general set words, j is a positive integer less than or equal to J, and k is a positive integer less than or equal to K; Divide the first count value corresponding to the j-th vertical domain word by the second count value corresponding to the k-th common word to obtain the word score of the j-th vertical domain word.

4. The vertical domain corpus extraction method according to claim 3, characterized in that: Calculating the vertical domain weight vector in the target sentence according to the word scores of the J vertical domain words to obtain the vertical domain score of the target sentence includes: Performing word segmentation processing on the target sentence to obtain a word segmentation result; According to the word scores of the J vertical domain words, calculate the vertical domain weight vector of each word in the word segmentation result; Counting the vertical domain weight vectors of all words in the word segmentation result to obtain the vertical domain score of the target sentence; Among them, the vertical domain weight vectors of all words in the word segmentation result are counted to obtain the vertical domain score of the target sentence. The formula is: s_score=w_score(w) / word_sum; Among them, s_score represents the vertical domain score of the target sentence, w_score(w) represents the sum of the vertical domain weight vectors of all words in the word segmentation result, and word_sum represents the total number of words in the word segmentation result.

5. A vertical domain corpus extraction device, characterized in that: include: Corpus acquisition module, used to acquire vertical domain corpus and general corpus; A vertical domain score calculation module, used to calculate a vertical domain score based on the vertical domain corpus and the general corpus; A corpus partitioning module, configured to partition the target sentence into a vertical domain corpus database or a non-vertical domain corpus database according to the vertical domain score; Calculating a vertical domain score based on the vertical domain corpus and the general corpus includes: Performing word segmentation processing on the vertical domain corpus and the general corpus, and retaining the word association relationship between the vertical domain corpus and the general corpus; The segmented vertical domain corpus is stored in a vertical domain set, and the segmented general corpus is stored in a general set; wherein the vertical domain set includes H vertical domain sentences, each vertical domain sentence includes a plurality of vertical domain words and word associations between the vertical domain words, and the general set includes L general sentences, each general sentence includes a plurality of general words and word associations between the general words, and H and L are positive integers; Based on the vertical domain set and the universal set, obtaining a vertical domain Ngram language model and a universal Ngram language model; Calculate the vertical domain score of the target sentence by using the vertical domain Ngram language model and the general Ngram language model; Based on the vertical domain set and the universal set, a vertical domain Ngram language model and a universal Ngram language model are obtained, including: Using kenlm, perform Ngram language model training on the vertical domain set and the universal set to obtain a vertical domain Ngram language model and a universal Ngram language model; The vertical domain score of the target sentence is calculated by using the vertical domain Ngram language model and the general Ngram language model, including: Input the target sentence into the vertical domain Ngram language model and the universal Ngram language model to obtain a vertical domain perplexity score and a universal perplexity score of the target sentence; Obtaining a vertical domain score of the target sentence through the vertical domain perplexity score and the universal perplexity score; Among them, the vertical domain perplexity score and the universal perplexity score of the target sentence are obtained, and the formula is: Where N represents N in Ngram language model training, M represents the length of the target sentence, W represents the target sentence, PP_g(W) is the perplexity of W sentence in the general Ngram language model, and PP_d(W) is the perplexity of the sentence in the vertical domain Ngram language model; The vertical domain score of the target sentence is obtained by using the vertical domain perplexity score and the universal perplexity score, and the formula is: s_score=ln(PP_g(W))-ln(PP_d(W)); Among them, s_score represents the vertical domain score of the target sentence.

6. A terminal device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, each step in the vertical domain corpus extraction method as described in any one of claims 1 to 4 is implemented.

7. A storage medium, the storage medium being a computer-readable storage medium, on which a computer program is stored, characterized in that: When the computer program is executed by a processor, each step in the vertical domain corpus extraction method as described in any one of claims 1 to 4 is implemented.