A service-specific domain language model modeling method and a modeling system

Abstract

The application belongs to the technical field of artificial intelligence language information processing, and particularly relates to a service specific field scarce resource language model modeling method, which comprises the following steps: step 1) obtaining a preprocessed sentence set composed of multiple segmented sentences; step 2) obtaining a sentence-topic distribution and a topic-word distribution; step 3) obtaining selected field characteristic words; step 4) obtaining translated field characteristic words; step 5) calculating and obtaining a field topic distribution by using the translated field characteristic words; step 6) calculating the correlation between the sentence-topic distribution and the field topic distribution of each segmented sentence; step 7) judging whether the correlation of each segmented sentence exceeds a preset threshold value, and selecting the sentences meeting the condition into the language model training corpus of the scarce resource specific field; step 8) obtaining the final language model training corpus; and step 9) training the scarce resource language model to obtain the scarce resource language model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence language information processing and speech recognition technology. Specifically, it relates to a method and system for modeling language models of scarce resources serving a specific domain. Background Technology

[0002] Speech recognition refers to the computer process of processing received speech signals, analyzing and recognizing them, and converting human speech signals into corresponding text or commands. Due to the dynamic, time-varying, transient, and random nature of speech signals, acoustic analysis and matching alone cannot achieve satisfactory recognition results. Therefore, it is necessary to combine speech signal processing with relevant linguistic knowledge for constraints and processing to improve the system's accuracy. Thus, a typical speech recognition system includes an acoustic model and a language model. The language model is used to characterize the inherent rules in natural language, providing contextual and semantic information between characters or words, and is a crucial component of a speech recognition system.

[0003] In language modeling, the dominant approach remains the statistical rule-based n-gram modeling technique. This technique boasts excellent modeling capabilities and is relatively simple to implement. With sufficient corpus, it can train high-performing models and has led to the development of effective data smoothing techniques suitable for practical applications. However, to overcome the weaknesses of n-gram modeling, researchers have proposed neural network-based language models to better describe the relationships between words. Neural network language models also require statistical learning from corpora; the corpus, as the foundation of knowledge, is crucial in neural network language modeling. Therefore, collecting, organizing, and training corpora for specific domains of language model applications will undoubtedly improve the modeling efficiency and performance.

[0004] The rapid development of language intelligence processing technology has further expanded the range of languages ​​applicable to language information technology applications, from resource-rich languages ​​such as Chinese and English to languages ​​with scarce resources such as Vietnamese and Kazakh. With the deep integration of information technology into social life, scarce resources are increasingly manifested as a scarcity of expert resources—that is, a lack of language experts in the corresponding languages ​​who can process the initial language resources to a certain extent according to the needs of the application, thereby improving the modeling efficiency and performance of language models. This scarcity of expert resources is even more pronounced when performing language modeling for specific domains.

[0005] On the other hand, resource-rich languages ​​such as Chinese and English have already accumulated considerable language resources and research findings. In particular, for Chinese, domestic researchers are able to effectively balance expert resources. Therefore, constructing a language modeling method and system that uses Chinese or other resource-rich languages ​​as a reference, targets languages ​​with scarce resources, and serves specific domains has practical value and significant importance.

[0006] Furthermore, significant progress has been made in topic semantic analysis in text processing in recent years, forming a new model for studying semantic content using language resources based on statistical distributions. Among these, topic analysis models based on Latent Dirichlet Allocation (LDA) distributions have been widely used in research, providing new insights into incorporating semantic information into language modeling for languages ​​with scarce resources.

[0007] In summary, existing language models struggle to construct language models for languages ​​with scarce resources serving specific domains, have low accuracy in acquiring semantic information from scarce resource corpora, cannot identify domain-specific information in scarce resource corpora, and have poor performance in serving specific domain applications. Summary of the Invention

[0008] To address the aforementioned shortcomings of existing technologies, this invention proposes a method for modeling scarce resource language models serving a specific neighborhood. The scarce language model established using this method can accurately acquire the topical semantic information of a specific scarce resource corpus within a particular domain. Furthermore, this method utilizes topical semantic processing to obtain the topical semantic information of the scarce resource language corpus, and then uses this topical semantic information to select corpora from specific domains for language modeling, thereby improving the modeling efficiency and processing performance of language models for scarce resource languages.

[0009] This invention provides a method for modeling a language model of scarce resources serving a specific domain, the method comprising:

[0010] Step 1) Collect scarce resource texts as initial corpus, preprocess the initial corpus to obtain preprocessed text; divide the preprocessed text into sentences, retain the order of sentences and paragraph information, and obtain a preprocessed sentence set composed of multiple segmented sentences.

[0011] Step 2) Using the Gibbs algorithm, the preprocessed set of statements is used to obtain the statement-topic distribution and topic-word distribution;

[0012] Step 3) In the pre-collected rich resource language texts, select domain feature words for a specific domain to obtain the selected domain feature words;

[0013] Step 4) Translate the selected domain feature words into words in rare languages ​​to obtain the translated domain feature words;

[0014] Step 5) Using the translated domain feature words, combined with the topic-word distribution obtained in Step 2), calculate and obtain the domain topic distribution;

[0015] Step 6) Calculate the correlation between the statement-topic distribution and the domain topic distribution for each segmented statement;

[0016] Step 7) Determine whether the relevance of each segmented statement exceeds the preset threshold, and select statements that meet the conditions into the language model training corpus for a scarce resource specific domain;

[0017] Step 8) Based on the coherence of the language expression, further filter the language model training data obtained in Step 7) to obtain the final language model training data;

[0018] Step 9) Use the final language model training corpus obtained in Step 8) to train the scarce resource language model and obtain the scarce resource language model.

[0019] As an improvement to the above technical solution, step 2) specifically includes:

[0020] The preprocessed statement set is represented by M segmented statements, denoted as S = {s1, ..., s2}. m ,...,s M}, where the Mth segmentation statement s M It is a word sequence of length N, denoted as s. M =(w1,…,w n ,...,w N ), where w n This represents the nth word in a word sequence;

[0021] For a word in the segmented sentence, a topic is randomly assigned from the K topic distributions in the range of [1…K] to form an initial Markov chain. The above process is repeated to assign a corresponding topic to all words in the segmented sentence and obtain the next state of the Markov chain. After multiple iterations, the Markov chain reaches a stable state.

[0022] Using the Gibbs sampling method, the topic-word distribution was obtained:

[0023]

[0024] And statement-topic distribution:

[0025]

[0026] in, Let w be the estimated value of the topic-word distribution, and w be the token for a new word extracted from topic j. n Probability estimation; C VK and C MK These are quantity matrices of dimensions V×K and M×K, respectively; where V is the total number of words; For the word w -n The frequency of occurrences of topic j; For the corresponding word w n The value for topic j is set to 0.02; For the word w v Frequency of belonging to topic j; β v,j For the corresponding word w v The value for topic j is set to 0.02;

[0027] in, The estimated value of the statement-topic distribution is given by the statement s. m Probability estimation of extracting new words from topic j; For specific delimiter statements s m The number of words assigned to topic j; α m,j For the corresponding split statement s m The value belongs to topic j; For specific delimiter statements s m The number of words assigned to topic k; α m,k For the corresponding split statement s m The value belonging to topic k.

[0028] As one improvement to the above technical solution, step 3) specifically includes:

[0029] Step 3-1) Collect rich resource language texts and use them as training corpus, and select texts involving specific domain content as the basis for extracting domain feature words;

[0030] Step 3-2) Calculate the information gain value G(h) of the words. This information gain value is the difference between the entropy of the document without considering any word features and the entropy of the document after considering word features.

[0031]

[0032] Where E(S) is the document entropy without considering any word features; E(S) h P(C) represents the document entropy after considering word features; j ) is C j P(h) is the probability of a document of class h appearing in the training corpus; P(C) is the probability of a document in the training corpus containing the word feature word h;j |h) is a document containing the word feature word h and belongs to C. j Conditional probability of class documents; To train the probability of documents in the corpus that do not contain the word feature word h; When the document does not contain the word feature word h and belongs to C j The conditional probability of a document class; M represents the total number of classifications for the training corpus, M=2, dividing the training corpus into two categories: domain-specific and non-application-specific;

[0033] Step 3-3) Repeat step 3-2) Calculate the information gain value of all words in each text involving specific domain content in the training corpus as feature words, and sort all words in each text involving specific domain content in descending order of information gain value. Based on experience, select the top 500 to 10,000 words as selected domain feature words.

[0034] As an improvement to the above technical solution, step 5) specifically includes:

[0035] The sequence of domain-specific words in the translated scarce resource language is denoted as the translated domain-specific words, and it is represented as E = {e1, ..., e}. n ,…,e N}; where e n This represents the nth domain feature word in the domain feature word sequence, where N represents the total number of domain feature words; e n Estimates of topic-word distribution As the topic distribution vector of words

[0036]

[0037] Among them, z i Domain-specific words e n Distribution value on the i-th topic; z K Domain-specific words e n Distribution value on the Kth topic;

[0038] Domain Topic Distribution D T for:

[0039] D T =[t1,...,t i ,...,t K ]

[0040] Among them, t i t represents the distribution value of a specific application domain on the i-th topic; K This represents the distribution value of the Kth topic in a specific application area;

[0041]

[0042] Among them, z ij Domain-specific words e j Distribution value on the j-th topic.

[0043] As one improvement to the above technical solution, step 6) specifically includes:

[0044] Suppose the preprocessed statement set S = {s1,...,s...} i ,...,s M}, where M is the number of statements, s i For the i-th segmented statement in the preprocessed statement set; let s i exist Sentence-topic distribution as a numerical representation of word LDA vectors:

[0045] s i =[y1,...,y j ,...,y K ]

[0046] Among them, y j s i Distribution value on the j-th topic;

[0047] The cosine similarity method is used to calculate the correlation C(s) between the statement-topic distribution and the domain-topic distribution. i ):

[0048]

[0049] Among them, y j For s i The component value of the j-th topic in the statement-topic distribution; t j Let be the component value of the j-th topic in the domain topic distribution;

[0050] Repeat the above process to obtain the correlation between the statement-topic distribution and the domain topic distribution for each segmented statement.

[0051] As an improvement to the above technical solution, step 7) specifically includes:

[0052] Determine whether the relevance of each statement exceeds a preset threshold.

[0053] If the relevance of a statement is greater than a preset threshold, then statements that meet the criteria will be included in the language model training corpus for a scarce resource specific domain.

[0054] If the relevance of a statement is less than or equal to a preset threshold, then the statement is deleted.

[0055] As one improvement to the above technical solution, step 8) specifically includes:

[0056] Step 8-1) If the selected delimiter is at the beginning of a paragraph, do not select the text preceding that delimiter;

[0057] If the selected segmentation statement is not at the beginning of the paragraph, then the text preceding the segmentation statement is selected as the candidate corpus and included in the language model training corpus for the scarce resource specific domain.

[0058] Step 8-2) If the selected delimiter is located at the end of a paragraph, then do not select the text following that delimiter;

[0059] If the selected segmentation statement is not located at the end of the paragraph, then the text following the segmentation statement is selected as the candidate corpus and included in the training corpus of the language model for the scarce resource specific domain.

[0060] Step 8-3) Based on Step 8-1) and Step 8-2) above, the language model training data obtained in Step 7) is screened again and deduplicated to obtain the final language model training data.

[0061] Specifically, the two candidate corpora obtained above are deduplicated. If they are already in the training corpora obtained in step 7), they are not counted again. The deduplicated candidate corpora are then included in the training corpora.

[0062] As one improvement to the above technical solution, step 9) specifically includes:

[0063] The modeling method uses the n-Gram language model, where n can be 2, 3, or 4:

[0064] 2-Gram language model:

[0065] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 )

[0066] Where p(S) represents the generation probability of the segmented statement S; p(w1w2...w n (w) represents the nth word w in the training corpus of the scarce resource language model. nThe probability of occurrence; p(w1) is the probability of w1 appearing in the entire final language model training corpus; p(w2|w1) is the conditional probability of w2 appearing in the corpus given that w1 appears; p(w n |w n-1 ) for in the corpus w n-1 Under the condition that it occurs, and w n The conditional probability of occurrence;

[0067] 3-Gram language model:

[0068] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 w n-2 )

[0069] Wherein, p(w n |w n-1 w n-2 ) for in the corpus w n-1 and w n-2 Under the condition that they occur simultaneously, and w n The conditional probability of occurrence;

[0070] 4-Gram language model:

[0071] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 w n-2 w n-3 )

[0072] Wherein, p(w n |w n-1 w n-2 w n-3 ) for in the corpus w n-1 w n-2 and w n-3 Under the condition that they occur simultaneously, and w n The conditional probability of occurrence;

[0073] Among them, w n-1 w is the distance from the nth word n The first word; w n-2 w is the distance from the nth word n The first two words; w n-3 w is the distance from the nth word n The first three words;

[0074] In the formulas for the 2, 3, and 4-Gram language models above, the maximum likelihood method is used to estimate the word w in the final language model training corpus. n The probability of occurrence p(w) n ) and conditional probability p(w n |w n- 1w n-2 ...w1), the probability of occurrence is p(w1). n The specific formula is as follows:

[0075]

[0076] Where, f(w) n ) is the word w n The frequency of occurrence in the training corpus; f(w) N ) represents the total word frequency in the training corpus; w N The total set of words involved in the training corpus for the final language model;

[0077] The formula for calculating conditional probability is as follows:

[0078]

[0079] Wherein, f(w1w2...w n ) represents the words w1w2...w n Frequency of occurrence in the training corpus; f(w1w2...w n-1 ) represents the words w1w2...w n-1 Frequency of occurrence in the training corpus;

[0080] Finally, by calculating the word w on the final language model training corpus, we can obtain the result. n The probability of occurrence p(w) n ) and conditional probability p(w n |w n-1 ), p(w n |w n-1 w n-2 ) and p(w n |w n-1 w n-2 w n-3 This led to the development of 2-Gram, 3-Gram, and / or 4-Gram language models as scarce resource language models, forming scarce resource language models that serve specific domains.

[0081] The present invention also provides a modeling apparatus for a scarce resource language model serving a specific neighborhood, the apparatus comprising:

[0082] The preprocessing module is used to collect scarce resource texts as initial corpus, preprocess the initial corpus to obtain preprocessed text; and segment the preprocessed text into sentences, preserving the order of sentences and paragraph information to obtain a preprocessed sentence set composed of multiple segmented sentences.

[0083] The LDA analysis module is used to obtain the statement-topic distribution and topic-word distribution from the preprocessed statement set using the Gibbs algorithm.

[0084] The domain topic distribution calculation module is used to select domain feature words for a specific domain from pre-collected rich resource language texts to obtain selected domain feature words; translate the selected domain feature words into words in rare languages ​​to obtain translated domain feature words; and use the translated domain feature words, combined with the obtained topic-word distribution, to calculate and obtain the domain topic distribution.

[0085] The domain corpus selection module is used to calculate the relevance between the statement-topic distribution and the domain topic distribution of each segmented statement; determine whether the relevance of each segmented statement exceeds a preset threshold, and select statements that meet the condition into the scarce resource of specific domain language model training corpus; further filter the obtained language model training corpus based on the coherence of language expression to obtain the final language model training corpus; and

[0086] The domain language model training module is used to train the scarce resource language model using the final language model training corpus, thus obtaining the scarce resource language model.

[0087] The present invention also provides an electronic device for modeling a language model of scarce resources in a specific domain, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described thereon.

[0088] The advantages of this invention compared to the prior art are:

[0089] 1. The method of this invention can effectively address the difficulties of modeling language with scarce resources by utilizing linguistic big data. Specifically addressing the challenges of modeling language with scarce resources in specific domains, particularly the scarcity of expert resources, this invention provides an effective and feasible modeling approach; it directly utilizes linguistic big data for modeling, serving specific domain applications and helping to improve modeling efficiency and performance; by employing topic semantic analysis technology to analyze scarce resource language data and extract semantic information, it provides rich clues for modeling language related to scarce resources in specific domains.

[0090] 2. The method of this invention can effectively address the difficulties in modeling scarce resource language models, such as the lack of semantic information support and the lack of expert resource service corpus classification and annotation, by utilizing language big data. For modeling scarce resource language models in specific domains, the method of this invention provides an effective and feasible implementation scheme; it directly utilizes language big data for modeling, serving specific domain applications, which helps to improve modeling efficiency and model performance; it uses topic semantic analysis technology to analyze scarce resource language data, mining semantic information to provide rich semantic information for serving scarce resource language modeling in specific domains.

[0091] 3. The method of the present invention introduces topic semantic analysis to analyze scarce resource language corpora, obtains corresponding semantic information, combines domain feature word knowledge of enriched resource languages, selects scarce resource language corpora involving specific domains to realize language model modeling, and serves language information processing applications in specific domains.

[0092] 4. The method of this invention provides an effective and feasible solution for modeling language languages ​​with scarce resources (especially scarce expert resources) in specific fields;

[0093] 5. Utilizing scarce language corpora related to specific domains for language modeling serves specific domain applications, which helps improve modeling efficiency and model performance;

[0094] 6. By using topic semantic analysis technology to perform semantic analysis on scarce resource language corpora, we can deeply mine the semantic information in the corpus, realize the comprehensive utilization of semantic information, effectively obtain scarce resource corpora in specific fields, and lay the foundation for language modeling. Attached Figure Description

[0095] Figure 1 This is a flowchart of the method for modeling a language model of scarce resources serving a specific neighborhood according to the present invention.

[0096] Figure 2 This is a schematic diagram of the structure of the scarce resource language modeling system of the present invention;

[0097] Figure 3 This is a diagram illustrating the LDA topic analysis method used in this invention. Detailed Implementation

[0098] The present invention will now be further described in conjunction with the accompanying drawings and examples.

[0099] This invention provides a method for modeling a scarce resource language model serving a specific neighborhood. This method utilizes LDA (Local Language Derivative Analysis) to analyze and identify scarce resource corpora in a specific domain, improving language modeling methods for domain-specific applications. Based on large-scale language data resources, this method combines LDA topic semantic analysis; it uses domain-specific feature words obtained from enriched resource languages ​​to obtain the topic distribution of the specific domain; it calculates the relevance between sentences in the scarce resource corpus and the specific domain, selecting sentences with a relevance greater than a preset threshold as training data to construct a scarce resource language model serving the specific domain. This invention's method can effectively address the difficulties of modeling scarce resource language models using large-scale language data.

[0100] This method utilizes rich language data, introduces topic semantic analysis, and incorporates domain-specific information from other resource-rich languages ​​to overcome the difficulties in constructing language models that reflect the characteristics of specific application domains, especially for languages ​​with scarce resources and a lack of expert resources. It provides a method for modeling language models for scarce resource languages ​​that utilize topic semantics to serve specific domains.

[0101] like Figure 1 As shown, the method includes:

[0102] Step 1) Collect scarce resource texts as initial corpus, preprocess the initial corpus to obtain preprocessed text; divide the preprocessed text into sentences, retain the order of sentences and paragraph information, and obtain a preprocessed sentence set composed of multiple segmented sentences.

[0103] Specifically, scarce resource text is collected as the initial corpus. Irrelevant display tags and content are removed from this initial corpus, retaining only the text content information. The character set encoding of the text in the initial corpus is standardized. The initial corpus is then segmented into sentences, preserving the sequential relationships and paragraph information between sentences to obtain a preprocessed set of sentences. In this embodiment, Russian is used as a scarce resource language, as it is relatively scarce in terms of expert resources compared to Chinese and English. A total of 2.1 GB of Russian text was collected and organized as the initial corpus, i.e., the corpus for analysis and processing.

[0104] Step 2) Perform LDA topic analysis on the preprocessed sentence set, and combine it with the Gibbs algorithm to obtain the sentence-topic distribution and topic-word distribution;

[0105] Specifically, the preprocessed statement set is represented as consisting of M segmented statements, denoted as S = {s1, ..., s2}. m ,...,s M}, where the Mth segmentation statement s MIt is a word sequence of length N, denoted as s. M =(w1,…,w n ,...,w N ), where w n This represents the nth word in a word sequence;

[0106] Figure 3 The structure of topic distribution and sentence distribution is given. Here, the outer box represents a text and the inner box represents a mixed selection of topics and words. α and β are the two prior parameters corresponding to the two distributions. K represents the preset number of topics.

[0107] Each segmented sentence in the preprocessed sentence set is represented as a mixture of K potential topics. The topics themselves are captured using probability distributions on words, and the generation process is as follows:

[0108] Step 2-1) Obtain topic-word distribution;

[0109]

[0110] Where β=(β1…β K ) T β is the prior distribution parameter of Dirichlet; Γ(x) is the gamma function, defined as... In the formula, u x-1 Let Γ(x) be the x-1th power of the integral variable in the gamma function, a special function commonly used in mathematical analysis. Here, x represents its independent variable, and in the above formula, Γ(x) takes the values ​​β and β respectively. k , μ is a continuous random variable; Let μ be the k-th continuous random variable k β k -1 power;

[0111] Based on the prior distribution parameter β of the topic-word Dirichlet distribution, a multinomial topic distribution φ is drawn for each topic. k A total of K topic distributions were extracted;

[0112] Step 2-2) Obtain statement-topic distribution:

[0113]

[0114] Where α=(α1…α K ) T α is the prior parameter of Dirichlet; Γ(x) is the gamma function, defined as... In the formula, u x-1The gamma function is the integral variable raised to the power of x-1 in the special function gamma function. The gamma function Γ(x) is a commonly used special function in mathematical analysis, where x represents its independent variable, and takes values ​​α in the above formula. k , μ is a continuous random variable; Let μ be the k-th continuous random variable k α k -1 power;

[0115] Based on the prior parameters α of the statement-topic Dirichlet distribution, a multinomial statement distribution θ is extracted for each segmented statement. m A total of M statements were extracted and distributed.

[0116] Steps 2-3) For each statement and the corresponding words in the statement:

[0117] From the distribution of polynomial statements θ m Extracting topics from z m ;

[0118] From the polynomial topic distribution φ k Extracting words from w n ;

[0119] The most important random variable in the model is the multinomial statement distribution θ. m and polynomial topic distribution φ k Here, it is assumed that they follow the Dirichlet multivariate probability distribution expressed above. Therefore, the maximum likelihood method can be directly used to estimate these two parameters to obtain the corresponding sentence-topic distribution and topic-word distribution. However, this results in local extrema. Therefore, the method of this invention uses the Gibbs method to obtain the corresponding sentence-topic distribution and topic-word distribution, thus avoiding the problem of local extrema.

[0120] Steps 2-4) Calculate α based on the preset number of topics K:

[0121] α = 3 / K;

[0122] All β values ​​are set to a fixed empirical value of β = 0.02; in this embodiment, all parameters related to β are set to the empirical value of 0.02.

[0123] Based on the preprocessed set of statements, the number of topics K is estimated using the model confusion analysis method. Using existing complexity calculation methods, the curve of confusion as a function of the number of topics is obtained. Based on this curve, the number of topics with lower confusion relative to the lowest confusion level is selected.

[0124] Steps 2-5) randomly assign a topic to a word in the sentence from the K topic distributions in the range [1…K], forming an initial Markov chain. Repeat this process, assigning a corresponding topic to all words in the sentence, and obtaining the next state of the Markov chain. After multiple iterations, the Markov chain reaches a stable state. Using the Gibbs sampling method, the sentence-topic distribution and topic-word distribution are obtained.

[0125]

[0126]

[0127] in, Let w be the estimated value of the topic-word distribution, and w be the token for a new word extracted from topic j. n Probability estimation; C VK and C MK These are quantity matrices of dimensions V×K and M×K, respectively; where V is the total number of words; For the word w -n The frequency of occurrences belonging to topic j, excluding w. n The frequency of occurrences of topic j; The preset hyperparameter β and corresponding word w n The value belonging to topic j is set to 0.02 in this embodiment; For the word w v Frequency of topic j, excluding word w n The frequency of words belonging to topic j after summing the words; β v,j The preset hyperparameter β and corresponding word w v The value belonging to topic j is set to 0.02 in this embodiment;

[0128] in, The estimated value of the statement-topic distribution is given by the statement s. m Probability estimation of extracting new words from topic j; For specific delimiter statements s m The number of words specified for topic j, excluding instance n, i.e., excluding w. n Frequency of belonging to topic j; α m,j The preset hyperparameter α and corresponding statement s m The value belongs to topic j; For specific delimiter statements s m The number of words specified for topic k, excluding instance n, i.e., excluding w. n Frequency of belonging to topic k; α m,k The preset hyperparameter α and corresponding statement s m The value belonging to topic k.

[0129] Step 3) In the pre-collected rich resource language texts, select domain feature words for a specific domain to obtain the selected domain feature words;

[0130] Specifically, step 3-1) collects rich resource language texts and uses them as training corpus, and selects texts involving specific domain content as the basis for extracting domain feature words;

[0131] Step 3-2) Calculate the information gain value G(h) of the words. This information gain value is the difference between the entropy of the document without considering any word features and the entropy of the document after considering word features. The specific formula for calculation is as follows:

[0132]

[0133] Where E(S) is the document entropy without considering any word features; E(S) h P(C) represents the document entropy after considering word features; j ) is C j P(h) is the probability of a document of class h appearing in the training corpus; P(C) is the probability of a document in the training corpus containing the word feature word h; j |h) is a document containing the word feature word h and belongs to C. j Conditional probability of class documents; To train the probability of documents in the corpus that do not contain the word feature word h; When the document does not contain the word feature word h and belongs to C j The conditional probability of a document class; M represents the total number of classifications for the training corpus, M=2, dividing the training corpus into two categories: domain-specific and non-application-specific;

[0134] Step 3-3) Repeat step 3-2) Calculate the information gain value of all words in each text involving specific domain content in the training corpus as feature words, and sort all words in each text involving specific domain content in descending order of information gain value. Based on experience, select the top 500 to 10,000 words as selected domain feature words.

[0135] In other specific embodiments, different numbers of candidate feature words are applied to the text classifier, and the optimal number of candidates is observed and selected as the domain feature words.

[0136] Specifically, different numbers of words are selected as domain feature words based on their information gain values, from largest to smallest.

[0137] Step 4) Translate the selected domain feature words into words in rare languages ​​to obtain the translated domain feature words;

[0138] Specifically, the selected domain-specific words are translated into words in rare languages ​​using relevant dictionaries such as Youdao Dictionary, Kingsoft PowerWord, Bing Dictionary, and Google Translate, resulting in translated domain-specific words.

[0139] Step 5) Using the translated domain feature words, combined with the topic-word distribution obtained in Step 2), calculate and obtain the domain topic distribution;

[0140] Specifically, the sequence of domain-specific words in the translated scarce resource language is denoted as the translated domain-specific words, and it is represented as E = {e1,…,e...} n ,…,e N}; where e n This represents the nth domain feature word in the domain feature word sequence, where N represents the total number of domain feature words; e n Estimates of topic-word distribution As the topic distribution vector of words

[0141]

[0142] Among them, z i Domain-specific words e n Distribution value on the i-th topic; z K Domain-specific words e n Distribution value on the Kth topic;

[0143] Domain Topic Distribution D T for:

[0144] D T =[t1,...,t i ,...,t K ]

[0145] Among them, t i t represents the distribution value of a specific application domain on the i-th topic; K This represents the distribution value of the Kth topic in a specific application area;

[0146]

[0147] Among them, z ij Domain-specific words e j Distribution value on the j-th topic.

[0148] Step 6) Calculate the correlation between the statement-topic distribution and the domain topic distribution for each segmented statement;

[0149] Specifically, assume the preprocessed statement set S = {s1,...,s...} i ,...,s M}, where M is the number of statements, s i For the i-th segmented statement in the preprocessed statement set; let s i exist Sentence-topic distribution as a numerical representation of word LDA vectors:

[0150] s i =[y1,...,y j ,...,y K ]

[0151] Among them, y j s i Distribution value on the j-th topic;

[0152] The cosine similarity method is used to calculate the correlation C(s) between the statement-topic distribution and the domain-topic distribution. i ):

[0153]

[0154] Among them, y j For s i The component value of the j-th topic in the statement-topic distribution; t j Let be the component value of the j-th topic in the domain topic distribution;

[0155] C(s i ) represents statement s i The cosine of the angle between the statement-topic distribution and the domain topic distribution, with the result between [0,1], is denoted as the correlation between the statement-topic distribution and the domain topic distribution; the closer the value of this calculation result is to 1, the higher the correlation between the two.

[0156] Repeat the above process to obtain the correlation between the statement-topic distribution and the domain topic distribution for each segmented statement.

[0157] Step 7) Determine whether the relevance of each segmented statement exceeds the preset threshold, and select statements that meet the conditions into the language model training corpus for a scarce resource specific domain;

[0158] Specifically, it determines whether the relevance of each statement exceeds a preset threshold.

[0159] If the relevance of a statement is greater than a preset threshold, then statements that meet the criteria will be included in the language model training corpus for a scarce resource specific domain.

[0160] If the relevance of a statement is less than or equal to a preset threshold, then the statement is deleted.

[0161] Step 8) Based on the coherence of the language expression, further filter the language model training data obtained in Step 7) to obtain the final language model training data;

[0162] Specifically, in step 8-1), if the selected segmentation statement is at the beginning of a paragraph, then the text preceding that segmentation statement is not selected;

[0163] If the selected segmentation statement is not at the beginning of the paragraph, then the text preceding the segmentation statement is selected as the candidate corpus and included in the language model training corpus for the scarce resource specific domain.

[0164] Step 8-2) If the selected delimiter is located at the end of a paragraph, then do not select the text following that delimiter;

[0165] If the selected segmentation statement is not located at the end of the paragraph, then the text following the segmentation statement is selected as the candidate corpus and included in the training corpus of the language model for the scarce resource specific domain.

[0166] Step 8-3) Based on Step 8-1) and Step 8-2) above, the language model training corpus is screened again and deduplicated to obtain the final language model training corpus;

[0167] Specifically, the two candidate corpora obtained above are deduplicated. If they are already in the training corpora obtained in step 7), they are not counted again. The deduplicated candidate corpora are then included in the training corpora.

[0168] Step 9) Use the final language model training corpus obtained in Step 8) to train the scarce resource language model and obtain the scarce resource language model.

[0169] Specifically, the modeling method uses the n-Gram language model, where n can take the value of 2, 3, or 4:

[0170] 2-Gram language model:

[0171] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 )

[0172] Where p(S) represents the generation probability of the segmented statement S; p(w1w2...w n (w) represents the nth word w in the training corpus of the scarce resource language model. nThe probability of occurrence; p(w1) is the probability of w1 appearing in the entire final language model training corpus; p(w2|w1) is the conditional probability of w2 appearing in the corpus given that w1 appears; p(w n |w n-1 ) for in the corpus w n-1 Under the condition that it occurs, and w n The conditional probability of occurrence;

[0173] 3-Gram language model:

[0174] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 w n-2 )

[0175] Wherein, p(w n |w n-1 w n-2 ) for in the corpus w n-1 and w n-2 Under the condition that they occur simultaneously, and w n The conditional probability of occurrence;

[0176] 4-Gram language model:

[0177] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 w n-2 w n-3 )

[0178] Wherein, p(w n |w n-1 w n-2 w n-3 ) for in the corpus w n-1 w n-2 and w n-3 Under the condition that they occur simultaneously, and w n The conditional probability of occurrence;

[0179] Among them, w n-1 w is the distance from the nth word n The first word; w n-2 w is the distance from the nth word n The first two words; w n-3 w is the distance from the nth word n The first three words;

[0180] In the formulas for the 2, 3, and 4-Gram language models above, the maximum likelihood method is used to estimate the word w in the final language model training corpus. n The probability of occurrence p(w) n ) and conditional probability p(w n |w n- 1w n-2 ...w1), the probability of occurrence is p(w1). n The specific formula is as follows:

[0181]

[0182] Where, f(w) n ) is the word w n The frequency of occurrence in the training corpus; f(w) N ) represents the total word frequency in the training corpus; w N The total set of words involved in the training corpus for the final language model;

[0183] The formula for calculating conditional probability is as follows:

[0184]

[0185] Wherein, f(w1w2...w n ) represents the words w1w2...w n Frequency of occurrence in the training corpus; f(w1w2...w n-1 ) represents the words w1w2...w n-1 Frequency of occurrence in the training corpus;

[0186] Finally, by calculating the word w on the final language model training corpus, we can obtain the result. n The probability of occurrence p(w) n ) and conditional probability p(w n |w n-1 ), p(w n |w n-1 w n-2 ) and p(w n |w n-1 w n-2 w n-3 This led to the development of 2-Gram, 3-Gram, and / or 4-Gram language models as scarce resource language models, forming scarce resource language models that serve specific domains.

[0187] The present invention also provides a modeling apparatus for a scarce resource language model serving a specific neighborhood, the apparatus comprising:

[0188] The preprocessing module is used to collect scarce resource texts as initial corpus, preprocess the initial corpus to obtain preprocessed text; and segment the preprocessed text into sentences, preserving the order of sentences and paragraph information to obtain a preprocessed sentence set composed of multiple segmented sentences.

[0189] The LDA analysis module is used to obtain the statement-topic distribution and topic-word distribution from the preprocessed statement set using the Gibbs algorithm.

[0190] The domain topic distribution calculation module is used to select domain feature words for a specific domain from pre-collected rich resource language texts to obtain selected domain feature words; translate the selected domain feature words into words in rare languages ​​to obtain translated domain feature words; and use the translated domain feature words, combined with the obtained topic-word distribution, to calculate and obtain the domain topic distribution.

[0191] The domain corpus selection module is used to calculate the relevance between the statement-topic distribution and the domain topic distribution of each segmented statement; determine whether the relevance of each segmented statement exceeds a preset threshold, and select statements that meet the condition into the scarce resource of specific domain language model training corpus; further filter the obtained language model training corpus based on the coherence of language expression to obtain the final language model training corpus; and

[0192] The domain language model training module is used to train the scarce resource language model using the final language model training corpus, thus obtaining the scarce resource language model.

[0193] The present invention also provides an electronic device for modeling a language model of scarce resources in a specific domain, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described thereon.

[0194] Example 1.

[0195] To better illustrate the method of the present invention, a specific embodiment is provided. In this embodiment, the method includes: [The method is described in the context of] rare resource languages.

[0196] Step 1: Preprocess the Russian text;

[0197] Russian text was collected through various methods, including downloading Russian text from the internet, purchasing or exchanging Russian text from third parties, and using OCR (Optical Character Recognition) to collect Russian text. This data was recorded as the initial corpus, serving as the foundation for analysis and processing. In this example, a total of 2.1GB of Russian text was collected and organized, forming a text corpus. Irrelevant display tags and content were removed from the Russian text, retaining only the text content information. The character set encoding of the Russian text was standardized. The preprocessed Russian text was segmented into sentences, preserving the sentence order and paragraph information, resulting in a preprocessed sentence set composed of multiple segmented sentences for subsequent steps.

[0198] Step 2: LDA topic analysis, establishing a topic analysis model;

[0199] 2-1. LDA Modeling

[0200] The preprocessed statement set is represented as consisting of M segmented statements, denoted as S = {s1, ..., s2}. M}, where the Mth segmentation statement s M It is a word sequence of length N, denoted as s. M =(w1,…,w N ), where w n This represents the nth word in the word sequence; the starting point of the LDA model is to represent each sentence as a mixture of K latent topics. The topics themselves appear explicitly in the sentence and are captured using probability distributions on the words. The generation process is as follows:

[0201] 1) Draw a multinomial distribution φ for each topic from the Dirichlet prior β. k A total of K distributions were sampled.

[0202] 2) Draw a multinomial distribution θ for each statement from the Dirichlet distribution prior α. m A total of M distributions were sampled.

[0203] 3) For each text and word in the sentence set:

[0204] From the distribution of polynomial statements θ m Extracting topics from z m ;

[0205] From the polynomial topic distribution φ k Extracting words from w n ;

[0206] The most important random variable in the model is the multinomial statement distribution θ. m and polynomial topic distribution φk Here, it is assumed that they follow the Dirichlet multivariate probability distribution expressed above. Therefore, the maximum likelihood method can be directly used to estimate these two parameters to obtain the corresponding sentence-topic distribution and topic-word distribution. However, this results in local extrema. Therefore, the method of this invention uses the Gibbs method to obtain the corresponding sentence-topic distribution and topic-word distribution, thus avoiding the problem of local extrema.

[0207] 2-2. Use the Gibbs algorithm to obtain the statement-topic distribution and topic-word distribution.

[0208] For a given word in a sentence, randomly assign a topic from the K topic distributions in the range [1…K] to form an initial Markov chain. Repeat the above process, assigning a corresponding topic to all words in the sentence and obtaining the next state of the Markov chain. After multiple iterations, the Markov chain can reach a stable state. Using the Gibbs sampling method, obtain the sentence-topic distribution and the topic-word distribution:

[0209]

[0210]

[0211] in, Let w be the estimated value of the topic-word distribution, and w be the token for a new word extracted from topic j. n Probability estimation; C VK and C MK These are quantity matrices of dimensions V×K and M×K, respectively; where V is the total number of words; For the word w -n The frequency of occurrences belonging to topic j, excluding w. n The frequency of occurrences of topic j; The preset hyperparameter β and corresponding word w n The value belonging to topic j is set to 0.02 in this embodiment; For the word w v Frequency of topic j, excluding word w n The frequency of words belonging to topic j after summing the words; β v,j The preset hyperparameter β and corresponding word w v The value belonging to topic j is set to 0.02 in this embodiment;

[0212] in, The estimated value of the statement-topic distribution is given by the statement s. M Probability estimation of extracting new words from topic j; For specific delimiter statements s MThe number of words specified for topic j, excluding instance n, i.e., excluding w. n Frequency of belonging to topic j; α m,j The preset hyperparameter α and corresponding statement s M The value belongs to topic j; For specific delimiter statements s M The number of words specified for topic k, excluding instance n, i.e., excluding w. n Frequency of belonging to topic k; α m,k The preset hyperparameter α and corresponding statement s M The value belonging to topic k.

[0213] 2-3. Setting LDA model parameters.

[0214] For the LDA model, based on general experience, the hyperparameter α = 3 / K is chosen, and all β values ​​are taken as fixed empirical values ​​β = 0.02. To determine a suitable number of topics K, the variation of disorder with K is calculated, and an appropriate number of topics K is determined.

[0215] The confusion level is calculated by taking the reciprocal of the geometric mean of the likelihood values ​​of each sentence group within the sentence set. The confusion level of the LDA model monotonically decreases as the likelihood value of the sentence group increases. The specific calculation formula is as follows:

[0216]

[0217] In the above formula, N is the number of statements in the statement set. s Let p(S) be the number of terms in statement S, and p(S) be the likelihood value of statement S. The LDA model is used here, therefore the likelihood value of a statement is given by the topic distribution of the statement and the word distribution of the topic, calculated as follows:

[0218]

[0219] In the formula, n(w, S) represents the number of times word w appears in statement S.

[0220] The table below shows the curve of disorder as a function of the K value in this example.

[0221] Table of Chaos Level as a Percentage of Topics

[0222] Number of topics chaos 10 1794.47 100 556.2525 200 344.6795 300 220.0548 400 190.9708 500 189.2065 600 187.2255 700 181.43 800 182.791 900 167.823 1000 160.856

[0223] As can be seen from the table, when the number of topics K is greater than 400, the disorder is relatively low. It's important to note that disorder decreases as the number of topics increases. While disorder can be very low with a large number of topics, the model's inductive reasoning ability is also compromised. Therefore, choosing the right K value doesn't necessarily mean selecting the lowest possible disorder, but rather choosing one with relatively low disorder that also results in a topic model that is easy to interpret. Considering these factors, a K value of 400 is chosen in this example.

[0224] 2-4. Calculated estimates of statement-topic distribution and topic-word distribution.

[0225] The following table shows the calculated statement-topic distribution:

[0226]

[0227]

[0228] The table below shows the word distribution for a given topic, denoted as Topic-Word Distribution:

[0229]

[0230] Step 3: Select domain-specific characteristic words in Chinese.

[0231] 3-1. Using Chinese texts as a rich resource language, collect texts and classify them as training data;

[0232] 3-2. Calculate the information gain of different words. The information gain value is the difference between the document entropy without considering any features and the document entropy after considering the feature. The specific formula for calculation is as follows:

[0233]

[0234] Where E(S) is the entropy of the text without considering any word features; E(S) h P(C) represents the entropy of the document after considering the features of the word. j ) is C j The probability of a document of class h appearing in the training corpus; P(h) is the probability of a document in the training corpus containing the feature word h; P(C j |h) refers to a document containing the feature word h and belonging to C. j Conditional probability of class documents; To train the probability of documents in the corpus that do not contain feature term h; When a document does not contain the feature word 'h', it belongs to C. jThe conditional probability of a document class; M represents the total number of classifications for the training corpus, M=2, dividing the training corpus into two categories: domain-specific and non-application-specific;

[0235] 3-3. Based on the information gain value of the words, select different numbers of words as feature words from largest to smallest.

[0236] Select the top-ranked keywords based on the sorting results. In this example, we select the top 1000 keywords. Specific examples of selected keywords are shown below.

[0237] …… destroy defense Disaster relief defense fire Ice and snow rainstorm Extreme cold typhoon rescue Lightning environment Ecology Protection hail Dense fog Thunderstorm drought ……

[0238] Step 4: Use a Chinese-Russian dictionary to translate the selected domain-specific terms into Russian vocabulary.

[0239] Example words:

[0240] …… …… destroy разрyшать defense защита Disaster relief Пoмoщьпристихийныхбедствиях defense защита fire Огoнь Ice and snow Ледиснег rainstorm ливень Extreme cold Чрезвычайнoхoлoднo typhoon тайфyн rescue Cпасать Lightning грoмимoлния environment среда Ecology Экoлoгия Protection 3ащита hail град Dense fog Густoйтуман Thunderstorm Грoза drought засуха …… ……

[0241] Step 5: Calculate the topic distribution vector in the domain.

[0242] The sequence of domain-specific words in the translated scarce resource language is denoted as the translated domain-specific words, and it is represented as E = {e1, ..., e}. N}; where e n Let N represent the nth domain feature word in the sequence, and N represent the total number of domain feature words; let e n Estimates of topic-word distribution As a topic distribution vector of words:

[0243]

[0244] Among them, z i Domain-specific words e n The distribution value on the i-th topic, where K represents the total number of potential topics;

[0245] The topic distribution D of the domain is as follows:

[0246] D T =[t1,...,t i ,...,t K ]

[0247] Among them, t i This represents the distribution value of a specific application domain on the i-th topic, and K represents the total number of potential topics.

[0248]

[0249] In the formula z ij Domain-specific words ej The distribution value on the i-th topic, where N represents the total number of feature words for the selected specific application domain.

[0250] Step 6: Calculate the relevance of statements to the domain, and select statements with high relevance as domain training corpus.

[0251] 6-1. Calculate relevance. After LDA analysis, the Russian sentence set S = {s1,...s...} i ,...,s M}, where M is the number of statements, s i For any statement in the set; with s i exist Sentence-topic distribution as a numerical representation of word LDA vectors:

[0252] s i =[y1,...,y j ,...,y K ]

[0253] In the formula y j Indicates the word s i The distribution value on the j-th potential LDA topic, where K represents the total number of topics defined in the LDA analysis.

[0254] The cosine of the angle between the statement and the domain topic is calculated as follows:

[0255]

[0256] In the formula C(s) i ) represents statement s i The cosine of the angle between the subject and the domain topic is calculated, and the result is between [0,1]. The larger the value, the higher the degree of relevance.

[0257] 6-2. Select sentences whose topic distribution vectors are similar to the word topic vector distributions as training corpora for language models in specific application domains where scarce resources are available. The specific processing method is as follows:

[0258] The cosine C(s) of the angle between the statement and the domain topic calculated in step 8. i The numerical value is set as a threshold value, and statements exceeding the threshold value are selected as training data for language models in specific application areas that are scarce resources.

[0259] 6-3. Considering the continuity of language expression, the nearest neighbor sentences of the sentences in the selected training corpus are selected and added to the language model training corpus. The specific processing method is as follows:

[0260] i. If the sentence selected in step 8) is at the beginning of the paragraph, then the sentence before that sentence is not selected; otherwise, the sentence before that sentence is selected as a candidate corpus and included in the training corpus.

[0261] ii. If the sentence selected in step 8) is at the end of the paragraph, then the sentence following that sentence is not selected; otherwise, the sentence following that sentence is selected as a candidate corpus and included in the training corpus.

[0262] iii. If the candidate sentences expanded in the above two steps have already been selected into the training corpus in step 8), they will not be counted again in the training corpus; the deduplicated candidate sentences will be included in the training corpus.

[0263] In this example, 130MB of text closely related to the domain was ultimately selected as the domain training corpus.

[0264] Step 7: Use an n-Gram language model and train the model using a selected, scarce resource of specific application domain corpus. Here, n can be 2, 3, or 4.

[0265] 2-Gram language model:

[0266] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 )

[0267] 3-Gram language model:

[0268] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 w n-2 )

[0269] 4-Gram language model:

[0270] p(S)=p(w1w2...w n )=p(w1)p(w2|w1)...p(w n |w n-1 w n-2 w n-3 )

[0271] Where p(S) represents the generation probability of the segmented statement S; p(w1w2...w n (w) represents the nth word w in the training corpus of the scarce resource language model. nThe probability of occurrence; p(w1) is the probability of w1 appearing in the entire final language model training corpus; p(w2|w1) is the conditional probability of w2 appearing in the corpus given that w1 appears; p(w n |w n-1 ) for in the corpus w n-1 Under the condition that it occurs, and w n The conditional probability of occurrence; p(w) n |w n-1 w n-2 ) for in the corpus w n-1 and w n-2 Under the condition that they occur simultaneously, and w n The conditional probability of occurrence; p(w) n |w n-1 w n-2 w n-3 ) for in the corpus w n-1 w n-2 and w n-3 Under the condition that they occur simultaneously, and w n The conditional probability of occurrence; w n-1 w n-2 w n-3 w is the distance from the nth word n The first word, the first two words, the first three words.

[0272] The maximum likelihood method is used to estimate the probability and conditional probability of word occurrence. The specific formula is as follows:

[0273]

[0274] In the formula, f(w) n ) is the word w n The frequency of occurrence in the training corpus; f(w) N ) represents the total word frequency in the training corpus; w N The total set of words involved in the training corpus, that is, the total set of words in the final language model training corpus;

[0275]

[0276] In the formula, f(w1w2...w n ) represents the words w1w2...w n Frequency of occurrence in the training corpus; f(w1w2...w n-1 ) represents the words w1w2...w n-1 The frequency of occurrence in the training corpus.

[0277] By calculating the word w on the final language model training corpus, we can obtain the word w. nThe probability of occurrence p(w) n ) and conditional probability p(w n |w n-1 ), p(w n |w n-1 w n-2 ) and p(w n |w n-1 w n-2 w n-3 This yields 2-Gram language models, 3-Gram language models, or 4-Gram language models, which are scarce resource language models, forming scarce resource language models that serve specific domains.

[0278] Perplexity (PPL) is generally used to measure the quality of a language model. The perplexity of a language model refers to how perplexing the language model is to the test set, and it is defined as follows:

[0279]

[0280] Generally, a smaller perplexity value indicates better model performance and a model that is closer to the actual corpus distribution.

[0281] In this example, we specifically calculated 2-Gram and 3-Gram domain-specific language models. Combined with the Knerser-Ney smoothing method, the perplexity of the language models is as follows:

[0282] Model Confusion 2-Gram 360.34 3-Gram 121.60

[0283] The model's overall performance is at a good level.

[0284] As can be seen from the specific embodiments above, this invention utilizes language big data and introduces topic semantic analysis methods to overcome the difficulties caused by the scarcity of expert resources, and realizes the modeling of language models in specific domains of languages ​​with scarce resources, demonstrating the feasibility and effectiveness of the method.

[0285] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for modeling a language model of scarce resources serving a specific domain, the method comprising: Step 1) Collect scarce resource texts as initial corpus, preprocess the initial corpus, and obtain preprocessed text; The preprocessed text is segmented into sentences, preserving the order of sentences and paragraph information, resulting in a preprocessed sentence set composed of multiple segmented sentences. Step 2) Using the Gibbs algorithm, the preprocessed set of statements is used to obtain the statement-topic distribution and topic-word distribution; Step 3) In the pre-collected rich resource language texts, select domain feature words for a specific domain to obtain the selected domain feature words; Step 4) Translate the selected domain feature words into words in rare languages ​​to obtain the translated domain feature words; Step 5) Using the translated domain feature words, combined with the topic-word distribution obtained in Step 2), calculate and obtain the domain topic distribution; Step 6) Calculate the correlation between the statement-topic distribution and the domain topic distribution for each segmented statement; Step 7) Determine whether the relevance of each segmented statement exceeds the preset threshold, and select statements that meet the conditions into the language model training corpus for a scarce resource specific domain; Step 8) Based on the coherence of the language expression, further filter the language model training data obtained in Step 7) to obtain the final language model training data; Step 9) Use the final language model training corpus obtained in Step 8) to train the scarce resource language model and obtain the scarce resource language model. Step 5) specifically includes: The sequence of domain-specific words in the translated scarce resource language is denoted as the translated domain-specific words, and its representation is as follows: ;in, Represents the first term in the domain feature word sequence. n Domain-specific characteristic words, N The total number of domain-specific words; Estimates of topic-word distribution As the topic distribution vector of words : in, Domain-specific words In the i Distribution values ​​across the themes; Domain-specific words In the K Distribution values ​​across the themes; Domain Theme Distribution for: in, t i Indicates the specific application area in the first i Distribution values ​​across the themes; For specific application areas in the first K Distribution values ​​across the themes; in, Domain-specific words In the j Distribution values ​​across the themes.

2. The method for modeling a language model of scarce resources in a specific service domain according to claim 1, characterized in that, Step 2) specifically includes: The preprocessed set of statements is represented as by M It consists of several segmented statements, denoted as , of which M A segmented statement It is a length of N The word sequence is denoted as ,in, Indicates the first word in the word sequence n One word; To split a word in a sentence in [ 1…K ]of K A topic is randomly assigned from the topic distribution to form the initial Markov chain. The above process is repeated. For all words in the segmented sentence, a corresponding topic is assigned, and the next state of the Markov chain is obtained. After multiple iterations, the Markov chain reaches a stable state. Using the Gibbs sampling method, the topic-word distribution was obtained: And statement-topic distribution: in, The estimated value of the topic-word distribution is given by the topic. j Extracting new word tokens w n Probability estimation; and The dimensions are respectively V×K and M×K The quantity matrix; where, V The total number of words; For words Belongs to the topic j Frequency; For corresponding words Belongs to the topic j The value is set to 0.02; For words Belongs to the topic j Frequency; For corresponding words Belongs to the topic j The value is set to 0.02; in, This is an estimate of the statement-topic distribution, which is the value of the statement-topic distribution in the statement. From the theme j Probability estimation of new word extraction; For specific delimited statements The topic is designated in the middle. j The number of words; For the corresponding delimited statement Belongs to the topic j The value; For specific delimited statements The topic is designated in the middle. k The number of words; For the corresponding delimited statement Belongs to the topic k The value of .

3. The method for modeling scarce resources in a service-specific domain according to claim 1, characterized in that, Step 3) specifically includes: Step 3-1) Collect rich resource language texts and use them as training corpus, and select texts involving specific domain content as the basis for extracting domain feature words; Step 3-2) Calculate the information gain value of the words. The information gain value is the difference between the entropy of the document without considering any word features and the entropy of the document after considering word features: in, The entropy of the document without considering any word features; The entropy of the document after taking word features into account; for The probability of a document of a certain type appearing in the training corpus; To train the corpus containing word feature words h The probability of a document; Documents containing word feature words And belong to Conditional probability of class documents; The training corpus does not contain word feature words. The probability of a document; The document does not contain word feature words. And belong to Conditional probability of class documents; This represents the total number of classifications applied to the training corpus. The training corpus is divided into two categories: domain-specific and non-application-specific. Step 3-3) Repeat step 3-2) Calculate the information gain value of all words in each text involving specific domain content in the training corpus as feature words, and sort all words in each text involving specific domain content in descending order of information gain value. Based on experience, select the top 500 to 10,000 words as selected domain feature words.

4. The method for modeling a language model of scarce resources in a specific service domain according to claim 1, characterized in that, Step 6) specifically includes: Assuming a preprocessed set of statements ,in, M For the number of statements, For the first statement in the preprocessed statement set i A split statement; with exist Sentence-topic distribution as a numerical representation of word LDA vectors: in, express In the j Distribution values ​​across the themes; The cosine similarity method is used to calculate the relevance between statement-topic distribution and domain-topic distribution. : in, for In the statement-topic distribution j The component values ​​of each theme; The first in the domain topic distribution j The component values ​​of each theme; Repeat the above process to obtain the correlation between the statement-topic distribution and the domain topic distribution for each segmented statement.

5. The method for modeling a language model of scarce resources in a specific service domain according to claim 1, characterized in that, Step 7) specifically includes: Determine whether the relevance of each statement exceeds a preset threshold. If the relevance of a statement is greater than a preset threshold, then statements that meet the criteria will be included in the language model training corpus for a scarce resource specific domain. If the relevance of a statement is less than or equal to a preset threshold, then the statement is deleted.

6. The method for modeling a language model of scarce resources in a specific service domain according to claim 1, characterized in that, Step 8) specifically includes: Step 8-1) If the selected delimiter is at the beginning of a paragraph, do not select the text preceding that delimiter; If the selected segmentation statement is not at the beginning of the paragraph, then the text preceding the segmentation statement is selected as the candidate corpus and included in the language model training corpus for the scarce resource specific domain. Step 8-2) If the selected segmentation statement is located at the end of a paragraph, then the text following that segmentation statement will not be selected; If the selected segmentation statement is not located at the end of the paragraph, then the text following the segmentation statement is selected as the candidate corpus and included in the training corpus of the language model for the scarce resource specific domain. Step 8-3) Based on Step 8-1) and Step 8-2) above, the language model training corpus obtained in Step 7) is screened again and deduplicated to obtain the final language model training corpus; Specifically, the two candidate corpora obtained above are deduplicated. If they are already in the training corpora obtained in step 7), they are not counted again. The deduplicated candidate corpora are then included in the training corpora.

7. The method for modeling a language model of scarce resources in a specific service domain according to claim 1, characterized in that, Step 9) specifically includes: use n - The modeling method of the Gram language model, specifically n The value can be 2, 3, or 4: 2-Gram language model: in, Represents a split statement The generation probability; The first in the training corpus of the scarce resource language model n Words The probability of occurrence; for The probability of occurrence in the entire final language model training corpus; In the corpus Under the conditions that it occurs, and The conditional probability of occurrence; In the corpus Under the conditions that it occurs, and The conditional probability of occurrence; 3-Gram language model: in, In the corpus and Under the condition of simultaneous occurrence, and The conditional probability of occurrence; 4-Gram language model: in, In the corpus 、 and Under the condition of simultaneous occurrence, and The conditional probability of occurrence; in, The distance to the nth word The first word; The distance to the nth word The first two words; The distance to the nth word The first three words; In the formulas for the 2, 3, and 4-Gram language models above, the maximum likelihood method is used to estimate the words in the final language model training corpus. probability of occurrence and conditional probability probability of occurrence The specific formula is as follows: in, For words The frequency of occurrence in the training corpus; To count the total word frequency in the training corpus; The total set of words involved in the training corpus for the final language model; The formula for calculating conditional probability is as follows: in, For words Frequency of occurrence in the training corpus; For words Frequency of occurrence in the training corpus; Finally, by calculating the information about words on the final language model training corpus. probability of occurrence and conditional probability 、 and This led to the development of 2-Gram, 3-Gram, and / or 4-Gram language models as scarce resource language models, forming scarce resource language models that serve specific domains.

8. An apparatus for performing the method for modeling a language model of scarce resources in a specific service domain as described in claim 1, characterized in that, The device includes: The preprocessing module is used to collect scarce resource texts as initial corpus, preprocess the initial corpus to obtain preprocessed text; and segment the preprocessed text into sentences, preserving the order of sentences and paragraph information to obtain a preprocessed sentence set composed of multiple segmented sentences. The LDA analysis module is used to obtain the statement-topic distribution and topic-word distribution from the preprocessed statement set using the Gibbs algorithm. The domain topic distribution calculation module is used to select domain feature words for a specific domain from pre-collected rich resource language texts to obtain selected domain feature words; translate the selected domain feature words into words in rare languages ​​to obtain translated domain feature words; and use the translated domain feature words, combined with the obtained topic-word distribution, to calculate and obtain the domain topic distribution. The domain corpus selection module is used to calculate the relevance between the statement-topic distribution and the domain topic distribution of each segmented statement; determine whether the relevance of each segmented statement exceeds a preset threshold, and select statements that meet the condition into the scarce resource of specific domain language model training corpus; further filter the obtained language model training corpus based on the coherence of language expression to obtain the final language model training corpus; and The domain language model training module is used to train the scarce resource language model using the final language model training corpus, thus obtaining the scarce resource language model.

9. An electronic device for modeling a language model of scarce resources serving a specific domain, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1-7.

Images