A method and system for extracting new words in a field

Abstract

The present invention provides a method and system for extracting new domain words, including collecting industry content data and extracting text data from the industry content data to obtain a professional domain text data set; converting the professional domain text data set according to a preset domain syntax dictionary to obtain a corresponding word vector data set; retrieving candidate words from the word vector data set that do not belong to the preset domain syntax dictionary, and if the cosine similarity of the candidate words and the cosine similarity of the preset domain syntax dictionary is greater than a preset threshold, the candidate words are determined to be candidate new words; filtering the candidate new words according to preset filtering rules to obtain domain new words. The present invention can realize the extraction of vocabulary in a specific domain and the extraction of new words in a domain by discovering new words and integrating multiple new words. It improves the accuracy of word segmentation, thereby improving the performance of natural language processing downstream tasks.

Description

Technical Field

[0001] The present invention relates to the technical field of extracting new domain words, and in particular to a method and system for extracting new domain words. Background Art

[0002] With the advancement of data-driven technology, new domain terms and colloquial expressions for specific business scenarios are being updated at an increasingly rapid pace across vertical industry UGC (User Generated Content) data, professional literature, industry news, internal corporate references, and other specialized fields. However, rapidly discovering new domain terms from massive, heterogeneous industry corpora and expanding domain lexicons are crucial for improving the accuracy of lexical analysis tasks and, in turn, the performance of downstream tasks related to unstructured data analysis. Therefore, providing relevant practitioners with the latest information in the field, enabling them to better explore and analyze the latest events, development trends, and community sentiment in the field, and capturing accurate material for office writing has become a major challenge. Summary of the Invention

[0003] The purpose of this invention is to propose a method and system for extracting new words in a field, so as to improve the accuracy of word segmentation for industry UGC, professional literature, and content data, thereby improving the accuracy of downstream tasks such as semantic retrieval and intelligent recommendation.

[0004] On the one hand, a method for extracting new domain words is provided, comprising:

[0005] Collecting industry content data and extracting text data from the industry content data to obtain a professional field text dataset;

[0006] Convert the professional domain text dataset according to a preset domain syntax dictionary to obtain a corresponding word vector dataset;

[0007] Retrieving candidate words from the word vector dataset that do not belong to the preset domain syntax dictionary, comparing the cosine similarity of the candidate words with the preset domain syntax dictionary, and determining that the candidate word is a candidate new word if the cosine similarity of the candidate word and the cosine similarity of the preset domain syntax dictionary is greater than a preset threshold;

[0008] The candidate new words are filtered using preset filtering rules to obtain domain new words.

[0009] Preferably, obtaining the preset domain syntax dictionary includes the following process:

[0010] Clean and annotate the professional field text dataset according to the preset general field dictionary to obtain preliminary sequence annotation results;

[0011] The preliminary sequence annotation results are supplemented with annotations based on preset standard vocabulary in the electric power professional field to obtain a professional field dictionary.

[0012] Preferably, obtaining the preset domain syntax dictionary through the following process also includes:

[0013] Re-segment and re-tag the professional domain dictionary sentence by sentence according to a preset natural language processing algorithm, and calculate the TF-IDF value of each domain vocabulary; and analyze the professional domain text dataset sentence by sentence according to a preset natural language processing algorithm to obtain a vocabulary sequence with dependency relationships and a part-of-speech tagging sequence corresponding to each vocabulary sequence;

[0014] Counting the number of dependency vocabulary sequences according to the part-of-speech tagging sequence, and when the number of dependency vocabulary sequences that meet a specific part-of-speech tagging sequence exceeds a preset threshold, adding the part-of-speech tagging sequence to a syntax template list to obtain a domain syntax template;

[0015] The vocabulary sequence is extracted without repetition. When there are words in the vocabulary sequence whose tf-idf value is greater than the judgment threshold and the vocabulary sequence contains words in the professional domain dictionary, the vocabulary sequence is added to the domain syntax template to obtain a preset domain syntax dictionary.

[0016] Preferably, obtaining the corresponding word vector dataset specifically includes:

[0017] Domain syntax templates extracted from a preset domain syntax dictionary;

[0018] Extracting words or phrases that conform to the domain syntax template from a professional domain text dataset according to the domain syntax template, and training the extracted words or phrases through the language module in the domain syntax template to obtain corresponding word vectors;

[0019] All corresponding word vectors are combined into corresponding word vector datasets.

[0020] Preferably, the preset filtering rules specifically include:

[0021] Frequency filtering, which is used to count the occurrence frequencies of candidate new words in the word vector dataset and filter out candidate new words whose occurrence frequencies are less than a preset occurrence frequency threshold;

[0022] The longest substring filter is used to extract candidate new words. When extracting candidate new words, the candidate new words will be extracted according to the preset length and a candidate new word will be marked as a substring of the associated candidate new word. If the frequency of occurrence of the longest substring is equal to that of the associated candidate new word, the candidate new word will be filtered out.

[0023] Common word filtering, used to filter out candidate new words that appear in the preset general domain dictionary;

[0024] Word formation rule filtering is used to filter candidate new words whose word formation part-of-speech sequences cannot be extracted through part-of-speech tagging.

[0025] On the other hand, a system for extracting new domain words is also provided, for implementing the method for extracting new domain words, comprising:

[0026] A data collection module is used to collect industry content data and extract text data from the industry content data to obtain a professional field text data set;

[0027] A word vector module is used to transform the professional field text dataset according to a preset field syntax dictionary to obtain a corresponding word vector dataset;

[0028] A new word screening module is used to retrieve candidate words from the word vector dataset that do not belong to the preset domain syntax dictionary, compare the cosine similarity of the candidate words with the preset domain syntax dictionary, and if the cosine similarity of the candidate words and the cosine similarity of the preset domain syntax dictionary is greater than a preset threshold, the candidate words are determined to be candidate new words; and the candidate new words are filtered according to preset filtering rules to obtain domain new words.

[0029] Preferably, the word vector module is further used to clean and annotate the professional field text dataset according to a preset general field dictionary to obtain a preliminary sequence annotation result;

[0030] The preliminary sequence annotation results are supplemented with annotations based on preset standard vocabulary in the electric power professional field to obtain a professional field dictionary.

[0031] Preferably, the word vector module is further used to re-segment and re-tag the professional domain dictionary sentence by sentence according to a preset natural language processing algorithm, and to count the TF-IDF value of each domain vocabulary; and to analyze the professional domain text dataset sentence by sentence according to a preset natural language processing algorithm to obtain a vocabulary sequence with dependency relationships and a part-of-speech tagging sequence corresponding to each vocabulary sequence;

[0032] Counting the number of dependency vocabulary sequences according to the part-of-speech tagging sequence, and when the number of dependency vocabulary sequences that meet a specific part-of-speech tagging sequence exceeds a preset threshold, adding the part-of-speech tagging sequence to a syntax template list to obtain a domain syntax template;

[0033] The vocabulary sequence is extracted without repetition. When there are words in the vocabulary sequence whose tf-idf value is greater than the judgment threshold and the vocabulary sequence contains words in the professional domain dictionary, the vocabulary sequence is added to the domain syntax template to obtain a preset domain syntax dictionary.

[0034] Preferably, the word vector module is further used for extracting domain syntax templates from a preset domain syntax dictionary;

[0035] Extracting words or phrases that conform to the domain syntax template from a professional domain text dataset according to the domain syntax template, and training the extracted words or phrases through the language module in the domain syntax template to obtain corresponding word vectors;

[0036] All corresponding word vectors are combined into corresponding word vector datasets.

[0037] Preferably, the new word screening module is further configured to filter according to the following preset filtering rules:

[0038] Frequency filtering, which is used to count the occurrence frequencies of candidate new words in the word vector dataset and filter out candidate new words whose occurrence frequencies are less than a preset occurrence frequency threshold;

[0039] The longest substring filter is used to extract candidate new words. When extracting candidate new words, the candidate new words will be extracted according to the preset length and a candidate new word will be marked as a substring of the associated candidate new word. If the frequency of occurrence of the longest substring is equal to that of the associated candidate new word, the candidate new word will be filtered out.

[0040] Common word filtering, used to filter out candidate new words that appear in the preset general domain dictionary;

[0041] Word formation rule filtering is used to filter candidate new words whose word formation part-of-speech sequences cannot be extracted through part-of-speech tagging.

[0042] In summary, the implementation of the embodiments of the present invention has the following beneficial effects:

[0043] The method and system for extracting new domain words provided by the present invention can realize the extraction of vocabulary in a specific domain by discovering new words and integrating multiple new word extraction methods. It can realize multi-angle and comprehensive new domain word extraction, and can improve the word segmentation accuracy of text analysis of professional documents and industry news, thereby improving the performance of downstream natural language processing tasks.

[0044] The system collects industry content data, including professional literature, industry news, internal company references, work logs, and user-generated content (UGC). The extracted text data undergoes preprocessing, including sentence segmentation, word segmentation, removal of stop words and meaningless symbols, part-of-speech tagging, and syntactic analysis. Two new word discovery algorithms are used to discover new words in the domain. An N-gram incremental algorithm is used to extract strings within a fixed length range from the corpus as candidate new words. New word candidates are identified using statistics such as information gain, adjacency entropy, term frequency-inverse document frequency, dice coefficient, adjacency category, and pointwise mutual information, combined with various screening rules. New word extraction and new word identification are implemented based on dependency syntactic analysis and word vector representation, respectively. Dependency syntactic analysis generates syntactic templates, which are then used to determine new word boundaries. The word2vec model is trained using the collected corpus data, converting candidate new words into vector representations. The semantic similarity between the candidate words and existing words is calculated to determine new words in the domain. A manual annotation task is then automatically generated, and the user confirms the completion of the new word approval and storage. By integrating the two new word discovery methods, the efficiency of new word discovery is improved and the workload of manual annotation is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, without paying any creative work, other drawings obtained based on these drawings still fall within the scope of the present invention.

[0046] Figure 1 Schematic diagram of the main process of a method for extracting new domain words in an embodiment of the present invention.

[0047] Figure 2 Schematic diagram of a system for extracting new domain words in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be described in further detail below with reference to the accompanying drawings.

[0049] like Figure 1 FIG. 1 is a schematic diagram of an embodiment of a method for extracting new domain words provided by the present invention. In this embodiment, the method includes the following steps:

[0050] Collect industry content data and extract text data from the industry content data to obtain a professional field text dataset; that is, collect industry content data such as professional literature, industry news, corporate internal references, work logs, industry UGC content, etc., extract the text data therein, and obtain a standardized professional field text dataset.

[0051] Furthermore, the professional field text dataset is transformed according to the preset domain syntax dictionary to obtain the corresponding word vector dataset; that is, based on the preset domain syntax dictionary (including syntax dictionary and syntax template), combined with dependency syntax analysis and word vector technology, new domain words are discovered for the latest text collection in a specific field.

[0052] In a specific embodiment, obtaining the preset domain syntax dictionary includes the following process:

[0053] The professional field text dataset is cleaned and annotated according to the preset general field dictionary to obtain preliminary sequence annotation results; that is, based on the general field dictionary, the extracted text data is preprocessed using the hanLP natural language processing toolkit to complete the basic cleaning and processing operations of sentence segmentation, word segmentation, stop word removal, and punctuation removal to obtain preliminary sequence annotation results for the collected corpus.

[0054] The preliminary sequence annotation results are supplemented with annotations based on the preset standard vocabulary in the electric power professional field to obtain a professional field dictionary; that is, the sequence annotation results are verified (manual or model-based automatic verification can be used), and valuable domain entities, entity attributes, proper nouns, terminology and other field vocabulary in the electric power professional field are supplemented with annotations to form a professional field dictionary D1.

[0055] The professional domain dictionary is re-segmented and tagged sentence by sentence according to a preset natural language processing algorithm, and the TF-IDF value of each domain word is calculated. The professional domain text dataset is then analyzed sentence by sentence according to the preset natural language processing algorithm to obtain a vocabulary sequence with dependency relationships and a POS tag sequence corresponding to each vocabulary sequence. In other words, the standardized electric power professional domain text dataset obtained by the first step of cleaning is loaded; the domain dictionary D1 obtained in the first step is loaded and re-segmented and POS tagged sentence by sentence using the hanLP natural language processing toolkit; the TF-IDF value of each domain word is calculated to generate a domain vocabulary L1 extracted from the current corpus, each of which contains a row of data (Word, posTag, TF, TF-IDF). The hanLP natural language processing toolkit is used to perform dependency syntactic analysis on the electric power professional domain text dataset sentence by sentence, and based on each identified dependency relationship, a vocabulary sequence with dependency relationships and a POS tag sequence corresponding to each vocabulary sequence are obtained. F-IDF (term frequency–inverse document frequency) is a commonly used weighting technique used in information retrieval and data mining. TF stands for Term Frequency, and IDF stands for Inverse Document Frequency. TF-IDF is a statistical method used to assess the importance of a word to a document set or a document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in a document, but decreases in inverse proportion to the frequency of its appearance in the corpus. Various forms of TF-IDF weighting are often used by search engines as a measure or rating of the relevance between a document and a user query. In addition to TF-IDF, search engines on the Internet also use ranking methods based on link analysis to determine the order in which documents appear in search results.

[0056] The number of dependency word sequences is counted by grouping the part-of-speech tagging sequences. When the number of dependency word sequences that match a specific part-of-speech tagging sequence exceeds a preset threshold, the part-of-speech tagging sequence is added to the syntax template list to obtain a domain syntax template. That is, the number of dependency word sequences is counted by grouping the part-of-speech tagging sequences. When the number of dependency word sequences that match a specific part-of-speech tagging sequence exceeds a threshold θ1, (θ1>1), the part-of-speech tagging sequence is added to the syntax template list. A complete set of syntax templates for the current corpus is generated.

[0057] The vocabulary sequence is extracted without duplication. When there are words in the vocabulary sequence whose tf-idf value is greater than the judgment threshold and the vocabulary sequence contains words in the professional domain dictionary, the vocabulary sequence is added to the domain syntax template to obtain a preset domain syntax dictionary. That is, the non-repeated dependency vocabulary sequences extracted from the corpus are analyzed one by one. When each dependency vocabulary sequence contains important domain vocabulary and the dependency vocabulary sequence is an instance of a specific template in the syntax template, the dependency vocabulary sequence is added to the syntax dictionary. The following rules are used to determine whether a word is an important word in the domain: a. There are words in the sequence whose tf-idf value is greater than a specific threshold θ2 (for the collected professional corpus, when θ2 is 0.01, the algorithm's F1-Score is the highest); b. There are words in the professional domain dictionary D1 in the sequence.

[0058] Specifically, the corresponding word vector data set is obtained by: extracting a domain syntax template from a preset domain syntax dictionary; extracting words or phrases that conform to the domain syntax template from a professional domain text data set based on the domain syntax template, and training the extracted words or phrases through the language module in the domain syntax template to obtain corresponding word vectors; and forming all corresponding word vectors into a corresponding word vector data set. That is, first, based on the extracted syntax template, extract words or phrases that conform to the template from the collected text, and use the word2vec module in gensim to train word vectors. The Word2Vec module loads the training corpus data set, trains professional domain word vectors, sets model parameters according to the following configuration, and trains word vectors. Where size: the dimension of each word vector; the word vector dimension is set to 200; window: the size of the context scan window during word vector training, set to 5, that is, considering the first 5 and last 5 words of the sample sentence; min-count: sets the minimum frequency of word occurrence; if a word appears less than 3 times in a document, it is discarded; workers: the number of concurrent training processes, which defaults to the number of processor cores on the current running machine. sg ({0,1}, optional) – the model training algorithm: 1: skip-gram; 0: CBOW. When the corpus dataset is less than 500MB, the skip-gram model is used; when the corpus dataset is larger, the CBOW model is used. alpha (float, optional) – the initial learning rate; iter (int, optional) – the number of iterations, defaults to 5; save the word vector dataset generated by training. Based on the domain word vector dataset generated by training, the group-max vector fusion algorithm is used to generate a vectorized representation of each word group in the syntactic dictionary, that is, the word vector representation. The specific conversion method is: compare the i-th dimension of the word vector of each word that constitutes the phrase, take the maximum value as the i-th dimension of the phrase vector, thereby obtaining the vector representation of each phrase in the syntactic dictionary, and saving the vectorized representation of the syntactic dictionary.

[0059] Furthermore, candidate words in the word vector dataset that do not belong to the preset domain syntax dictionary are retrieved, and the cosine similarity of the candidate words is compared with the preset domain syntax dictionary. If the cosine similarity of the candidate word and the cosine similarity of the preset domain syntax dictionary is greater than a preset threshold, the candidate word is determined to be a candidate new word; that is, the cosine similarity of the vector representation of each word or phrase with the professional domain dictionary and syntax dictionary is calculated. In the syntax template library, for unregistered instances that meet the template, if there is a registered phrase or word with a similarity greater than a certain threshold, the unregistered word can be considered a candidate new word belonging to a certain field.

[0060] Furthermore, the candidate new words are filtered using a preset filtering rule to obtain domain new words. That is, the candidate new words are filtered using a trained filtering rule model to improve the domain conformity and novelty of the domain new words discovered.

[0061] In a specific embodiment, the preset filtering rules specifically include: frequency filtering, which is used to count the frequency of occurrence of candidate new words in the word vector data set, and filter out candidate new words whose frequency of occurrence is less than a preset frequency threshold; in the new word discovery task, meaningful new words will appear multiple times, representing meaningful expressions in a specific context. A large number of low-frequency character strings are filtered out by setting a character string frequency threshold. Longest substring filtering, which is used to extract candidate new words. When extracting candidate new words, candidate new words will be extracted according to a preset length and a candidate new word will be marked as a substring of an associated candidate new word. If the frequency of occurrence of the substring with the longest length is equal to that of the associated candidate new word, the candidate new word will be filtered out; since the process of extracting candidate new words is to divide the corpus into strings of a certain length, there will be a candidate new word that is a substring of another candidate new word. Set the longest substring filtering rule. If the frequency of occurrence of the longest substring is equal to that of the parent string, it means that the longest substring does not appear as a single word, but appears as part of the parent string. The longest substring with the frequency of occurrence of the longest substring equal to that of the parent string will be filtered out. Common word filtering is used to filter out candidate new words that appear in the pre-set general domain dictionary and filter out candidate words that appear in the registered dictionary (basic vocabulary). Word formation rule filtering is used to filter out candidate new words whose word formation sequence cannot be extracted through part-of-speech tagging. Word formation sequence rules are extracted through part-of-speech tagging and candidate words that do not meet the rules are filtered out.

[0062] Specifically, when extracting candidate new words, strings within a fixed length range are extracted from the corpus, and the string length is set to 2 to N. Depending on the situation of the corpus, N is set to a maximum of 7. N-gram is used to segment the candidate new words and count the word frequencies. Since the candidate strings are extracted directly from the corpus by segmenting the text, which contains a large number of low-frequency strings, a threshold is usually set for the frequency of occurrence. Only high-frequency strings exceeding the threshold are retained, and strings below the threshold are filtered out, thereby greatly reducing the scale of candidate new words and reducing the computational complexity of subsequent operations. Mutual information is calculated to evaluate the cohesion degree, and left and right adjacency entropy is used to measure the degree of freedom of candidate new words. Based on the calculated statistics, a variety of filtering rules are applied to filter candidate words. The main rules are:

[0063] Frequency filtering: In new word discovery tasks, meaningful new words often appear multiple times, representing meaningful expressions in a specific context. By setting a frequency threshold for string occurrences, we can filter out a large number of low-frequency strings.

[0064] Longest substring filtering: Because the process of extracting candidate new words involves segmenting the corpus into strings of a certain length, it's possible that a candidate new word is a substring of another candidate new word. We set a longest substring filtering rule. If the longest substring has the same frequency as the parent string, it means that the longest substring is not a standalone word but appears as part of the parent string. These longest substrings with the same frequency as the parent string will be filtered out.

[0065] Common word filtering: Filter out candidate words that appear in the registered dictionary (basic vocabulary).

[0066] Word formation rule filtering: Extract word formation part-of-speech sequence rules through part-of-speech tagging and filter candidate words that do not meet the rules.

[0067] Unfiltered candidate words are transformed into new words and enter the new word annotation task, which is manually reviewed. After passing the manual review, the discovered new words are imported into the domain dictionary.

[0068] like Figure 2 FIG. 1 is a schematic diagram of an embodiment of a system for extracting new domain words provided by the present invention. In this embodiment, the system includes:

[0069] The data collection module is used to collect industry content data and extract text data from the industry content data to obtain a professional field text data set.

[0070] The word vector module is used to transform the professional field text dataset according to the preset field syntax dictionary to obtain the corresponding word vector dataset.

[0071] Specifically, the word vector module is also used to clean and annotate the professional field text data set according to the preset general field dictionary to obtain a preliminary sequence annotation result; and supplement the preliminary sequence annotation result according to the preset standard vocabulary in the power professional field to obtain a professional field dictionary.

[0072] The word vector module is also used to re-segment and re-tag the professional domain dictionary sentence by sentence according to a preset natural language processing algorithm, and count the tf-idf value of each domain vocabulary; and analyze the professional domain text data set sentence by sentence according to the preset natural language processing algorithm to obtain a vocabulary sequence with dependency and a part-of-speech tagging sequence corresponding to each vocabulary sequence; group and count the number of dependency vocabulary sequences according to the part-of-speech tagging sequence, and when the number of dependency vocabulary sequences that meet a specific part-of-speech tagging sequence exceeds a preset threshold, add the part-of-speech tagging sequence to the syntax template list to obtain a domain syntax template; extract the vocabulary sequence without duplication, and when there is a vocabulary with a tf-idf value greater than the judgment threshold in the vocabulary sequence and the vocabulary sequence contains a vocabulary in the professional domain dictionary, add the vocabulary sequence to the domain syntax template to obtain a preset domain syntax dictionary.

[0073] The word vector module is also used to extract a domain syntax template from a preset domain syntax dictionary; extract words or phrases that conform to the domain syntax template from a professional domain text dataset based on the domain syntax template, and train the extracted words or phrases through the language module in the domain syntax template to obtain corresponding word vectors; and combine all corresponding word vectors into a corresponding word vector dataset.

[0074] A new word screening module is used to retrieve candidate words from the word vector dataset that do not belong to the preset domain syntax dictionary, compare the cosine similarity of the candidate words with the preset domain syntax dictionary, and if the cosine similarity of the candidate words and the cosine similarity of the preset domain syntax dictionary is greater than a preset threshold, the candidate words are determined to be candidate new words; and the candidate new words are filtered according to preset filtering rules to obtain domain new words.

[0075] Specifically, the new word screening module is also used to filter according to the following preset filtering rules: frequency filtering, which is used to count the occurrence frequency of candidate new words in the word vector data set, and filter out candidate new words whose occurrence frequency is less than a preset occurrence frequency threshold; longest substring filtering, which is used to extract candidate new words according to a preset length and mark a candidate new word as a substring of an associated candidate new word. If the occurrence frequency of the substring with the longest length is equal to that of the associated candidate new word, the candidate new word is filtered out; common word filtering, which is used to filter out candidate new words that appear in a preset general domain dictionary; word formation rule filtering, which is used to filter out candidate new words whose word formation part-of-speech sequence cannot be extracted through part-of-speech tagging.

[0076] In summary, the implementation of the embodiments of the present invention has the following beneficial effects:

[0077] The method and system for extracting new domain words provided by the present invention can realize the extraction of vocabulary in a specific domain by discovering new words and integrating multiple new word extraction methods. It can realize multi-angle and comprehensive new domain word extraction, and can improve the word segmentation accuracy of text analysis of professional documents and industry news, thereby improving the performance of downstream natural language processing tasks.

[0078] The system collects industry content data, including professional literature, industry news, internal company references, work logs, and user-generated content (UGC). The extracted text data undergoes preprocessing, including sentence segmentation, word segmentation, removal of stop words and meaningless symbols, part-of-speech tagging, and syntactic analysis. Two new word discovery algorithms are used to discover new words in the domain. An N-gram incremental algorithm is used to extract strings within a fixed length range from the corpus as candidate new words. New word candidates are identified using statistics such as information gain, adjacency entropy, term frequency-inverse document frequency, dice coefficient, adjacency category, and pointwise mutual information, combined with various screening rules. New word extraction and new word identification are implemented based on dependency syntactic analysis and word vector representation, respectively. Dependency syntactic analysis generates syntactic templates, which are then used to determine new word boundaries. The word2vec model is trained using the collected corpus data, converting candidate new words into vector representations. The semantic similarity between the candidate words and existing words is calculated to determine new words in the domain. A manual annotation task is then automatically generated, and the user confirms the completion of the new word approval and storage. By integrating the two new word discovery methods, the efficiency of new word discovery is improved and the workload of manual annotation is reduced.

[0079] The above disclosure is merely a preferred embodiment of the present invention and certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A method for extracting new domain words, characterized in that: include: Collecting industry content data and extracting text data from the industry content data to obtain a professional field text dataset; Convert the professional domain text dataset according to a preset domain syntax dictionary to obtain a corresponding word vector dataset; Retrieving candidate words from the word vector dataset that do not belong to the preset domain syntax dictionary, comparing the cosine similarity of the candidate words with the preset domain syntax dictionary, and determining that the candidate word is a candidate new word if the cosine similarity of the candidate word and the cosine similarity of the preset domain syntax dictionary is greater than a preset threshold; Filter the candidate new words using preset filtering rules to obtain domain new words; The process of obtaining the preset domain syntax dictionary includes: Clean and annotate the professional field text dataset according to the preset general field dictionary to obtain preliminary sequence annotation results; Supplementing the annotation of the preliminary sequence annotation results according to the preset standard vocabulary in the electric power professional field to obtain a professional field dictionary; Re-segment and re-tag the professional domain dictionary sentence by sentence according to a preset natural language processing algorithm, and calculate the TF-IDF value of each domain vocabulary; and analyze the professional domain text dataset sentence by sentence according to a preset natural language processing algorithm to obtain a vocabulary sequence with dependency relationships and a part-of-speech tagging sequence corresponding to each vocabulary sequence; Counting the number of dependency vocabulary sequences according to the part-of-speech tagging sequence, and when the number of dependency vocabulary sequences that meet a specific part-of-speech tagging sequence exceeds a preset threshold, adding the part-of-speech tagging sequence to a syntax template list to obtain a domain syntax template; The vocabulary sequence is extracted without repetition. When there are words in the vocabulary sequence whose tf-idf value is greater than the judgment threshold and the vocabulary sequence contains words in the professional domain dictionary, the vocabulary sequence is added to the domain syntax template to obtain a preset domain syntax dictionary.

2. The method according to claim 1, wherein The obtaining of the corresponding word vector dataset specifically includes: Domain syntax templates extracted from a preset domain syntax dictionary; Extracting words or phrases that conform to the domain syntax template from a professional domain text dataset according to the domain syntax template, and training the extracted words or phrases through the language module in the domain syntax template to obtain corresponding word vectors; All corresponding word vectors are combined into corresponding word vector datasets.

3. The method according to claim 1, wherein The preset filtering rules specifically include: Frequency filtering, which is used to count the occurrence frequencies of candidate new words in the word vector dataset and filter out candidate new words whose occurrence frequencies are less than a preset occurrence frequency threshold; The longest substring filter is used to extract candidate new words. When extracting candidate new words, the candidate new words will be extracted according to the preset length and a candidate new word will be marked as a substring of the associated candidate new word. If the frequency of occurrence of the longest substring is equal to that of the associated candidate new word, the candidate new word will be filtered out. Common word filtering, used to filter out candidate new words that appear in the preset general domain dictionary; Word formation rule filtering is used to filter candidate new words whose word formation part-of-speech sequences cannot be extracted through part-of-speech tagging.

4. A system for extracting new words in a field, used to implement the method according to any one of claims 1 to 3, characterized in that: include: A data collection module is used to collect industry content data and extract text data from the industry content data to obtain a professional field text data set; A word vector module is used to transform the professional field text dataset according to a preset field syntax dictionary to obtain a corresponding word vector dataset; A new word screening module is used to retrieve candidate words from the word vector dataset that do not belong to the preset domain syntax dictionary, compare the cosine similarity of the candidate words with the preset domain syntax dictionary, and if the cosine similarity of the candidate words and the cosine similarity of the preset domain syntax dictionary is greater than a preset threshold, determine that the candidate words are candidate new words; and filter the candidate new words according to preset filtering rules to obtain domain new words; The word vector module is also used to clean and annotate professional field text datasets according to a preset general field dictionary to obtain preliminary sequence annotation results; Supplementing the annotation of the preliminary sequence annotation results according to the preset standard vocabulary in the electric power professional field to obtain a professional field dictionary; The word vector module is also used to re-segment and re-tag the professional domain dictionary sentence by sentence according to a preset natural language processing algorithm, and calculate the TF-IDF value of each domain vocabulary; and analyze the professional domain text data set sentence by sentence according to the preset natural language processing algorithm to obtain a vocabulary sequence with dependency relationship and a part-of-speech tagging sequence corresponding to each vocabulary sequence; Counting the number of dependency vocabulary sequences according to the part-of-speech tagging sequence, and when the number of dependency vocabulary sequences that meet a specific part-of-speech tagging sequence exceeds a preset threshold, adding the part-of-speech tagging sequence to a syntax template list to obtain a domain syntax template; The vocabulary sequence is extracted without repetition. When there are words in the vocabulary sequence whose tf-idf value is greater than the judgment threshold and the vocabulary sequence contains words in the professional domain dictionary, the vocabulary sequence is added to the domain syntax template to obtain a preset domain syntax dictionary.

5. The system according to claim 4, wherein: The word vector module is also used to extract domain syntax templates from a preset domain syntax dictionary; Extracting words or phrases that conform to the domain syntax template from a professional domain text dataset according to the domain syntax template, and training the extracted words or phrases through the language module in the domain syntax template to obtain corresponding word vectors; All corresponding word vectors are combined into corresponding word vector datasets.

6. The system according to claim 5, wherein: The new word screening module is also used to filter according to the following preset filtering rules: Frequency filtering, which is used to count the occurrence frequencies of candidate new words in the word vector dataset and filter out candidate new words whose occurrence frequencies are less than a preset occurrence frequency threshold; The longest substring filter is used to extract candidate new words. When extracting candidate new words, the candidate new words will be extracted according to the preset length and a candidate new word will be marked as a substring of the associated candidate new word. If the frequency of occurrence of the longest substring is equal to that of the associated candidate new word, the candidate new word will be filtered out. Common word filtering, used to filter out candidate new words that appear in the preset general domain dictionary; Word formation rule filtering is used to filter candidate new words whose word formation part-of-speech sequences cannot be extracted through part-of-speech tagging.

Images