Field new word extraction method, device, electronic device and storage medium
Through natural language processing model and graph algorithm, a new field word graph model is constructed, which solves the accuracy and efficiency of field word extraction in the existing technology, and realizes the accurate screening and updating of field new words.
Patent Information
- Application Number
- CN202311564031.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-22
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-11-22
AI Technical Summary
It is difficult for the prior art to accurately extract new words in the field, especially in large-scale text data processing, and relying on artificial experience and word frequency statistics methods have problems such as subjectivity and difficulty in discriminating polysemous words.
The domain new word extraction model obtained by using natural language processing model training is constructed through text preprocessing and BERT model training, and the domain new word graph model is constructed in combination with graph algorithms. The PageRank algorithm and graph embedding technology are used to filter and evaluate candidate new words to generate word cloud And update the new thesaurus.
It realizes more accurate extraction and screening of new words in the field, improves the efficiency and accuracy of new words extraction, and reduces the limitations of manual intervention and word frequency statistics.
Smart Images

Figure CN117610552B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for extracting new words in a field. Background Art
[0002] New words in a field refer to new vocabulary or terms that appear in a specific field or profession. With the development of science and technology, new concepts, technologies and phenomena are constantly emerging in different fields and industries. New words in a field play an important role in professional exchanges, academic research, industry reporting and internal communication in a field. They help to accurately convey information in a specific field, improve communication efficiency, and promote further development and innovation in the field.
[0003] In academic research and literature analysis, the extraction of new domain words can help researchers track and understand the latest research results and topics; in information retrieval and text mining, new domain words can also be used to improve the search effect and accuracy. However, due to the particularity and professionalism of new domain words, existing methods for extracting new domain words, such as manually marking and identifying new words in the field, rely on manual experience and professional knowledge, are easily affected by subjective factors, and cannot process large-scale text data; or, through frequency statistics, analyze the frequency of occurrence of words in the text and the background knowledge of a specific field, and determine words with high frequency but less occurrence in the general corpus as new domain words. This method cannot accurately determine whether a word is a new word, and it is also difficult to distinguish polysemous words.
[0004] Therefore, there is an urgent need for a method, device, electronic device and storage medium for extracting new words in a field to solve the above problems. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides a method, device, electronic device and storage medium for extracting new words in a field.
[0006] The present invention provides a method for extracting new words in a field, comprising:
[0007] Obtain target text data for field new word extraction;
[0008] Input the target text data into a domain new word extraction model to obtain the new word probability of each target word in the target text data output by the domain new word extraction model as a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels;
[0009] Determining candidate domain new words from a plurality of the target words according to the new word probability of each of the target words;
[0010] Constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data;
[0011] According to the domain new word graph model, a target domain new word is determined from a plurality of candidate domain new words.
[0012] According to a method for extracting new words in a domain provided by the present invention, the new word extraction model in the domain is trained by the following steps:
[0013] Acquire sample text data corresponding to various industry fields, and perform text preprocessing on the sample text data to obtain preprocessed sample text data, wherein the text preprocessing at least includes sentence segmentation processing, word segmentation processing, stop word removal processing, and punctuation removal processing;
[0014] By using domain new word labels, sample domain new words in each of the sample text data after the text preprocessing are marked, and sample text data sets of various industry fields are constructed;
[0015] The pre-trained BERT model is trained using the sample text dataset, and if the training result meets the preset conditions, the domain new word extraction model is obtained.
[0016] According to a method for extracting new domain words provided by the present invention, the method of constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data includes:
[0017] Taking each of the candidate domain new words and other target words in the target text data as nodes, and judging whether there is a co-occurrence relationship between each of the nodes in the target text data according to the context information of the target text data;
[0018] If it is determined that the co-occurrence relationship exists, the co-occurrence relationship is used as the edge between the nodes, and the weight of each edge is determined according to the co-occurrence frequency of the nodes at both ends of each edge in the target text data, so as to construct the domain new word graph model.
[0019] According to a method for extracting new domain words provided by the present invention, determining a target domain new word from a plurality of candidate domain new words according to the domain new word graph model comprises:
[0020] Based on the PageRank algorithm, according to the position of each node in the domain new word graph model and the connection relationship between each node, the importance of each node in the domain new word graph model is evaluated to obtain the importance score of each node in the domain new word graph model;
[0021] The candidate domain new words whose importance scores in the domain new word graph model meet a preset new word importance threshold are determined as the target domain new words.
[0022] According to a method for extracting new words in a field provided by the present invention, the step of obtaining target text data to be subjected to new word extraction in a field comprises:
[0023] In a preset text data collection period, a plurality of target text data are obtained, wherein the text contents of each target text data are different, so as to construct a corresponding domain new word graph model through each target text data;
[0024] After evaluating the importance of each node in the domain new word graph model based on the PageRank algorithm according to the position of each node in the domain new word graph model and the connection relationship between each node to obtain the importance score of each node in the domain new word graph model, the method further includes:
[0025] Obtaining the domain new word graph model corresponding to each of the target text data;
[0026] Through graph alignment technology and graph embedding technology, all the domain new word graph models are fused to obtain a domain new word fusion graph;
[0027] Obtaining word association relationship pairs corresponding to each of the candidate new domain words in the new domain word fusion graph, wherein the word association relationship pairs are formed by the candidate new domain words having the edges in the new domain word fusion graph;
[0028] According to the importance score of each of the nodes in the domain new word fusion graph, obtaining the word association score of each of the word association relationship pairs;
[0029] Determine the word association relationship pairs whose word association scores meet a preset word association threshold as target word association relationship pairs, and generate a corresponding word cloud according to the target word association relationship pairs;
[0030] After determining the target domain new word from the plurality of candidate domain new words according to the domain new word graph model, the method further includes:
[0031] The target domain new words and the word cloud are stored in a domain new word library.
[0032] According to a method for extracting new domain words provided by the present invention, determining candidate new domain words from a plurality of target words according to the new word probability of each target word comprises:
[0033] Based on the order of the probability of the new words from large to small, all the target words are sorted, and according to the sorting result, the first N target words are selected as the pending words;
[0034] Obtain the number of occurrences of each of the pending words in the target text data, and determine whether the number of occurrences is greater than a preset frequency threshold, if so, determine the pending words as the candidate domain new words, wherein the preset frequency threshold is generated according to the number of occurrences of all domain new words in the domain new word library in all historical text data; the historical text data is the text data corresponding to the newly added domain new words obtained when the domain new word library is newly added at a historical moment;
[0035] After storing the target domain new words and the word cloud in a domain new word library, the method further includes:
[0036] storing the target text data corresponding to the target domain new words in the domain new word library to obtain an updated domain new word library;
[0037] Based on the updated new domain vocabulary, the preset frequency threshold is updated to obtain an updated preset frequency threshold.
[0038] According to a method for extracting new domain words provided by the present invention, after obtaining the number of occurrences of each of the pending words in the target text data and judging whether the number of occurrences is greater than a preset frequency threshold, if so, determining the pending words as the candidate domain new words, the method further comprises:
[0039] The obtained plurality of candidate domain new words are constructed into a candidate domain new word set, and the candidate domain new word set is screened, wherein the screening process includes:
[0040] Determine whether any candidate new word in the candidate new word set belongs to a substring of other candidate new words in the field; if so, delete the candidate new word belonging to the substring from the candidate new word set to obtain a filtered candidate new word set;
[0041] The step of constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data includes:
[0042] A domain new word graph model is constructed according to the co-occurrence relationship of each of the candidate domain new words in the candidate domain new word set after the screening process in the target text data.
[0043] The present invention also provides a device for extracting new words in a field, comprising:
[0044] A text data input module is used to obtain target text data to be extracted from new words in the field;
[0045] A prediction module is used to input the target text data into a domain new word extraction model to obtain the probability of each target word in the target text data output by the domain new word extraction model being a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels;
[0046] A candidate new word screening module, used to determine candidate field new words from a plurality of the target words according to the new word probability of each of the target words;
[0047] A graph model building module, used to build a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data;
[0048] The graph model processing module is used to determine a target domain new word from a plurality of candidate domain new words according to the domain new word graph model.
[0049] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned methods for extracting new words from a field is implemented.
[0050] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the method for extracting new words in a field as described in any one of the above is implemented.
[0051] The method, device, electronic device and storage medium for extracting new domain words provided by the present invention predict the probability of new words in words in text data through a new domain word extraction model obtained by training a natural language processing model, and then select candidate new domain words according to the predicted new word probabilities to construct a new domain word graph model. Finally, the new domain words in the text data are obtained through the new domain word graph model, so as to more accurately extract the new domain words in the text. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0053] Figure 1 A schematic diagram of the process of extracting new domain words provided by the present invention;
[0054] Figure 2 A schematic diagram of the process of constructing the relationship of new domain words based on the graph algorithm provided by the present invention;
[0055] Figure 3 A schematic diagram of the process of extracting new words in candidate fields based on the deep learning model provided by the present invention;
[0056] Figure 4 A schematic diagram of the structure of the device for extracting new words in a field provided by the present invention;
[0057] Figure 5 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION
[0058] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0059] In a specific industry or field, it is necessary to accurately identify and extract new words in the field (such as professional terms, industry-specific vocabulary, emerging words, and non-standardized colloquial expressions). These new words in the field can help professionals obtain the latest information in the field in a timely manner, gain an in-depth understanding of industry development trends, event dynamics, and community public opinion, thereby providing accurate materials and data support for related research.
[0060] In the current method of extracting new words in a specific field, experts or practitioners can read and analyze texts in a specific field to identify and annotate new words in it. However, this method is time-consuming and labor-intensive, relies on manual experience and professional knowledge, is easily affected by subjective factors, and cannot process large-scale text data. Alternatively, new words in a field can be extracted through word frequency statistics. This method analyzes the frequency of occurrence of words in the text and the background knowledge of a specific field, and determines words with high frequency but less occurrence in the general corpus as new words in the field. However, this method may ignore some context-related features and cannot accurately determine whether a word is a new word. It also has difficulties in distinguishing polysemous words.
[0061] Based on the problems existing in the existing methods for extracting new domain words, the present invention analyzes the text through a deep learning model to identify and extract candidate new domain words in the text; then, a graph algorithm is used to screen and evaluate the word relationships between these candidate new domain words, thereby achieving accurate extraction of new domain words and word relationships, and at the same time also improving the efficiency of new word extraction.
[0062] Figure 1 A schematic diagram of the process of extracting new words in a field provided by the present invention is shown in FIG. Figure 1 As shown, the present invention provides a method for extracting new words in a field, comprising:
[0063] Step 101: Obtain target text data for domain new word extraction.
[0064] In the present invention, target text data can be obtained through various different domain platforms, so that relevant domain new words in these target text data can be extracted through the subsequent domain new word extraction process. In one embodiment, the target text data can be obtained by the following methods: 1. User generated content (UGC), such as text content published by users on platforms such as social media, forums and blogs, which usually reflects the user's personal views, emotions and discussion topics, and may contain some new words in the field; 2. Internal documents of the enterprise, including internal reports, meeting minutes, project documents and product documents, etc., which may contain enterprise-specific terms, abbreviations and emerging words in the industry; 3. Community discussion content, including discussions and interactions between users in online communities, groups or forums, and these discussion texts may involve topics and related terms in specific fields; 4. Industry report content, including industry reports, research reports and market analysis reports published by professional institutions, consulting companies and research institutions, etc., which usually contain in-depth research and analysis of specific fields, and may involve emerging words and industry terms. In one embodiment, the target text data can be preprocessed in advance, such as word segmentation, sentence segmentation, removal of stop words and meaningless symbols, etc., to retain the target words that need to be predicted for new domain words later, thereby converting the original target text data into input data that is more suitable for deep learning model processing, that is, obtaining the target text data after text preprocessing.
[0065] Step 102: input the target text data into a domain new word extraction model to obtain the probability of each target word in the target text data output by the domain new word extraction model being a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels.
[0066] In the present invention, before extracting new words in the field, it is necessary to train the natural language processing model in the early stage to obtain the new word extraction model in the field. Specifically, collect and organize sample text data marked with new word labels in the field. These sample text data can come from the annotations of experts or professionals in the field, and the new word labels in the field can be binary labels, that is, indicating whether the word is a new word in the field. Further, the natural language processing model is trained using the prepared sample text data and features, and after the model is trained, a part of the test set marked with new word labels in the field is used to evaluate the trained model. The evaluation indicators can include accuracy, recall rate and F1 value, etc., so as to tune and optimize the model according to the evaluation results, such as adjusting model parameters and increasing training samples.
[0067] After the model is determined to be trained, the probability of each target word being a new word can be obtained by inputting the target text data into the trained domain new word extraction model, that is, the probability of the word at each position in the target text data being predicted as a domain new word, and this probability indicates how likely the model thinks that the word at this position is a new word. In one embodiment, words with a higher probability of being a new word can be screened out according to a set threshold.
[0068] Step 103: determining candidate domain new words from the plurality of target words according to the new word probability of each target word.
[0069] In the present invention, a probability threshold can be determined, and the threshold is used to filter out words with a higher probability of new words from the target text data as candidate domain new words. The selection of the threshold can be adjusted according to actual needs and domain characteristics. In one embodiment, all target words can also be sorted from high to low according to the probability of new words according to the probability of new words of each target word; then, the sorted target word list is traversed one by one, and the probability of new words of each word is compared with the set threshold in turn. If the probability of new words of a certain word is higher than the threshold, and the length of the word is also within the preset new word length (such as setting the maximum new word length), it will be selected as a candidate domain new word.
[0070] Step 104: construct a domain new word graph model according to the co-occurrence relationship of each of the candidate domain new words in the target text data.
[0071] In the present invention, through the previous steps, a group of candidate domain new words are obtained, and these words will be used as nodes to form a new word graph model. At the same time, in the target text data, the co-occurrence relationship between the candidate domain new words is counted. In the present invention, the co-occurrence relationship can be calculated based on the frequency of occurrence in the context window of the word, reflecting the degree of connection between the candidate domain new words.
[0072] Furthermore, based on the co-occurrence relationship, the new words in the candidate field and other words in the target text are taken as nodes, and the co-occurrence relationship is taken as the edge to construct a graph model consisting of the new words and all the words in the text. This model describes in detail the co-occurrence relationship between the words in the target text data, which will be used for the analysis of new words and their related words based on graph algorithms in subsequent steps.
[0073] Step 105 : determining a target domain new word from a plurality of candidate domain new words according to the domain new word graph model.
[0074] In the present invention, according to the preset requirements, an appropriate graph algorithm is selected to evaluate and analyze the domain new word graph model. Furthermore, the graph algorithm calculates the importance score of each node according to the position of the node in the graph and the connection relationship with other nodes. For nodes belonging to candidate domain new words, if they are connected to more other nodes or to nodes with higher importance, they usually get higher scores. Then, several nodes with higher rankings and belonging to candidate domain new words are selected as target domain new words. These words are considered to have higher importance and representativeness in the field. At the same time, the selected target domain new words have higher accuracy and practicality.
[0075] The method for extracting new domain words provided by the present invention predicts the probability of new words for words in text data through a new domain word extraction model obtained by training a natural language processing model, and then selects candidate new domain words according to the predicted new word probabilities to construct a new domain word graph model, and finally obtains the new domain words in the text data through the new domain word graph model, thereby more accurately extracting the new domain words in the text.
[0076] Based on the above embodiment, the domain new word extraction model is trained by the following steps:
[0077] Sample text data corresponding to various industry fields are obtained, and the sample text data are subjected to text preprocessing to obtain preprocessed sample text data, wherein the text preprocessing at least includes sentence segmentation processing, word segmentation processing, stop word removal processing, and punctuation removal processing.
[0078] In the present invention, after obtaining the sample text data of multiple industry fields, it is necessary to clean and organize the original sample text data, including sentence splitting, word segmentation, stop word removal, and removal of meaningless symbols. Specifically, through sentence splitting, the long sample text data is decomposed into independent sentences. In one embodiment, the SpaCy natural language processing library is used to split the text into sentences according to punctuation marks and other language rules; then, through word segmentation, each sentence is further decomposed into individual words or tokens (this process is also called tokenization). In this embodiment, the Jieba library can be used to complete the word segmentation function; further, stop word removal is performed on the sample text data after sentence splitting and word segmentation. Among them, stop words refer to words that frequently appear in the text but do not contain much information, such as "de", "shi", and "he", etc. In this embodiment, these stop words in the sample text data are removed through a predefined stop word list; finally, some meaningless symbols in the sample file data are removed, such as punctuation marks, numbers, and other non-alphabetic characters. These symbols usually do not help much in the extraction of new words. The above text preprocessing operations are all to convert the original sample text data into input data suitable for processing by a deep learning model. It should be noted that in the actual scenario of new word extraction in a specific field, the above text preprocessing process can also be performed on the obtained original text data to make the text data input into the new word extraction model in the specific field more suitable for model input.
[0079] Further, through the new word tags in the specific field, the sample new words in each of the sample text data after the above text preprocessing are marked, and the sample text data sets of various industry fields are constructed.
[0080] In the present invention, the words in the sample text data after text preprocessing are marked, and the marking objects are the words that are not common in the general corpus but are common in a specific field. By analyzing the sample text data, these sample new words can be identified (it can be understood that these sample new words are historical new words in the specific field and have been determined as new words in the sample text data at a historical moment), and they are marked using the new word tags in the specific field. When constructing the sample text data set, it can be clearly known which words are new words in the specific field, so as to better process and analyze this data.
[0081] Through the sample text data set, the pre-trained BERT model is trained. If the training result meets the preset conditions, the new word extraction model in the specific field is obtained.
[0082] In the present invention, firstly, the BERT (Bidirectional Encoder Representations from Transformers) model version used for new word extraction can be determined according to actual needs (such as based on data volume, computing resources and task requirements), including BERT-Base, BERT-Large and BERT-Mini, etc. The model parameters and structure of each version are slightly different. The present invention adopts BERT-Base, which has a parameter number of 110 million and is suitable for document data sets of hundreds of thousands to millions of orders of magnitude, thereby meeting the task requirements of domain new word extraction. Preferably, in one embodiment, a BERT model pre-trained on a large-scale corpus can be used for initialization, thereby utilizing the language knowledge learned by the pre-trained model to accelerate model training and improve model performance.
[0083] Furthermore, during the training process, before the preprocessed sample text data is input into the BERT model, it is converted into an embedding vector through three embedding layers, namely Token Embeddings, Segment Embeddings, and Position Embeddings, and then the converted embedding vector is input into the BERT model, where Token Embeddings converts each word into a corresponding word vector; Segment Embeddings is used to distinguish different sentences; Position Embeddings is used to indicate the position of a word in a sentence.
[0084] Then, the BERT model learns context information through training data and predicts where new words may appear. The goal of model training is to minimize the gap between model prediction and actual results. In the present invention, gradient descent can be used in the training process. The recommended values of the gradient descent method and key indicators are as follows: learning rate (recommended value is 0.001), batch size (recommended value is 64), number of iterations (recommended value is 100, with early stopping strategy), optimizer (Adam is recommended), and loss function (mean square error is recommended).
[0085] Furthermore, in order to prevent the model from overfitting, the present invention uses dropout and weightdecay regularization strategies, as well as a learning rate decay strategy, during the training process, and performs rolling fine-tuning on the trained BERT model in combination with the historical new word library, so that the model can better recognize and extract new words, and update the new word library more effectively. At the same time, in the present invention, the performance of the model is tested by regularly using the validation set during the training process to monitor the progress of the model training, and the parameters of the model are adjusted and optimized by the performance of the model on the validation set.
[0086] The present invention uses the BERT model to construct a domain new word extraction model, thereby dynamically extracting new words from large-scale unlabeled text data. Compared with the traditional new word extraction method based on word frequency and information entropy, the BERT model can better understand the contextual information of words, thereby extracting new words more accurately; at the same time, the historical new word thesaurus is used to perform rolling fine-tuning on the BERT model, so that the model can better recognize and extract new words, and update the new word thesaurus more effectively.
[0087] On the basis of the above embodiment, the method of constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data includes:
[0088] Taking each of the candidate domain new words and other target words in the target text data as nodes, and judging whether there is a co-occurrence relationship between each of the nodes in the target text data according to the context information of the target text data;
[0089] If it is determined that the co-occurrence relationship exists, the co-occurrence relationship is used as the edge between the nodes, and the weight of each edge is determined according to the co-occurrence frequency of the nodes at both ends of each edge in the target text data, so as to construct the domain new word graph model.
[0090] In the present invention, all candidate domain new words and other words (i.e., non-candidate domain new words) in the target text data are defined as nodes in the domain new word graph model to be constructed, and each node represents a unique word. Then, the connection relationship between the nodes is defined, that is, the construction of the edge. In the present invention, the co-occurrence relationship of words (including candidate domain new words and non-candidate domain new words) in the target text data is used as the basis of the edge, that is, if two words appear in the same context (such as the same sentence, the same paragraph or the same article), an edge is established between the two nodes, thereby obtaining the word co-occurrence information in the target text data, reflecting the correlation between the words.
[0091] Furthermore, each edge needs to be assigned a weight, and the size of the weight usually depends on the co-occurrence frequency of the two words in the target text data, that is, the higher the co-occurrence frequency, the greater the weight, indicating that the two words are more related. In one embodiment, the term frequency-inverse document frequency index (Term Frequency-Inverse Document Frequency, referred to as TF-IDF) can also be used as a weight to measure the importance of a word in a document, thereby reducing the weight of some common words and increasing the weight of some rare but important words.
[0092] Finally, based on the nodes and edges defined in the above steps, a graph model consisting of candidate domain new words and all other words in the target text data is constructed for the target text data, namely, the domain new word graph model. This model describes the co-occurrence relationship between words in detail for subsequent analysis of domain new words and their related words based on graph algorithms.
[0093] On the basis of the above embodiment, the step of determining a target domain new word from a plurality of candidate domain new words according to the domain new word graph model includes:
[0094] Based on the PageRank algorithm, according to the position of each node in the domain new word graph model and the connection relationship between each node, the importance of each node in the domain new word graph model is evaluated to obtain the importance score of each node in the domain new word graph model;
[0095] The candidate domain new words whose importance scores in the domain new word graph model meet a preset new word importance threshold are determined as the target domain new words.
[0096] In the present invention, each node in the domain new word graph model corresponding to the target text data can be evaluated by a graph algorithm, and the graph algorithm can select a page ranking (PageRank) algorithm or a hyperlink-based topic search (HyperlinkInduced Topic Search, referred to as HITS). Preferably, in the present invention, the PageRank algorithm can be used to more effectively evaluate the importance of the node. Compared with the HITS algorithm, the PageRank algorithm is more suitable for the evaluation process of the candidate domain new word graph model.
[0097] Furthermore, each node in the candidate domain new word graph model is evaluated by the PageRank algorithm, and the candidate domain new word graph model calculates the importance score of each node based on the node's position in the candidate domain new word graph model and its connection relationship with other nodes. In this process, if a node is connected to more other nodes or to nodes with higher importance, the node will usually get a higher score.
[0098] In one embodiment, a preset new word importance threshold is set, and only words whose importance scores exceed this threshold are considered new words, wherein the setting of the preset new word importance threshold usually needs to be determined according to the needs of the actual application and the characteristics of the data. For example, if you want to extract more domain new words, you can set a lower threshold; if you are more concerned about the accuracy of new words, you can set a higher threshold. In one embodiment, the 75th percentile of the importance scores of all candidate domain new words is used as the preset new word importance threshold.
[0099] Based on the above embodiment, the step of obtaining target text data to be subjected to domain new word extraction includes:
[0100] In a preset text data collection period, a plurality of target text data are obtained, wherein the text contents of each target text data are different, so as to construct a corresponding domain new word graph model through each target text data;
[0101] After evaluating the importance of each node in the domain new word graph model based on the PageRank algorithm according to the position of each node in the domain new word graph model and the connection relationship between each node to obtain the importance score of each node in the domain new word graph model, the method further includes:
[0102] Obtaining the domain new word graph model corresponding to each of the target text data;
[0103] Through graph alignment technology and graph embedding technology, all the domain new word graph models are fused to obtain a domain new word fusion graph;
[0104] Obtaining word association relationship pairs corresponding to each of the candidate new domain words in the new domain word fusion graph, wherein the word association relationship pairs are formed by the candidate new domain words having the edges in the new domain word fusion graph;
[0105] According to the importance score of each of the nodes in the domain new word fusion graph, obtaining the word association score of each of the word association relationship pairs;
[0106] Determine the word association relationship pairs whose word association scores meet a preset word association threshold as target word association relationship pairs, and generate a corresponding word cloud according to the target word association relationship pairs;
[0107] After determining the target domain new word from the plurality of candidate domain new words according to the domain new word graph model, the method further includes:
[0108] The target domain new words and the word cloud are stored in a domain new word library.
[0109] In the present invention, if multiple target text data are obtained within a preset text data collection period (these target text data can be selected according to actual needs, such as selecting text data with closer fields as much as possible), the above steps can be used to obtain the field new word graph model corresponding to each target text data.
[0110] Furthermore, after completing the importance analysis of the domain new word graph model corresponding to each target text data, we use graph alignment and graph embedding techniques to perform node alignment, edge fusion, and point-edge addition operations on each domain new word graph model, thereby integrating all domain new word graph models into a unified graph model, namely, the domain new word fusion graph.
[0111] In the present invention, based on graph alignment technology and graph embedding technology, similar nodes in new word graph models in various fields are matched and aligned, so as to match nodes with similar semantics in new word graph models in different fields and find the corresponding relationship between them; then, the edges in the new word graph models in various fields are merged, wherein the edge fusion can be performed based on the edge weight, co-occurrence frequency or other metrics to retain the edge information in various fields and form a more comprehensive relationship network in the integrated graph model; finally, the integrated graph model can be added with new point edges as needed, such as new candidate domain new words or new co-occurrence relationships predicted based on new target text data, so that the information of the graph can be enriched by adding additional nodes or edges. Through the above steps, the present invention can integrate the new word graph models in various fields and fuse them into a unified domain new word fusion graph, and the integrated domain new word fusion graph will provide more comprehensive and rich semantic information and knowledge support.
[0112] Furthermore, each edge in the domain new word fusion graph is traversed to determine whether the two nodes at both ends of the edge contain the candidate domain new word. If so, the two nodes are used as a pair of words in the word association relationship pair to form a relationship pair. This step is repeated until all edges are traversed, thereby obtaining the word association relationship pairs corresponding to each candidate domain new word in the domain new word fusion graph. These relationship pairs are formed by the candidate domain new words with edges in the fusion graph, and can be used to further analyze and process the association relationship between words.
[0113] Further, the word association score is calculated based on the importance score of each node obtained in the above embodiment. In one embodiment, the importance scores of the two nodes connected to the word association relationship pair can be weighted averaged to obtain the word association score of the word association relationship pair, and these scores can be used to quantify the degree of correlation between the word association relationship pairs.
[0114] After obtaining the word association scores of each of the word association pairs in the domain new word fusion diagram, the word association pairs that meet the conditions are screened out as target word association pairs according to the preset word association threshold, that is, the word association pairs whose word association scores are higher than or equal to the preset threshold are selected. Only the word association pairs whose scores exceed this threshold will be considered to have significantly associated words. Specifically, the setting of this preset word association threshold also depends on the needs of the actual application and the characteristics of the data. For example, if you want to extract stronger word relationships, you can set a higher threshold; if you pay more attention to the breadth of word relationships, you can set a lower threshold. In one embodiment, the Louvain community detection algorithm can be used to find the median of the community value as the preset word association threshold.
[0115] In the present invention, a corresponding word cloud can be generated through the target word association relationship pair. The word cloud is a visualization tool used to display the frequency or importance of words in text data. When generating a word cloud, the words in the target word association relationship pair can be used as input, and the size, color and other attributes of the words can be determined according to the word association score. Finally, the generated word cloud is visualized, which can be a static image or an interactive word cloud, which can be scaled, rotated and other operations according to the user's operation, so as to form a correlation word cloud in which multiple new words are connected to related words based on the importance score of each node.
[0116] Figure 2 This is a schematic diagram of the process of constructing the relationship between new domain words based on the graph algorithm provided by the present invention, which can be referred to Figure 2 As shown, the present invention applies graph algorithms to the process of extracting new words in the field. By constructing a co-occurrence network of new words in the field and applying graph algorithms, and using graph alignment and graph embedding techniques, it is possible to more comprehensively and accurately evaluate and determine new words in the field, and can flexibly adjust the number and accuracy of new word extraction to meet different application requirements.
[0117] Finally, the domain new words and word cloud results after threshold exclusion are manually reviewed and screened. In this way, these new words not only appear frequently in the text, but also have a close co-occurrence relationship with other domain new words, so they can be considered as representative domain new words. Furthermore, the domain new words and word correlation relationships (i.e., word clouds) obtained are recorded in the domain new word library. This domain new word library will serve as an important resource for natural language processing tasks, providing support for subsequent new word extraction, lexical analysis, semantic retrieval and other tasks; at the same time, the storage of domain new words also means that domain knowledge can be continuously updated and improved, and the latest developments and development trends in the field can be tracked and mastered in real time.
[0118] On the basis of the above embodiment, the step of determining candidate domain new words from a plurality of target words according to the new word probability of each target word includes:
[0119] Based on the order of the probability of the new words from large to small, all the target words are sorted, and according to the sorting result, the first N target words are selected as the pending words;
[0120] Obtain the number of occurrences of each of the pending words in the target text data, and determine whether the number of occurrences is greater than a preset frequency threshold, if so, determine the pending words as the candidate domain new words, wherein the preset frequency threshold is generated according to the number of occurrences of all domain new words in the domain new word library in all historical text data; the historical text data is the text data corresponding to the newly added domain new words obtained when the domain new word library is newly added at a historical moment;
[0121] In the present invention, the preprocessed target text data is input into the domain new word extraction model, and the domain new word extraction model predicts the probability of the word being a new word for each position in the target text; then, for each position predicted by the model, the corresponding vocabulary is extracted, sorted from large to small according to the probability of new words, and the top N target words are selected as pending words. In one embodiment, it is also necessary to limit the length of the pending new words to avoid extracting too long domain new words. For example, the length of the longest word in the current domain new word vocabulary plus one is taken as the upper limit of the length limit, and the lower limit is set to 2, usually ranging from 2 to 7, and the specific length can be adjusted according to actual needs.
[0122] Furthermore, the number of occurrences of each pending new word in the target text data is counted, and a frequency threshold, i.e., a preset frequency threshold, is set. Only the pending new words whose frequency exceeds this threshold will be retained as candidate domain new words. In one embodiment, the setting of the preset frequency threshold can be based on half of the average number of occurrences of all words in the historical new word vocabulary in the historical text data in which they are located, as the preset frequency threshold.
[0123] After storing the target domain new words and the word cloud in a domain new word library, the method further includes:
[0124] storing the target text data corresponding to the target domain new words in the domain new word library to obtain an updated domain new word library;
[0125] Based on the updated new domain vocabulary, the preset frequency threshold is updated to obtain an updated preset frequency threshold.
[0126] In the present invention, the preset frequency threshold changes dynamically with the update of the domain new word database. A real domain new word usually appears multiple times in the text, rather than appearing once or twice occasionally. By setting a dynamic frequency threshold, the present invention can filter out a large number of candidate domain new words that appear accidentally and meaninglessly, thereby improving the accuracy of extracting domain new words.
[0127] On the basis of the above embodiment, after obtaining the number of occurrences of each of the pending words in the target text data and determining whether the number of occurrences is greater than a preset frequency threshold, and if so, determining the pending words as the candidate field new words, the method further includes:
[0128] The obtained plurality of candidate domain new words are constructed into a candidate domain new word set, and the candidate domain new word set is screened, wherein the screening process includes:
[0129] Determine whether any candidate new word in the candidate new word set belongs to a substring of other candidate new words in the field; if so, delete the candidate new word belonging to the substring from the candidate new word set to obtain a filtered candidate new word set;
[0130] The step of constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data includes:
[0131] A domain new word graph model is constructed according to the co-occurrence relationship of each of the candidate domain new words in the candidate domain new word set after the screening process in the target text data.
[0132] In the present invention, among the candidate domain new words, there may be some candidate domain new words that are substrings of another candidate domain new word. For example, if “deep learning” and “learning” are both candidate domain new words, then “learning” is a substring of “deep learning.” In this case, it is necessary to retain the longer candidate domain new word, namely “deep learning,” and filter out the shorter candidate domain new word, namely “learning.”
[0133] In one embodiment, for candidate domain new words that have appeared in the current domain new word library, these candidate domain new words can be marked. Although these words have been stored, new texts may provide new word relationship information, which is still valuable for the subsequent domain new word extraction process. Through the above screening and processing operations, the quality of candidate domain new words can be further improved, and preparations can be made for the subsequent new word relationship construction and storage.
[0134] Figure 3 The schematic diagram of the extraction process of candidate domain new words based on the deep learning model provided by the present invention can be referred to Figure 3As shown, the present invention uses the BERT deep learning model to better understand the contextual information of words, thereby more accurately extracting new domain words. In one embodiment, by conducting an experimental evaluation of the domain new word extraction method provided by the present invention and the existing domain new word extraction method, the evaluation results show that the accuracy of the present method in the new word extraction task is about 9.5% higher than that of the existing word frequency statistics method. The calculation method is: (accuracy of this method 92% - accuracy of word frequency statistics method 84%) / 84% * 100%.
[0135] In the model training strategy, the present invention uses the historical new word database to perform rolling fine-tuning on the deep learning model, which can effectively update the new word database and continuously improve the model's new word extraction ability over time. In the experiment, the rolling fine-tuning strategy improved the model's new word extraction performance by about 16% compared to the fixed training strategy. The calculation method is: (rolling fine-tuning strategy prediction accuracy 87% - fixed training strategy prediction accuracy 75%) / 75% * 100%.
[0136] Moreover, the present invention can effectively filter out accidental and meaningless new words in the candidate domain by setting a dynamic frequency threshold to filter new words in the candidate domain. In the experiment, the dynamic frequency filtering method reduced about 10% of invalid new words compared with the fixed threshold filtering method, and avoided about 5% of valid new words from being filtered.
[0137] In addition, in the subsequent analysis process of new words in candidate fields, the present invention innovatively introduces graph models and graph algorithms, which can not only be used to extract new words, but also analyze the relationship between new words and other words to form a word correlation cloud map, thereby further enriching the semantic information of new words.
[0138] The following is a description of the domain new word extraction device provided by the present invention. The domain new word extraction device described below and the domain new word extraction method described above can be referenced to each other.
[0139] Figure 4 The schematic diagram of the structure of the device for extracting new words in a field provided by the present invention is as follows: Figure 4As shown, the present invention provides a device for extracting new domain words, including a text data input module 401, a prediction module 402, a candidate new word screening module 403, a graph model construction module 404 and a graph model processing module 405, wherein the text data input module 401 is used to obtain target text data to be subjected to domain new word extraction; the prediction module 402 is used to input the target text data into the domain new word extraction model to obtain the new word probability of each target word in the target text data output by the domain new word extraction model as a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels; the candidate new word screening module 403 is used to determine candidate domain new words from multiple target words according to the new word probability of each target word; the graph model construction module 404 is used to construct a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data; the graph model processing module 405 is used to determine the target domain new word from multiple candidate domain new words according to the domain new word graph model.
[0140] The domain new word extraction device provided by the present invention predicts the new word probability of words in text data through the domain new word extraction model obtained by training the natural language processing model, and then selects candidate domain new words to construct a domain new word graph model based on the predicted new word probability. Finally, the domain new words in the text data are obtained through the domain new word graph model, so as to more accurately extract the domain new words in the text.
[0141] The device provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.
[0142] Figure 5 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (Processor) 501, a communication interface (Communications Interface) 502, a memory (Memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 complete mutual communication through the communication bus 504. The processor 501 can call the logic instructions in the memory 503 to execute the domain new word extraction method, which includes: obtaining target text data to be extracted with domain new words; inputting the target text data into the domain new word extraction model to obtain the new word probability of each target word in the target text data output by the domain new word extraction model as a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word tags; determining candidate domain new words from multiple target words according to the new word probability of each target word; constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data; and determining a target domain new word from multiple candidate domain new words according to the domain new word graph model.
[0143] In addition, the logic instructions in the above-mentioned memory 503 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.
[0144] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the domain new word extraction method provided by the above-mentioned methods, and the method includes: obtaining target text data to be subjected to domain new word extraction; inputting the target text data into a domain new word extraction model to obtain the new word probability of each target word in the target text data output by the domain new word extraction model as a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels; determining candidate domain new words from multiple target words according to the new word probability of each target word; constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data; and determining a target domain new word from multiple candidate domain new words according to the domain new word graph model.
[0145] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the domain new word extraction method provided by the above-mentioned embodiments, the method comprising: obtaining target text data for which domain new word extraction is to be performed; inputting the target text data into a domain new word extraction model to obtain the new word probability of each target word in the target text data output by the domain new word extraction model as a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels; determining candidate domain new words from multiple target words according to the new word probability of each target word; constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data; and determining a target domain new word from multiple candidate domain new words according to the domain new word graph model.
[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for extracting new words in a field. It is characterized in that include: Obtain target text data for field new word extraction; Input the target text data into a domain new word extraction model to obtain the new word probability of each target word in the target text data output by the domain new word extraction model as a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels; Determining candidate domain new words from a plurality of the target words according to the new word probability of each of the target words; Constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data; According to the domain new word graph model, a target domain new word is determined from a plurality of candidate domain new words.
2. The method for extracting new domain words according to claim 1, It is characterized in that The domain new word extraction model is trained by the following steps: Acquire sample text data corresponding to various industry fields, and perform text preprocessing on the sample text data to obtain preprocessed sample text data, wherein the text preprocessing at least includes sentence segmentation processing, word segmentation processing, stop word removal processing, and punctuation removal processing; By using domain new word labels, sample domain new words in each of the sample text data after the text preprocessing are marked, and sample text data sets of various industry fields are constructed; The pre-trained BERT model is trained using the sample text dataset, and if the training result meets the preset conditions, the domain new word extraction model is obtained.
3. The method for extracting new domain words according to claim 1, It is characterized in that The step of constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data includes: Taking each of the candidate domain new words and other target words in the target text data as nodes, and judging whether there is a co-occurrence relationship between each of the nodes in the target text data according to the context information of the target text data; If it is determined that the co-occurrence relationship exists, the co-occurrence relationship is used as the edge between the nodes, and the weight of each edge is determined according to the co-occurrence frequency of the nodes at both ends of each edge in the target text data, so as to construct the domain new word graph model.
4. The method for extracting new domain words according to claim 3, It is characterized in that Determining a target domain new word from a plurality of candidate domain new words according to the domain new word graph model includes: Based on the PageRank algorithm, according to the position of each node in the domain new word graph model and the connection relationship between each node, the importance of each node in the domain new word graph model is evaluated to obtain the importance score of each node in the domain new word graph model; The candidate domain new words whose importance scores in the domain new word graph model meet a preset new word importance threshold are determined as the target domain new words.
5. The method for extracting new domain words according to claim 4, It is characterized in that The step of obtaining target text data to be subjected to domain new word extraction includes: In a preset text data collection period, a plurality of target text data are obtained, wherein the text contents of each target text data are different, so as to construct a corresponding domain new word graph model through each target text data; After evaluating the importance of each node in the domain new word graph model based on the PageRank algorithm according to the position of each node in the domain new word graph model and the connection relationship between each node to obtain the importance score of each node in the domain new word graph model, the method further includes: Obtaining the domain new word graph model corresponding to each of the target text data; Through graph alignment technology and graph embedding technology, all the domain new word graph models are fused to obtain a domain new word fusion graph; Obtaining word association relationship pairs corresponding to each of the candidate new domain words in the new domain word fusion graph, wherein the word association relationship pairs are formed by the candidate new domain words having the edges in the new domain word fusion graph; According to the importance score of each of the nodes in the domain new word fusion graph, obtaining the word association score of each of the word association relationship pairs; Determine the word association relationship pairs whose word association scores meet a preset word association threshold as target word association relationship pairs, and generate a corresponding word cloud according to the target word association relationship pairs; After determining the target domain new word from the plurality of candidate domain new words according to the domain new word graph model, the method further includes: The target domain new words and the word cloud are stored in a domain new word library.
6. The method for extracting new domain words according to claim 5, It is characterized in that The step of determining candidate domain new words from a plurality of the target words according to the new word probabilities of the respective target words comprises: Based on the order of the probability of the new words from large to small, all the target words are sorted, and according to the sorting result, the first N target words are selected as the pending words; Obtain the number of occurrences of each of the pending words in the target text data, and determine whether the number of occurrences is greater than a preset frequency threshold, if so, determine the pending words as the candidate domain new words, wherein the preset frequency threshold is generated according to the number of occurrences of all domain new words in the domain new word library in all historical text data; the historical text data is the text data corresponding to the newly added domain new words obtained when the domain new word library is newly added at a historical moment; After storing the target domain new words and the word cloud in a domain new word library, the method further includes: storing the target text data corresponding to the target domain new words in the domain new word library to obtain an updated domain new word library; Based on the updated new domain vocabulary, the preset frequency threshold is updated to obtain an updated preset frequency threshold.
7. The method for extracting new domain words according to claim 6, It is characterized in that After obtaining the number of occurrences of each of the pending words in the target text data and determining whether the number of occurrences is greater than a preset frequency threshold, and if so, determining the pending words as the candidate field new words, the method further includes: The obtained plurality of candidate domain new words are constructed into a candidate domain new word set, and the candidate domain new word set is screened, wherein the screening process includes: Determine whether any candidate new word in the candidate new word set belongs to a substring of other candidate new words in the field; if so, delete the candidate new word belonging to the substring from the candidate new word set to obtain a filtered candidate new word set; The step of constructing a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data includes: A domain new word graph model is constructed according to the co-occurrence relationship of each of the candidate domain new words in the candidate domain new word set after the screening process in the target text data.
8. A device for extracting new words in a field, It is characterized in that include: A text data input module is used to obtain target text data to be extracted from new words in the field; A prediction module is used to input the target text data into a domain new word extraction model to obtain the probability of each target word in the target text data output by the domain new word extraction model being a domain new word, wherein the domain new word extraction model is obtained by training a natural language processing model with sample text data marked with domain new word labels; A candidate new word screening module, used to determine candidate field new words from a plurality of the target words according to the new word probability of each of the target words; A graph model building module, used to build a domain new word graph model according to the co-occurrence relationship of each candidate domain new word in the target text data; The graph model processing module is used to determine a target domain new word from a plurality of candidate domain new words according to the domain new word graph model.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the method for extracting new words in a field as claimed in any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the method for extracting new words in a field as claimed in any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Method and system for constructing multi-emotion dictionary for internet
CN103544246A
Microblog-based neologism emotional tendency judgment method
CN105138510A
Method and device for identifying new words based on information entropy, server and storage medium
CN110807322A
Method and system for extracting new words in field
CN114266256A
Method, system and device for recognizing field words in voice and medium
CN114743551A
