Method, system, terminal and medium for automatic discovery and evolution of professional terms and research fields

Abstract

The application provides a professional term and research field automatic discovery and evolution method, system, terminal and medium, can accurately and automatically discover new professional terms from academic literature in the same research field, saves a lot of manual labeling cost, can discover new research fields according to actual needs under various disciplines in an iterative progressive manner, and establishes a hierarchical relationship between disciplines and fields, has processing and analysis capability in a multilingual environment, can automatically discover professional terms and research fields for academic literature in multiple languages including Chinese, English and the like, can fully cover all disciplines and research fields under the condition that the number of academic literature is sufficient, covers rare professional and research fields, has dynamic updating and evolution functions, can timely adjust previously discovered professional terms and research fields, and quickly discovers new professional terms and research fields, and realizes the goal of real-time discipline tracking and talent assessment.

Description

Technical Field

[0001] This application relates to the field of computer natural language processing technology, and in particular to methods, systems, terminals and media for the automatic discovery and evolution of technical terms and research fields. Background Technology

[0002] Currently, with the rapid development of theories and technologies in various disciplines, new professional terms are emerging one after another. It is almost impossible to rely entirely on manual compilation of terminology lists and their hierarchical relationships. Existing terminology lists or databases are relatively outdated and cannot reflect the new achievements and dynamics of discipline development. Therefore, there is an urgent need for the automatic construction of discipline terminology databases and the automatic discovery technology of research sub-fields, and for timely evolution to track the dynamics and trends of discipline development.

[0003] From a linguistic perspective, languages ​​with rich morphology can often identify terms based on morphological and part-of-speech analysis. However, languages ​​lacking morphological variation (such as Chinese) often rely on statistical indicators like word co-occurrence to generate terminology lists, leading to lower accuracy in terminology identification. From a subject-specific perspective, while terminology databases in fields like medicine and biology are relatively well-developed, they still face the challenge of rapidly emerging and updating new specialized terms. Other disciplines lack systematic research in this area, severely impacting the accuracy and timeliness of tracking disciplinary development and evaluating scientific and technological talent.

[0004] Currently, the construction of unsupervised, fully automated subject-specific corpora still faces challenges such as low accuracy and excessively high corpus quality requirements. While supervised training models achieve higher accuracy, they require large amounts of high-quality labeled datasets for training. Preparing labeled datasets demands significant human and material resources and can lead to inconsistent labeling results due to varying standards and interpretations among different labelers. Supervised methods cannot cover all current subjects and subfields. Furthermore, many existing terminology extraction tools are designed for single languages, exhibiting poor scalability and difficulty in handling terminology variations. Summary of the Invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method, system, terminal and medium for automatic discovery and evolution of professional terms and research fields, which is used to solve the problems in the prior art such as the inability to update professional terms in a timely manner, low accuracy of constructing subject corpora, excessively high requirements for corpus quality, and large investment of manpower and resources in annotation.

[0006] To achieve the above and other related objectives, the first aspect of this application provides a method for automatic discovery and evolution of professional terms and research fields, comprising: preprocessing a set of academic documents to be processed; performing a primary classification on the preprocessed set of academic documents based on a rule classifier; the primary classification result includes academic documents that can be clearly classified and academic documents that cannot be clearly classified; performing a secondary classification on the academic documents that cannot be clearly classified based on a BERT classification model; constructing a subject category corpus based on the academic documents that can be clearly classified by the rule classifier and the academic documents obtained by the secondary classification by the BERT classification model; and classifying each subject category corpus. The strings of academic documents in the corpora of the aforementioned disciplines are calculated to obtain the terminology feature parameters corresponding to each string; the terminology feature parameters include at least: C_value, basic frequency value, combination frequency value, and singularity value; the terminology feature parameters corresponding to each string are calculated to obtain the terminology recognition threshold corresponding to each string; the terminology in the corpora of the aforementioned disciplines is identified and obtained based on the terminology recognition threshold corresponding to each string; the terminology in the corpora of the aforementioned disciplines is clustered by topic based on the LDA model to obtain topic words and generate new disciplines.

[0007] In some embodiments of the first aspect of this application, the preprocessing method includes cleaning and standardizing the collection of academic documents to be processed.

[0008] In some embodiments of the first aspect of this application, the C_value is calculated in the following ways: Where t is a string, |t| is the length of string t, TF(t) is the word frequency of string t, s is a long string of words containing string t, and TF(s) is the word frequency of long string s.

[0009] In some embodiments of the first aspect of this application, the fundamental frequency value is calculated as follows: Basic(t) = |t|logf(t) + αe t Where Basic is the basic frequency value, t is a string, f(t) is the term frequency of string t, et is the number of long strings containing string t, and α is a custom weight.

[0010] In some embodiments of the first aspect of this application, the calculation method of the combined frequency value includes: ComboBasic(t)=|t|logf(t)+αe t +βe ′ tWhere ComboBasic is the combined frequency value, t is a string, f(t) is the term frequency of string t, et is the number of long strings containing string t, α is a custom weight, e't is the number of other terms contained in string t, and β is a custom weight.

[0011] In some embodiments of the first aspect of this application, the singularity value is calculated in the following ways: Wherein, Weirdness is the singularity value, t is a string, TFtarget(t) is the word frequency of string t in a domain-specific corpus, TFreference(t) is the word frequency of string t in a general corpus, and N is normalization to eliminate the influence of the size of the domain-specific corpus and the general corpus.

[0012] In some embodiments of the first aspect of this application, the determination is made based on the technical term recognition threshold corresponding to each string to identify and obtain the technical terms in each corpus, specifically including: if the technical term recognition threshold corresponding to the string is greater than a preset threshold, then the string is a technical term of the corresponding corpus.

[0013] In some embodiments of the first aspect of this application, the computation process of the LDA model includes the following: determining the number of topic words as a hyperparameter of the LDA model, and randomly initializing the topic distribution and vocabulary distribution of each document and each topic; finding the optimal topic distribution and vocabulary distribution through iterative optimization, wherein in each iteration, the algorithm updates the topic distribution and vocabulary distribution of each document and each topic by inference through a Bayesian network based on the current topic and vocabulary distribution; gradually converging through multiple iterative sampling and updating processes until the topic distribution and vocabulary distribution reach a stable state; after the algorithm converges, each document will obtain a topic distribution and each topic will obtain a vocabulary distribution.

[0014] To achieve the above and other related objectives, a second aspect of this application provides an automatic discovery and evolution system for professional terms and research fields, comprising: an academic literature set preprocessing module for preprocessing the academic literature set to be processed; a subject category corpus acquisition module for performing a primary classification of the preprocessed academic literature set based on a rule classifier; the primary classification result includes academic literature that can be clearly classified and academic literature that cannot be clearly classified; a secondary classification of the academic literature that cannot be clearly classified based on a BERT classification model; and a subject category corpus composed of academic literature that can be clearly classified by the rule classifier and academic literature obtained by the secondary classification by the BERT classification model; and a professional term feature parameter acquisition module for... The strings of academic documents in each of the aforementioned subject category corpora are calculated to obtain the terminology feature parameters corresponding to each string; the terminology feature parameters include at least: C_value, basic frequency value, combination frequency value, and singularity value; a terminology recognition threshold acquisition module is used to calculate the terminology feature parameters corresponding to each string to obtain the terminology recognition threshold corresponding to each string; a terminology judgment and recognition module is used to judge and identify the terminology in each of the aforementioned subject category corpora according to the terminology recognition threshold corresponding to each string; a subject subcategory generation module is used to perform topic clustering on the terminology in each of the aforementioned subject category corpora based on the LDA model to obtain topic words and generate new subject categories.

[0015] To achieve the above and other related objectives, a third aspect of this application provides an electronic terminal, comprising: a processor and a memory; the memory for storing a computer program; and the processor for executing the computer program stored in the memory to enable the electronic terminal to execute the method for automatic discovery and evolution of technical terms and research fields.

[0016] To achieve the above and other related objectives, a fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for automatic discovery and evolution of technical terms and research fields.

[0017] As described above, the technical terms and research fields of this application for automatic discovery and evolution methods, systems, terminals, and media have the following beneficial effects:

[0018] (1) It can accurately and automatically discover new professional terms from academic literature in the same research field, saving a lot of manual annotation costs;

[0019] (2) It can use topic models to discover new research areas in an iterative manner according to actual needs under various disciplines and establish hierarchical relationships between disciplines and fields;

[0020] (3) Possesses the ability to process and analyze multilingual environments, and can automatically discover professional terms and research fields in academic literature in multiple languages, including Chinese and English;

[0021] (4) With a sufficient number of academic documents, it can cover all disciplines and research fields, including obscure majors and research areas;

[0022] (5) It has dynamic update and evolution functions, and can adjust the previously discovered professional terms and research fields in a timely manner, and quickly discover new professional terms and research fields, so as to achieve the goal of real-time discipline tracking and talent evaluation. Attached Figure Description

[0023] Figure 1 The diagram shown is a flowchart illustrating an automatic discovery and evolution method for technical terms and research fields according to an embodiment of this application.

[0024] Figure 2 The diagram shows a specific embodiment of the method for automatic discovery and evolution of technical terms and research fields in one aspect of this application.

[0025] Figure 3 The diagram shown is a structural schematic of an automatic discovery and evolution system for technical terms and research fields, as described in one embodiment of this application.

[0026] Figure 4 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of this application. Detailed Implementation

[0027] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0028] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of this application. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of this application. The following detailed description should not be considered limiting, and the scope of the embodiments of this application is defined only by the claims of the published patent. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0029] In this application, unless otherwise expressly specified and limited, the terms "installation," "connection," "linking," "fixing," and "holding" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application according to the specific circumstances.

[0030] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the invention.

[0032] Before providing a further detailed description of the present invention, the nouns and terms used in the embodiments of the present invention are explained, and the nouns and terms used in the embodiments of the present invention are subject to the following interpretations:

[0033] <1> Natural Language Processing (NLP) is an important branch of computer science and artificial intelligence that studies the theories and methods for enabling effective communication between humans and computers using natural language. NLP is not simply the study of natural language itself, but rather the development of computer systems, especially the software systems, that can effectively achieve natural language communication.

[0034] <2> Deep learning: Deep learning is a type of machine learning that uses multi-layered neural networks to capture high-level semantic features from input, and then performs tasks such as classification, regression, and generation. It is often conducted in a supervised learning format, using labeled datasets for training. Deep learning methods can directly process high-dimensional, complex raw input data, and compared to traditional methods, rely less on manual feature engineering to extract features from the input data.

[0035] <3> BERT, short for Bidirectional Encoder Representations from Transformers, is a deep learning model built upon bidirectional Transformers, specifically designed for masked language models and next sentence prediction tasks. After pre-training, BERT models can be fine-tuned by simply adding an output layer when dealing with specific natural language processing problems, without requiring significant changes to the underlying model. BERT models have convenient applications in a wide range of tasks, such as sentiment classification, question answering, and language reasoning.

[0036] <4> Latent Dirichlet Allocation (LDA) is a topic model that effectively represents document content, offering a highly compressed representation of the document's content. It assigns the topic of each document in a document set according to a probability distribution. LDA is based on the bag-of-words model, which assumes a document is a collection of words without any inherent order or sequence. A document can contain multiple topics, and each word in the document is generated from one of these topics. It is also an unsupervised learning algorithm, requiring only a document set and a specified number of topics for training, without the need for manually labeled training sets. Another advantage of LDA is that it can identify words to describe each topic. LDA is currently used in text mining, including text topic recognition, text classification, and text similarity calculation. The LDA computation steps are as follows: Initialize the document-topic distribution matrix and the topic-word distribution matrix; iteratively update the document-topic distribution matrix and the topic-word distribution matrix using Gibbs sampling or variational inference until convergence.

[0037] <5> Topic Modeling: Topic modeling is a commonly used model in natural language processing (NLP) to automatically extract topic information from large amounts of documents. The core idea of ​​topic modeling is that each document can be viewed as a mixture of multiple topics, with each topic consisting of a set of words. Topic modeling is an unsupervised learning method that can automatically extract topic information from large amounts of documents. The importance of topic modeling lies in its ability to help us understand the topic structure of document sets, which is helpful for document classification, clustering, and information retrieval. Topic modeling can reduce the dimensionality of high-dimensional text data to a low-dimensional topic space, facilitating subsequent analysis and processing. Common algorithms for topic modeling include LSA (Latent Semantic Analysis), PLSA (Probabilistic Latent Semantic Analysis), and LDA (Latent Dirichlet Allocation).

[0038] <6>FudanDNN-NLP tool: The FudanDNN-NLP tool is a deep learning-based Chinese natural language processing tool developed by the Robotics Research Laboratory of the School of Computer Science, Fudan University. This tool can currently be used for Chinese word segmentation, named entity recognition, part-of-speech tagging, sentence classification, semantic analysis, knowledge base access, and dialogue question answering. The advantages of deep learning methods are that there is no need to perform pre-task feature selection (feature engineering), the system requires fewer parameters (saving memory overhead), and the actual usage speed is better than other systems with similar performance.

[0039] <7>Topic Coherence: Used to determine the correct number of topics. Topic coherence is applied to the top N words of the topic. It is defined as the mean / median of the pairwise similarity scores of the words in the topic. A good model will produce coherent topics, that is, topics with high topic coherence scores. There are many methods to calculate coherence, such as C_UCI, C_UMASS, C_NPMI, CV, and CP. The higher the coherence of a topic, the better the topic.

[0040] <8>Coefficient of Variance (CV): This method is based on a sliding window, performs one-set segmentation on topic words (any two words in a set form a word pair for comparison), and uses Normalized Pointwise Mutual Information (NPMI) and cosine similarity to indirectly obtain the coherence degree.

[0041] As Figure 1 shown, a schematic flowchart of an automatic discovery and evolution method for professional terms and research fields in an embodiment of the present application is presented. The method mainly includes the following steps:

[0042] Step S11: Preprocess the to-be-processed academic literature set. The to-be-processed academic literature set contains multiple academic literatures, and each academic literature is preprocessed. The preprocessing manner includes cleaning and standardizing the to-be-processed academic literature set, where the cleaning includes deleting irrelevant information, invalid characters, special characters, etc.; the standardization includes unifying punctuation marks, unifying fonts, etc.

[0043] For example, irrelevant content is deleted, such as journal URLs, paper URLs, email address information, etc.; invalid characters are deleted, such as garbled characters caused by inconsistent encoding (such as "锟斤拷"), blank characters, symbols without semantic information, etc.; special characters are deleted. The unification of punctuation marks in standardization refers to unifying full-width or half-width, and the unification of fonts refers to unifying traditional Chinese or simplified Chinese.

[0044] It should be noted that in this application, the purpose of cleaning and standardizing the collection of academic documents to be processed is to achieve a unified standard for academic document data, so as to facilitate further processing of the academic document data.

[0045] Step S12: Perform a first classification on the preprocessed set of academic documents to be processed based on the rule classifier; the first classification result includes academic documents that can be clearly classified and academic documents that cannot be clearly classified; perform a second classification on the academic documents that cannot be clearly classified based on the BERT classification model; and form a subject category corpus based on the academic documents that can be clearly classified by the rule classifier and the academic documents that have been classified by the BERT classification model.

[0046] It should be noted that classifying the preprocessed collection of academic documents involves categorizing each document into its respective secondary subject category. The classification standard used in this embodiment is a two-tiered subject classification system formed by integrating and referencing various subject classification standards and the research field classification frameworks of well-known journals. This system includes primary subjects and secondary subjects derived from primary subjects.

[0047] For example, combining Figure 2 The first-level disciplines include: mathematics, mechanics, physics, chemistry, artificial intelligence, etc. The first-level discipline of artificial intelligence also includes second-level disciplines such as: artificial intelligence fundamentals, machine learning, machine perception and pattern recognition, natural language processing, knowledge representation and processing, intelligent systems and applications, and artificial intelligence inspired by cognitive and neuroscience.

[0048] Specifically, a rule-based classifier performs a primary classification of the preprocessed set of academic documents. This involves matching keywords from the documents with the names of the relevant subject categories to categorize documents into their corresponding sub-subjects. Considering cross-disciplinary and cross-domain situations, each document can be classified into one or more sub-subjects. However, a rule-based classifier based solely on keywords cannot classify all academic documents. Therefore, some documents cannot be clearly classified. In this embodiment, the BERT classification model is used to perform a secondary classification of these documents, thereby determining their respective sub-subject categories.

[0049] The BERT classification model is a type of deep learning classification model. When using BERT, it judges whether academic documents are definitively categorized into specific sub-disciplines based on predicted probabilities and pre-set thresholds. BERT is a pre-trained language representation model; therefore, definitively categorizable academic documents obtained by classifying a pre-processed set of academic documents using a rule-based classifier can be input into the BERT model for training, resulting in more accurate classification. The BERT classification model is easy to use, highly stable, and possesses strong language representation and feature extraction capabilities, demonstrating good performance in natural language processing problems.

[0050] Step S13: Calculate the terminology feature parameters corresponding to each string in the academic literature corpus of each subject category. The terminology feature parameters include at least: C_value, basic frequency, combination frequency, and singularity. This requires obtaining the strings from the academic literature and performing word segmentation. For Chinese literature, the FudanDNN-NLP tool can be used for word segmentation. This tool supports word segmentation at various granularities under the constraint of known terms or vocabulary to ensure that terminology is not split during the segmentation process.

[0051] In simple terms, the method calculates the characteristic parameters of the professional terms corresponding to the strings in academic literature with clearly defined disciplines. These parameters include the C_value, fundamental frequency, combinatorial frequency, and singularity. The frequency of the string's appearance in the academic literature can be obtained through these characteristic parameters. It should be understood that this embodiment of the invention comprehensively uses these four characteristic parameters—C_value, fundamental frequency, combinatorial frequency, and singularity—for professional terminology recognition, making it more targeted and accurate.

[0052] Step S14: Calculate the technical term feature parameters corresponding to each string to obtain the technical term recognition threshold corresponding to each string.

[0053] It should be explained that after obtaining the terminology feature parameters of each string, they need to be fused and calculated. For example, the C_value, basic frequency value, combination frequency value, and singularity value are added together to obtain the terminology recognition threshold. In this embodiment, terminology discovery is performed by comprehensively considering the C_value, basic frequency value, combination frequency value, and singularity value, and the above-mentioned terminology feature parameters are combined in a linear manner. Specifically, for each set of academic literature under a second-level discipline, the C_value, basic frequency value, combination frequency value, and singularity value of each string in each academic literature are obtained, and these values ​​are fused and calculated to obtain the terminology recognition threshold. Each string corresponds to a terminology recognition threshold.

[0054] Step S15: Determine the professional terms in the corpus of each subject category by judging according to the professional term recognition threshold corresponding to each string.

[0055] In some examples, the determination is made based on the technical term recognition threshold corresponding to each string. If the technical term recognition threshold corresponding to a string is greater than a preset threshold, then the string is a technical term in the corresponding corpus. Specifically, each academic document includes multiple strings, each corresponding to a technical term recognition threshold. A preset threshold is set for technical term recognition. Each technical term recognition threshold is compared with the preset threshold. If the technical term recognition threshold of a string is less than the preset threshold, then the string is not a technical term; if the technical term recognition threshold of a string is greater than the preset threshold, then the string is a technical term.

[0056] Step S16: Based on the LDA model, perform topic clustering on the professional terms in the corpus of each subject category to obtain topic words and generate new subject categories.

[0057] It should be noted that, based on the identification of technical terms, an unsupervised LDA model is used to discover topics and topic words. LDA is a probabilistic generative model used to classify words in a text or document collection into multiple topics. LDA assumes that documents are generated from a mixture of multiple topics, and that each topic has a certain word distribution. The calculation process of the LDA model includes the following steps:

[0058] Step S161: Determine the number of topic words as a hyperparameter of the LDA model, and randomly initialize the topic distribution and vocabulary distribution for each document and each topic. Determine the number of topics K as a hyperparameter of the algorithm, and randomly initialize the vocabulary distribution for each topic and the topic distribution for each document. The number of topics K is determined using the CV algorithm in Topic Coherence, which is a lexical relevance-based indicator that measures topic coherence by calculating the point mutual information of word co-occurrences.

[0059] Step S162: Find the optimal topic distribution and vocabulary distribution through iterative optimization. In each iteration, the algorithm updates the topic distribution of each document and the vocabulary distribution of each topic based on the current topic and vocabulary distributions using Bayesian network inference. Specifically, LDA uses Bayesian inference to estimate the hidden variables, i.e., the topic distribution of each document and the vocabulary distribution of each topic. It uses techniques such as Gibbs sampling or variational inference to approximate the posterior probability distribution. In Gibbs sampling, a word is extracted from the document, its topic distribution is assumed, and the topic distribution of that word is updated based on the topic distributions of other words. This sampling process is repeated multiple times to obtain an approximate posterior probability distribution.

[0060] Step S163: Gradually converge through multiple iterative sampling and update processes until the topic distribution and vocabulary distribution reach a stable state.

[0061] Step S164: After the algorithm converges, each document will have a topic distribution, and each topic will have a vocabulary distribution. These distributions can be used to interpret text data and perform tasks such as text classification and topic analysis.

[0062] It needs to be explained that after obtaining topic terms through LDA model topic clustering, that is, after forming multiple topics under the second-level subject classification, these multiple topics are equivalent to new subject categories under the second-level subjects. Each topic can be regarded as a sub-sub-subject category, that is, a new research subfield. Combined with... Figure 2 This indicates that further exploration of machine learning at the secondary discipline level using the LDA model can yield new sub-disciplines or research areas, including: deep learning, computer vision, natural language processing, reinforcement learning, and transfer learning.

[0063] Furthermore, the LDA model can be iteratively applied to newly discovered research subfields to further identify more specific areas. When updating terminology and research fields, only the input academic literature needs to be from a specified period. Utilizing the aforementioned automatic discovery and evolution method for terminology and research fields, terminology and research fields from different periods can be automatically discovered, allowing for the observation and study of terminology evolution and disciplinary progress.

[0064] It is important to emphasize that the method for automatic discovery and evolution of professional terms and research fields provided by this invention utilizes a hybrid approach combining computer technology with linguistics and statistics. It first automatically categorizes a large collection of academic literature into various disciplines, then identifies professional terms from these categories, and automatically discovers research subfields through topic modeling. The method in this embodiment can be used for precise talent discovery and recommendation, dynamic tracking of discipline development, etc. It accurately and automatically discovers new professional terms in an unsupervised manner, and iteratively discovers new research fields and hierarchical relationships between fields. It possesses multilingual environment processing and analysis capabilities, can fully cover all disciplines and research fields, and has dynamic update and evolution functions.

[0065] Furthermore, the automatic discovery and evolution method for professional terms and research fields described in this embodiment can promptly identify and discover new professional terms from a large amount of academic literature, achieving a high accuracy rate of over 90%. This method has been successfully used to construct a professional terminology database covering all disciplines and multiple fields, eliminating the need for extensive manual corpus annotation and saving significant manpower costs. It can also be applied to talent assessment and recommendation, providing necessary domain knowledge support and intellectual backing for talent evaluation and recommendation.

[0066] In one embodiment of this application, the calculation method of the C_value includes:

[0067]

[0068] Where t is a string, |t| is the length of string t, TF(t) is the word frequency of string t, s is a long string of words containing string t, and TF(s) is the word frequency of long string s.

[0069] According to Formula 1, the calculation of C_value can be divided into the following two cases:

[0070] Case 1: If t is not a nested string, i.e. The C_value depends on the length of string t, |t|, and the term frequency TF(t) of t.

[0071] Case 2: If t is a nested word string, meaning it represents the frequency of long strings s containing t, for example, if t is "oil", then s could be word strings containing t such as "China National Petroleum Corporation". ∑ s TF(s) is the sum of the frequencies of all terms in s. Let be the size of the set s containing t.

[0072] It should be noted that C-values ​​are a method for evaluating the importance of words. It determines the importance of words by calculating the frequency of their occurrence in the text and the stability of their unique context within the text. The C-values ​​method assumes that important words will appear frequently in the text and that their contextual information is relatively stable. A higher C-value indicates that the word is more important in the text.

[0073] In one embodiment of this application, the calculation method of the fundamental frequency value includes:

[0074] Basic(t) = |t|logf(t) + αe t ; (Formula 2)

[0075] Where Basic is the basic frequency value, t is the string, f(t) is the term frequency of string t, et is the number of long strings containing string t, and α is a custom weight.

[0076] Basic frequency (BFM) is a relatively simple term extraction method that relies solely on word frequency. In the BFM method, the importance of a word is directly proportional to its frequency of occurrence in the text. This means that words that appear frequently in the text are considered more important.

[0077] In one embodiment of this application, the calculation method for the combined frequency value includes:

[0078] ComboBasic(t) =|t|logf(t)+αe t +βe ′ t ; (Formula 3)

[0079] Where ComboBasic is the combined frequency value, t is the string, f(t) is the term frequency of string t, et is the number of long strings containing string t, α is the custom weight, e't is the number of other terms contained in string t, and β is the custom weight.

[0080] ComboBasic is a term extraction method that combines the Basic approach with combinatorial metrics. Combinatorial metrics include a set of indicators used to assess the importance of words in text. ComboBasic uses these combinatorial metrics to comprehensively consider factors such as word frequency and contextual stability in order to more accurately determine word importance.

[0081] In one embodiment of this application, the singularity value is calculated as follows:

[0082]

[0083] Where Weirdness is the singularity value, t is a string, and TF is a string. target (t) represents the word frequency of string t in a domain-specific corpus, TF reference (t) represents the word frequency of string t in a general corpus, and N represents normalization to eliminate the influence of the size of the domain-specific corpus and the general corpus.

[0084] Weirdness is a method for calculating word importance by comparing the frequency of a word's occurrence in a specific corpus with its frequency in a large, general corpus. If a word appears significantly more frequently in a specific corpus than in a general corpus, it is considered relatively strange and thus an important word.

[0085] like Figure 3 The diagram illustrates the structure of an automated discovery and evolution system for technical terms and research fields, as shown in an embodiment of the present invention. The system includes:

[0086] The academic literature collection preprocessing module 301 is used to preprocess the academic literature collection to be processed.

[0087] The subject category corpus acquisition module 302 is used to perform a primary classification on the preprocessed set of academic documents to be processed based on a rule classifier; the primary classification result includes academic documents that can be clearly classified and academic documents that cannot be clearly classified; a secondary classification is performed on the academic documents that cannot be clearly classified based on the BERT classification model; and a subject category corpus is formed based on the academic documents that can be clearly classified by the rule classifier and the academic documents obtained by the secondary classification by the BERT classification model.

[0088] The terminology feature parameter acquisition module 303 is used to calculate the terminology feature parameters corresponding to each string in the academic literature corpus of each subject category; the terminology feature parameters include at least: C_value, basic frequency value, combination frequency value, and singularity value;

[0089] The terminology recognition threshold acquisition module 304 is used to calculate the terminology feature parameters corresponding to each string to obtain the terminology recognition threshold corresponding to each string.

[0090] The terminology identification module 305 is used to identify and obtain the terminology in the corpus of each subject category by making a determination based on the terminology identification threshold corresponding to each string.

[0091] The subject sub-category generation module 306 is used to perform topic clustering on the professional terms in the corpus of each subject category based on the LDA model to obtain topic words and generate new subject categories.

[0092] It should be understood that the division of the various modules or units in the above system is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules or units can be implemented entirely in software through processing element calls; they can be implemented entirely in hardware; or some modules or units can be implemented by processing element calls to software, while others are implemented in hardware.

[0093] For example, these modules or units can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). As another example, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together to form a system-on-a-chip (SOC).

[0094] Since the implementation method of the automatic discovery and evolution system of technical terms and research fields in this embodiment is similar to the implementation method of the automatic discovery and evolution method of technical terms and research fields provided above, it will not be described again.

[0095] like Figure 4 The diagram illustrates the structure of an electronic terminal in one embodiment of this application. The electronic terminal 40 provided in this example includes a memory 41 and a processor 42. The memory 41 stores computer programs; the processor 42 runs the computer programs to implement the aforementioned method for automatic discovery and evolution of technical terms and research fields.

[0096] Optionally, the number of the memory 41 can be one or more, and the number of the processor 42 can be one or more.

[0097] Optionally, the processor 42 in the electronic terminal 40 will perform the following... Figure 1The steps described involve loading one or more instructions corresponding to the process of an application into memory 41, and having the processor 42 run the application stored in the first memory 41, thereby realizing various functions in the automatic discovery and evolution method of technical terms and research fields.

[0098] Optionally, the memory 41 may include, but is not limited to, high-speed random access memory and non-volatile memory. For example, one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices; the processor 42 may include, but is not limited to, a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] Optionally, the processor 42 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0100] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for automatic discovery and evolution of technical terms and research fields.

[0101] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0102] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0103] In summary, this application provides a method, system, terminal, and medium for the automatic discovery and evolution of professional terms and research fields. This application utilizes a hybrid approach combining computer technology with linguistics and statistics to automatically categorize academic literature collections containing a large amount of academic literature into various disciplines, then identify professional terms from these collections, and automatically discover research subfields through topic models. This application can accurately and automatically discover new professional terms from academic literature within the same research field, saving significant manual annotation costs; it can iteratively discover new research fields within each discipline category using topic models according to actual needs, and establish hierarchical relationships between disciplines and fields; it possesses multilingual environment processing and analysis capabilities, enabling automatic discovery of professional terms and research fields from academic literature in multiple languages, including Chinese and English; when the amount of academic literature is sufficient, it can provide full coverage of all disciplines and research fields, including less common specialties and research areas; it has dynamic update and evolution functions, enabling timely adjustments to previously discovered professional terms and research fields, and rapid discovery of new professional terms and research fields, achieving the goals of real-time discipline tracking and talent assessment. Therefore, this application effectively overcomes the various shortcomings of the prior art and has high industrial application value.

[0104] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for automatic discovery and evolution of technical terms and research fields, characterized in that, include: Preprocess the collection of academic documents to be processed; The preprocessed set of academic documents to be processed is classified once using a rule classifier; the first classification result includes academic documents that can be clearly classified and academic documents that cannot be clearly classified; the academic documents that cannot be clearly classified are classified a second time using a BERT classification model; the academic documents that can be clearly classified by the rule classifier and the academic documents that are classified a second time by the BERT classification model are combined to form a subject category corpus. The strings of academic documents in the corpora of each subject category are calculated to obtain the professional term feature parameters corresponding to each string; The technical terminology feature parameters include at least: C_value, fundamental frequency, combination frequency, and singularity. The technical term feature parameters corresponding to each string are calculated to obtain the technical term recognition threshold corresponding to each string; The professional terms in the corpus of each subject category are identified and obtained by judging based on the professional term recognition threshold corresponding to each string. Based on the LDA model, the professional terms in the corpora of each subject category are clustered to obtain topic words and generate new subject categories.

2. The method for automatic discovery and evolution of professional terms and research fields according to claim 1, characterized in that, The preprocessing method includes cleaning and standardizing the collection of academic documents to be processed.

3. The method for automatic discovery and evolution of professional terms and research fields according to claim 1, characterized in that, The calculation method for the C_value includes: Where t is a string, |t| is the length of string t, TF(t) is the word frequency of string t, s is a long string of words containing string t, and TF(s) is the word frequency of long string s.

4. The method for automatic discovery and evolution of technical terms and research fields according to claim 1, characterized in that, The calculation method for the fundamental frequency value includes: Basic(t)=|t|logf(t)+αe t ; Where Basic is the basic frequency value, t is the string, f(t) is the term frequency of string t, and e is the frequency of the word in string t. t α represents the number of words in a long string containing string t, and α is a custom weight.

5. The method for automatic discovery and evolution of professional terms and research fields according to claim 1, characterized in that, The calculation method for the combined frequency value includes: ComboBasic(t)=|t|logf(t)+αe t +βe ′ t ; Where ComboBasic is the combination frequency value, t is a string, f(t) is the term frequency of string t, and e t e' represents the number of words in a long string containing string t, where α is a custom weight; t β represents the number of other terms contained in string t, and β is a custom weight.

6. The method for automatic discovery and evolution of technical terms and research fields according to claim 1, characterized in that, The method for calculating the singularity value includes: Where Weirdness is the singularity value, t is a string, and TF is a string. target (t) represents the word frequency of string t in a domain-specific corpus, TF reference (t) represents the word frequency of string t in a general corpus, and N represents normalization to eliminate the influence of the size of the domain-specific corpus and the general corpus.

7. The method for automatic discovery and evolution of technical terms and research fields according to claim 1, characterized in that, The professional terms in each corpus are identified and obtained by determining the professional terminology recognition threshold corresponding to each string, specifically including: If the threshold for identifying the technical term corresponding to the string is greater than a preset threshold, then the string is a technical term from the corresponding corpus.

8. The method for automatic discovery and evolution of technical terms and research fields according to claim 1, characterized in that, The calculation process of the LDA model includes the following: The number of topic words is determined as a hyperparameter of the LDA model, and the topic distribution and vocabulary distribution of each topic are randomly initialized for each document. The algorithm finds the optimal topic distribution and vocabulary distribution through iterative optimization. In each iteration, the algorithm updates the topic distribution of each document and the vocabulary distribution of each topic by inferring from a Bayesian network based on the current topic and vocabulary distribution. The algorithm gradually converges through multiple iterative sampling and updating processes until the topic distribution and vocabulary distribution reach a stable state. After convergence, each document will have a topic distribution, and each topic will have a vocabulary distribution.

9. An automatic discovery and evolution system for technical terms and research fields, characterized in that, include: The academic literature collection preprocessing module is used to preprocess the academic literature collection to be processed. The subject category corpus acquisition module is used to perform a primary classification of the preprocessed set of academic documents to be processed based on a rule classifier; the primary classification results include academic documents that can be clearly classified and academic documents that cannot be clearly classified; a secondary classification is performed on the academic documents that cannot be clearly classified based on the BERT classification model; the subject category corpus is composed of the academic documents that can be clearly classified by the rule classifier and the academic documents that are classified by the BERT classification model. The professional term feature parameter acquisition module is used to calculate the professional term feature parameters corresponding to each string in the academic literature corpus of each subject category. The technical terminology feature parameters include at least: C_value, fundamental frequency, combination frequency, and singularity. The terminology recognition threshold acquisition module is used to calculate the terminology feature parameters corresponding to each string to obtain the terminology recognition threshold corresponding to each string. The professional terminology identification module is used to identify and obtain professional terms in the corpus of each subject category by judging according to the professional terminology identification threshold corresponding to each string. The subject sub-category generation module is used to perform topic clustering on the professional terms in the corpus of each subject category based on the LDA model to obtain topic words and generate new subject categories.

10. An electronic terminal, characterized in that, include: Processor and memory; The memory is used to store computer programs; The processor is configured to execute the computer program stored in the memory to enable the electronic terminal to perform the automatic discovery and evolution method of technical terms and research fields as described in any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for automatic discovery and evolution of technical terms and research fields as described in any one of claims 1 to 8.

Images