Term corpus generation method and system based on literature content deep mining

Abstract

This invention provides a method and system for generating terminology corpora based on in-depth mining of document content, relating to the field of terminology recognition technology. The method includes: acquiring scientific and technological document data; preprocessing the data to construct a candidate terminology set; traversing the candidate terminology set for multidimensional analysis to extract a multidimensional feature set, which includes syntactic and semantic features, statistical features, and domain knowledge features; performing dynamic weighted calculations based on the syntactic and semantic features, statistical features, and domain knowledge features to obtain a comprehensive score; and backtracking to the candidate terminology set based on the comprehensive score for filtering, generating a terminology training corpus. This invention solves the technical problems of existing technologies that use rules for semi-automatic generation or filtering of corpora. The formulation and maintenance of these rules require extensive professional knowledge and experience, resulting in high rule construction costs, poor domain adaptability, and difficulty in addressing the complexity and diversity of terminology in scientific and technological documents.

Description

Technical Field

[0001] This invention relates to the field of terminology recognition technology, specifically to a method and system for generating terminology corpora based on in-depth mining of document content. Background Technology

[0002] With the exponential growth in the number of scientific and technological documents, accurate identification of domain-specific terms in these documents is crucial for constructing a computable domain knowledge organization system and improving the efficiency of intelligent information retrieval. As the core carrier of professional knowledge and the core representation of domain information, the identification technology of domain-specific terms has become a core research direction in the field of natural language processing, providing fundamental support for building a domain knowledge management system and promoting the development of intelligent information processing technology. The construction of a high-quality corpus to support the training of terminology recognition models is the cornerstone of research in this field.

[0003] Existing technologies mainly focus on the construction methods of training corpora for domain terminology recognition. From the perspective of database construction methods supporting model training, they have generally gone through four development stages: manual construction, rule-based construction, model iterative construction, and prompting engineering-based construction. Among these, manually constructed databases are of high quality but consume significant human and material resources; rule-based databases require expert knowledge support, have high rule construction costs, poor domain adaptability, and struggle to cope with the complexity and diversity of terminology in scientific literature; model iterative methods are easily affected by the model's recognition results, and the model's interpretability and domain adaptability remain to be addressed, with poor adaptability when transferring between different domains; prompting engineering-based methods make it difficult to trace the source of the corpus, and its reliability needs improvement. In summary, although significant progress has been made in the construction of training corpora for domain terminology recognition, many shortcomings and deficiencies still exist. Summary of the Invention

[0004] This application provides a method and system for generating terminology corpora based on in-depth mining of document content. It aims to solve the technical problems of existing technologies that use rules to semi-automatically generate or screen corpora. The formulation and maintenance of rules require a lot of professional knowledge and experience, resulting in high rule construction costs, poor domain adaptability, and difficulty in dealing with the complexity and diversity of terminology in scientific and technological literature.

[0005] The first aspect disclosed in this application provides a method for generating terminology corpus based on in-depth mining of document content. The method includes: acquiring scientific and technological document data; preprocessing the scientific and technological document data to construct a candidate terminology set; traversing the candidate terminology set to perform multidimensional analysis and extract a multidimensional feature set, wherein the multidimensional feature set includes syntactic and semantic features, statistical features, and domain knowledge features; performing dynamic weighted calculation based on the syntactic and semantic features, the statistical features, and the domain knowledge features to obtain a comprehensive score; and backtracking to the candidate terminology set based on the comprehensive score for screening to generate terminology training corpus.

[0006] The second aspect disclosed in this application provides a terminology corpus generation system based on in-depth mining of document content. The system is used in the aforementioned terminology corpus generation method based on in-depth mining of document content. The system includes: a candidate terminology set construction module, used to acquire scientific and technological document data, preprocess the data, and construct a candidate terminology set; a multidimensional analysis module, used to traverse the candidate terminology set for multidimensional analysis and extract a multidimensional feature set, the multidimensional feature set including syntactic and semantic features, statistical features, and domain knowledge features; a weighted calculation module, used to perform dynamic weighted calculation based on the syntactic and semantic features, the statistical features, and the domain knowledge features to obtain a comprehensive score; and a filtering module, used to backtrack to the candidate terminology set based on the comprehensive score for filtering, generating terminology training corpus.

[0007] One or more technical solutions provided in this application have at least the following beneficial effects: By acquiring high-quality data from authoritative scientific and technological literature sources and performing preprocessing, the accuracy of subsequent analysis is ensured. Automated data collection and preprocessing reduce manual intervention and significantly improve data acquisition efficiency. By comprehensively evaluating the effectiveness of candidate terms using syntactic and semantic features, statistical features, and domain knowledge features, the accuracy of term selection is improved, potential domain terms are discovered, and candidate terms are ensured to meet domain standards, increasing their domain relevance and reliability. By dynamically adjusting the weights of different features, the evaluation criteria can be adjusted according to the specific data and needs, making the comprehensive term score more accurate, especially adaptively adjusting under different corpus sizes and domain characteristics. By calculating the comprehensive score, the quality of each candidate term can be accurately measured, ensuring that the final selected terms best meet domain standards and improving the quality of the term training corpus. Based on the comprehensive score, terms that meet domain standards are automatically selected, resulting in a more efficient term training corpus, reducing interference from irrelevant terms, and providing a solid data foundation for subsequent term recognition model training. Through automated screening and optimization processes, large amounts of scientific and technological literature data can be processed, improving the efficiency of corpus construction and avoiding errors and biases that may occur in traditional manual screening processes.

[0008] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the terminology corpus generation method based on in-depth mining of document content provided in the embodiments of this application.

[0010] Figure 2 A schematic diagram of the terminology corpus generation system based on in-depth mining of document content provided in this application embodiment.

[0011] Figure labeling: Candidate term set construction module 10, multidimensional analysis module 20, weighted calculation module 30, and screening module 40. Detailed Implementation

[0012] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0013] Example 1, as Figure 1 As shown in the embodiments of this application, a method for generating terminology corpus based on in-depth mining of document content is provided, the method comprising: Acquire scientific and technological literature data, preprocess the scientific and technological literature data, and construct a candidate term set.

[0014] Relevant scientific literature data is obtained from academic databases, such as Web of Science. This data primarily includes article titles, abstracts, and keywords. Sources are from high-impact journals in Q1 quartiles, ensuring the data's authority and reliability. The scientific literature data undergoes preprocessing, including text segmentation, redundancy removal, part-of-speech tagging, and dependency parsing. The purpose of preprocessing is to remove noisy data, making the dataset clean and conducive to subsequent terminology recognition. From the preprocessed data, all potential candidate terms are extracted, constructing a candidate term set. Candidate terms include author-annotated keywords, terms from specialized thesaurus, automatically annotated keywords from the literature, or terms extracted through algorithms.

[0015] The candidate term set is traversed for multidimensional analysis to extract a multidimensional feature set, which includes syntactic and semantic features, statistical features, and domain knowledge features.

[0016] Extracting syntactic and semantic features involves analyzing the position of terms in sentences to determine whether they appear in key syntactic positions such as subject, object, and predicate. If a term appears in these positions, it indicates that it has a higher probability of being a term. Analyzing the part-of-speech combination, overall structure, and length of terms reveals that common terms are often composed of nouns or adjectives combined with nouns, and their length is usually between 2 and 8 words. Determining the sentence type in which the term appears indicates that terms frequently appear in sentence types such as research background sentences, research question sentences, and research method sentences, with different weights assigned to different sentence types.

[0017] Statistical features are extracted. Specifically, the significance of each candidate term in a specific domain is calculated using term frequency-inverse document frequency (TF-IDF) to identify terms that appear frequently in that domain. The co-occurrence of words within terms is analyzed using an N-Gram model, and the combination strength within terms is captured through a dynamic window. The statistical correlation between terms is measured by mutual information values ​​to identify potential relationships between terms.

[0018] Extract domain knowledge features. Specifically, score terms based on their source, such as whether they come from expert annotations or authoritative thesaurus, with more authoritative terms receiving higher scores. Calculate scores based on the matching degree between candidate terms and authoritative domain thesaurus, using three methods: exact matching, fuzzy matching, and component matching.

[0019] A comprehensive score is obtained by dynamically weighting the syntactic and semantic features, the statistical features, and the domain knowledge features.

[0020] Based on the characteristics of different domains, such as the regularity of term formation and the salience of term distribution, the weights of each feature are adjusted. The weight coefficients are adaptively adjusted according to the corpus size, domain characteristics, and the discriminative power of the features. For example, in domains with strong regularity in term formation, the weight of syntactic and semantic features is increased; in domains with high salience of term distribution, the weight of statistical features is increased. Each candidate term will receive a comprehensive score based on a weighted score of its syntactic and semantic features, statistical features, and domain knowledge features. The comprehensive score can be calculated using a weighted average or weighted fusion algorithm, ultimately forming a comprehensive evaluation of each candidate term.

[0021] The candidate term set is back-filtered based on the comprehensive score to generate term training corpus.

[0022] Based on the comprehensive score of each candidate term, a backtracking screening process is performed. If a term has a high comprehensive score, exceeding a preset threshold, it is considered a high-quality term and can be included in the training corpus. If the score is low, its inclusion is determined through manual review or an automatic filtering mechanism. Finally, the screened terms are organized and output as a standardized term training corpus, which includes the term itself, the sentences in which the term is located, and the score information of various features. This high-quality term data is used for subsequent training of the domain terminology recognition model.

[0023] Furthermore, the method for acquiring scientific and technological literature data, preprocessing the data, and constructing a candidate term set includes: Acquire scientific and technological literature data, perform quality screening on the scientific and technological literature data, and construct a basic dataset; extract text data based on the basic dataset, traverse the text data to perform text cleaning, and generate standard text data, which includes title data, abstract data, and keyword data; label the title data, abstract data, and keyword data according to multiple data sources to determine multiple candidate terms; construct an authoritative terminology database to merge and deduplicate the multiple candidate terms, and construct the candidate terminology set.

[0024] The source of scientific literature is selected from academic databases such as Web of Science and CNKI to obtain relevant literature data. This data includes the titles, abstracts, and keywords of the articles. The selection criteria for this data are based on journal impact and timeliness to ensure the authority of the data. After obtaining the data, a quality screening process is performed to remove low-quality, duplicate, or irrelevant literature. The screening criteria include the publication date of the article, the journal's impact factor, and relevance to the field. This quality screening ensures that the selected literature data meets the requirements of subsequent processing. The screened literature data constitutes the basic dataset, which serves as the foundation for subsequent text processing and terminology extraction.

[0025] Text data, including titles, abstracts, and keywords, is extracted from each document in the basic dataset. This text data is the core of subsequent analysis and processing; therefore, the extraction stage must ensure data integrity and accuracy. Text cleaning aims to remove noise, irrelevant data, and formatting issues. Specific operations include: removing punctuation, special characters, and spaces; converting capitalization; removing common stop words; and performing word segmentation to divide the text into individual words. Through cleaning and standardization, standard text data is generated, including title data, abstract data, and keyword data. All text is formatted uniformly, making the data more standardized and facilitating subsequent terminology extraction and analysis.

[0026] The obtained title data, abstract data, and keyword data are annotated by combining multiple data sources, including keywords annotated by literature authors, authoritative domain terminology databases, and automated annotation tools. Based on these multiple data sources, a set of candidate terms is determined. These candidate terms are single words or phrases that represent core concepts in the field.

[0027] Standardized glossaries in relevant fields are collected, such as the IUPAC Goldbook used in chemistry, Gene Ontology in biology, or the Materials Project in materials science. In addition to authoritative thesaurus within the field, automated methods are used to generate a terminology database, such as extracting common terms based on large language models (e.g., ChatGPT). Based on the authoritative terminology database, candidate terms from different sources are merged, as different sources may extract the same or similar terms. Specifically, duplicates are removed by comparing candidate terms with those in the authoritative database to ensure the standardization of the terminology set. For word forms such as singular / plural, case variations, and synonyms, merging is performed to ensure the uniformity and consistency of the terminology set. After merging and deduplication, a candidate terminology set without duplicates and covering terms from various fields is finally obtained. This set is used for subsequent terminology feature extraction and comprehensive evaluation.

[0028] Furthermore, the construction process of syntactic and semantic features includes the following methods: Syntactic analysis is performed on the candidate term set to extract syntactic position features. Based on these features, it is determined whether the candidate term set appears in the subject, object, or predicate position, and a syntactic position feature score is calculated. Lexical structure features are extracted from the candidate term set, and part-of-speech tagging is performed based on these features. A lexical structure feature score is calculated based on the lexical structure features. Step analysis is performed on the candidate term set to extract step type features. Based on these features, it is determined whether the candidate term set appears in the research question sentence, research object sentence, or research background sentence, and a step type feature score is calculated. The syntactic position feature score, lexical structure feature score, and step type feature score are weighted and fused to generate the syntactic-semantic features.

[0029] Using syntactic analysis tools, such as Stanford's dependency parser, the sentences containing terms in the candidate term set are syntactically analyzed. Syntactic analysis identifies the grammatical relationships between words in the sentence, including subject-verb and object relationships. The position of each candidate term in the sentence is analyzed to determine if it appears in key syntactic positions such as subject, object, or predicate nominative. If a term appears in a subject, object, or predicate nominative position, it receives bonus points; if it appears in other positions, the score is adjusted according to its syntactic position. Terms appearing in key positions indicate that they are core domain terms and therefore receive higher scores.

[0030] For each candidate term, its lexical structure features are extracted, and the part-of-speech combinations of its constituent words are analyzed. For example, domain terms are composed of noun-noun combinations or adjective-noun combinations. Part-of-speech tagging tools, such as POS taggers, are used to tag the candidate terms, identifying the part of speech of each word in the term. Based on the part-of-speech identification results, a lexical structure feature score is calculated. The calculation method includes: if a term is composed of noun-noun combinations or adjective-noun combinations, it is considered a standard term form and given a higher score; for other less common part-of-speech combinations, the score is correspondingly lower; it is also determined whether the overall part of speech of the term is a noun phrase; if it is, the score is higher; otherwise, it is lower.

[0031] Word step analysis refers to analyzing the context of the sentence containing candidate terms to determine which word step the term appears in. Different word step types affect the domain relevance of terms, thus requiring word step analysis. Sentences in scientific literature can be divided into different word step types, such as research background sentences, research question sentences, research method sentences, and research result sentences, each carrying different semantic information. Based on the word step type of the sentence containing each candidate term, word step type features are extracted. Terms often appear frequently in sentences of certain specific word step types; for example, research method sentences often contain terms describing experimental techniques or tools. For each candidate term, a higher score is given if it appears in common word steps such as research background sentences, research question sentences, and research method sentences; a lower score is given if it appears in other less common word step types. Specifically, by analyzing the characteristics of speech step types, it is determined whether the candidate term set appears in different speech step types. For example, if a term appears in a research background sentence, it means that the candidate term has strong background and universality. The candidate terms are assigned weights according to the speech step type. Candidate terms in the research question sentence, research object sentence, research background sentence, etc. will be assigned higher scores, while other speech step types will be assigned lower scores. According to the score allocation results, the speech step type feature score is calculated by statistically analyzing the frequency of the candidate term set in various speech steps.

[0032] To comprehensively evaluate the syntactic and semantic features of candidate terms, different weight coefficients are assigned based on the importance of each feature. Specifically, the weights of syntactic position features, lexical structure features, and step type features are adjusted according to the characteristics of the domain. A weighted fusion method is used to calculate the comprehensive syntactic and semantic feature score for each candidate term. This score represents the degree of syntactic and semantic adaptation of the term and can also reflect the importance of the term in the domain.

[0033] Furthermore, the lexical structure feature score is calculated based on the part-of-speech recognition results. Methods include: Based on the part-of-speech tagging (POS) results, multiple POS combinations are determined, including double noun combinations and noun-adjective combinations. When the POS result is a double noun combination or a noun-adjective combination, a first sub-score is generated; when the POS result is not a double noun combination or a noun-adjective combination, a second sub-score is generated, wherein the first sub-score is greater than the second sub-score. Based on the candidate term set, POS tagging is performed; when the recognition result is a noun phrase, a third sub-score is generated; when the recognition result is not a noun phrase, a fourth sub-score is generated, wherein the third sub-score is greater than the fourth sub-score. Based on the candidate term set, segment length is identified, and a length sub-score is calculated. The first or second sub-score, the third or fourth sub-score, and the length sub-score are weighted and summed to generate the lexical structure feature score.

[0034] Based on the part-of-speech recognition results, the composition of candidate terms is analyzed to determine whether they conform to common part-of-speech combinations. Double noun combinations are composed of two nouns and are usually used as terms in scientific and technological literature; noun-adjective combinations are composed of a noun and an adjective and are also usually used as terms.

[0035] If the part-of-speech combination of a candidate term is a double noun combination or a noun-adjective combination, i.e., it conforms to common terminology patterns, a first sub-score is generated. These combinations indicate that the term is more likely to be a domain term, and therefore a higher score is assigned. If the part-of-speech combination of a candidate term does not conform to double noun combinations or noun-adjective combinations, a second sub-score is generated. These terms do not conform to common terminology formation patterns, and therefore a lower score is assigned.

[0036] Part-of-speech tagging is performed again, specifically checking whether the candidate terms are noun phrases. Noun phrases are a common form of domain terminology, consisting of one or more nouns. If the candidate term's part-of-speech tagging result is a noun phrase, a third sub-score is generated because noun phrases are the most common form of domain terminology, and therefore a higher score is assigned. If the candidate term's part-of-speech tagging result is not a noun phrase, for example, containing verbs or other parts of speech, a fourth sub-score is generated. These terms do not conform to the common structure of domain terminology and are therefore assigned a lower score. The third sub-score is higher than the fourth sub-score because noun phrases are the standard form of domain terminology.

[0037] The length of each candidate term is determined by calculating the number of words it contains, i.e., the token length. Term length typically ranges from 2 to 8 words. Longer terms can more accurately describe concepts within a specific domain. A length sub-score is assigned to each term based on its length. Terms of moderate length, such as 2-5 words, are common in domain terminology and therefore receive higher scores. Shorter or longer terms do not conform to common domain terminology patterns and thus receive lower scores.

[0038] The first, second, third, fourth and length sub-scores are weighted and summed to obtain the lexical structure feature score of each candidate term. This score reflects whether the lexical structure of the term conforms to the common features of domain terms. Term with a higher score is more in line with the composition pattern of domain terms.

[0039] Furthermore, the process and methods for constructing domain knowledge features include: Based on the multiple data sources, an authority score is generated to produce a source authority score. Terminology matching is performed according to the source authority score to construct an authoritative domain thesaurus. The candidate term set is matched against the authoritative domain thesaurus, and a thesaurus matching score is calculated. Based on the thesaurus matching score, it is determined whether the candidate term set originates from the authoritative domain thesaurus. When the thesaurus matching score is greater than or equal to a preset thesaurus matching threshold, the candidate term set is determined to originate from the authoritative domain thesaurus, and a score-adding operation is performed on the candidate term set to generate domain knowledge features. When the thesaurus matching score is less than the preset thesaurus matching threshold, the candidate term set is determined not to originate from the authoritative domain thesaurus, and a score-subtracting operation is performed on the candidate term set to generate domain knowledge features.

[0040] Data sources include keywords annotated by literature authors, authoritative domain terminology databases, and automated annotation tools. Different authoritative scores are given to candidate terms based on different data sources. Generally, terms annotated by domain experts or authoritative terminology databases will receive higher scores.

[0041] Candidate terms are matched against authoritative thesaurus within the domain. Authoritative thesauruses include: glossaries compiled by domain experts, such as the IUPAC Chemical Terminology and Gene Ontology; and extended thesauruses automatically indexed by academic databases. By matching and validating the standardization of the terms, an authoritative domain thesaurus is constructed. The purpose of this thesaurus is to ensure that candidate terms conform to domain standards, facilitating subsequent training and model building.

[0042] Candidate terms are matched against a predefined authoritative domain thesaurus to verify their consistency with standard terms. During the matching process, a matching score is calculated based on the degree of matching between the candidate terms and the terms in the domain thesaurus.

[0043] Set a vocabulary matching threshold T know This value is obtained through empirical research or judgment by domain experts; for example, setting T... know =0.3 is used to determine whether candidate terms conform to domain standards. If the term matching score of a candidate term is greater than or equal to the threshold, the term is considered to originate from an authoritative domain thesaurus; if the term matching score is lower than the threshold, the term is considered to not conform to domain standards and may be a noisy term.

[0044] For candidate terms whose word list matching score is greater than or equal to the threshold, they are considered to meet the domain standards and can be given a higher domain knowledge feature score. The scoring operation can be combined with the source authority score based on the authority of the term, such as keywords marked by the author, domain word list or academic database, for example, adding 10 points. After adding the score, the domain knowledge feature is generated.

[0045] For candidate terms with a vocabulary matching score below a threshold, they are considered non-compliant with domain standards and a score reduction should be applied. For example, terms from large language models or automated indexing can have their scores reduced by a certain amount, such as 10 points. After the score reduction, domain knowledge features are generated. Through these steps, the construction process of domain knowledge features can effectively evaluate the domain relevance and authority of each candidate term, ensuring that the final term set meets professional standards and providing reliable data support for subsequent term recognition model training.

[0046] Furthermore, a comprehensive score is obtained by dynamically weighting the syntactic and semantic features, the statistical features, and the domain knowledge features. The method includes: Based on the syntactic and semantic features, the statistical features, and the domain knowledge features, score allocation calculations are performed to determine the syntactic and semantic feature scores, statistical feature scores, and domain knowledge feature scores. A corpus size parameter is defined to perform dynamic weight analysis on the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores, generating multiple dynamic weight coefficients. These multiple dynamic weight coefficients include a first weight coefficient, a second weight coefficient, and a third weight coefficient, wherein the first weight coefficient, the second weight coefficient, and the third weight coefficient correspond to the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores. The syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores are weighted and fused according to the first weight coefficient, the second weight coefficient, and the third weight coefficient to calculate a comprehensive score.

[0047] Each candidate term is assigned a score based on its various features, including syntactic and semantic features, statistical features, and domain knowledge features. These scores reflect the candidate term's performance and adaptability on each feature dimension. Each candidate term receives a different score on these features, which reflects the term's relevance, standardization, and adaptability.

[0048] The corpus size parameter is determined based on the corpus size, such as the number of documents or the size of the dataset. The corpus size affects the weight of each feature in the overall score. For example, in small corpora, more reliance is needed on domain knowledge features to ensure the domain relevance of candidate terms, while in large corpora, statistical features are more critical. Based on the corpus size parameter, the weight coefficients of each feature are dynamically adjusted. For example, if the corpus is small, domain knowledge features are given higher weight; while for large corpora, statistical features are given higher weight.

[0049] The first weighting coefficient is related to the syntactic and semantic feature scores, reflecting the term's importance at the syntactic level; the second weighting coefficient is related to the statistical feature scores, typically reflecting the term's performance at the data statistics level; and the third weighting coefficient is related to the domain knowledge feature scores, reflecting the term's domain relevance. These dynamic weighting coefficients adaptively adjust according to different conditions, thereby optimizing the term's overall score.

[0050] The first, second, and third weighting coefficients are used to weight and fuse the syntactic and semantic feature scores, statistical feature scores, and domain knowledge feature scores to generate a total comprehensive score. Terms with higher comprehensive scores indicate that they perform better in each feature dimension, are more in line with the standards of domain terms, and therefore have higher credibility.

[0051] Furthermore, by defining the corpus size parameter, dynamic weight analysis is performed on the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores to generate multiple dynamic weight coefficients. The method includes: Based on the corpus size parameters, a target domain is determined for term formation analysis to generate a term formation regularity index; a term distribution analysis is performed based on the target domain to generate a term distribution significance index; when the term formation regularity index is higher than a first index threshold, the first weight coefficient is increased, and the second and third weight coefficients are decreased to generate a first dynamic weight coefficient; when the term distribution significance index is higher than a second index threshold, the second weight coefficient is increased, and the first and third weight coefficients are decreased to generate a second dynamic weight coefficient; when the corpus size is less than a preset size threshold, the third weight coefficient is increased, and the first and second weight coefficients are decreased to generate a third dynamic weight coefficient.

[0052] Based on the corpus of the target domain, the formation patterns of terms are analyzed. For example, in some domains, terms tend to be combinations of nouns, while in others, they are more often combinations of adjectives and nouns. By analyzing these patterns, the word-formation characteristics of terms in that domain are determined. Based on the analysis results, a term formation pattern index is generated, which reflects the consistency and standardization of the word-formation patterns of terms in the target domain.

[0053] This analysis examines the distribution of terms within the target domain in the corpus. This includes the frequency of term occurrences in different documents and the frequency of term occurrences in specific contexts. The significance of term distribution is correlated with the core nature of the term within the domain. Based on the frequency and importance of term distribution, a significance index for term distribution is generated. A high significance value indicates that the term is highly representative within the domain and may be a core term in that domain; a low significance value indicates that the term is relatively marginal and occurs less frequently.

[0054] If the term formation regularity index is higher than the preset first index threshold, that is, the term formation in the domain is relatively standardized, it indicates that the term formation regularity is of high importance to the term recognition in the domain. At this time, the first weight coefficient related to syntactic and semantic features is increased, because terms with strong term formation regularity usually have relatively fixed syntactic and semantic structures. At the same time, the second and third weight coefficients of statistical features and domain knowledge features are decreased, because in domains with strong term formation regularity, the terms rely more on grammatical structure and vocabulary composition patterns than on frequency distribution or domain knowledge.

[0055] If the significance of a term's distribution is higher than the preset second indicator threshold, it indicates that the term is highly representative in the target domain and will usually appear in more literature. In this case, increase the second weight coefficient related to statistical features, because the high frequency of occurrence and statistical importance of a term become the key to assessing whether a term is a domain term. At the same time, reduce the first and third weight coefficients of syntactic and semantic features and domain knowledge features, because when the significance of a term's distribution is strong, the statistical features of the term are more important.

[0056] If the corpus size is smaller than the preset size threshold, for example, if the number of documents is small or the dataset is small, it means that the training data is relatively scarce. In this case, the third weight coefficient related to domain knowledge is increased because the domain relevance of terms is crucial to the accuracy of term recognition in small corpora. At the same time, the first and second weight coefficients of syntactic and semantic features and statistical features are reduced because statistical features are insufficient to provide effective guidance in small corpora, and the regularity of grammatical structures is also affected by the scarcity of data.

[0057] Furthermore, the method of backtracking to the candidate term set based on the comprehensive score to generate term training corpus includes: A hierarchical decision threshold is preset, including a first threshold and a second threshold, wherein the first threshold is greater than the second threshold. A threshold comparison result is generated by comparing the comprehensive score with the hierarchical decision threshold. The candidate terminology set is divided into a directly adopted subset, a manually reviewed subset, and an automatically filtered subset according to the threshold comparison result. The directly adopted subset is included in the terminology training corpus. The manually reviewed subset is transmitted to a manually reviewed unit for review and confirmation. Based on the confirmation result, the manually reviewed subset is included in the terminology training corpus. The automatically filtered subset is removed and not included in the terminology training corpus, thus generating the terminology training corpus.

[0058] A hierarchical decision-making criterion is established to make screening decisions based on the comprehensive scores of candidate terms. This hierarchical decision threshold includes a first threshold and a second threshold, which are used to distinguish different sets of terms. The first threshold directly adopts terms with higher comprehensive scores; if a candidate term's comprehensive score is higher than this threshold, it is considered a high-quality term and can be directly adopted. The second threshold includes terms with moderate scores in a subset for manual review; if a candidate term's comprehensive score falls between the first and second thresholds, it requires further manual review to confirm its inclusion in the final training corpus.

[0059] The overall score of each candidate term is compared with a first threshold and a second threshold to generate a threshold comparison result: if the overall score is greater than the first threshold, the term is assigned to the direct adoption subset; if the overall score is between the first threshold and the second threshold, the term is assigned to the manual review subset; if the overall score is lower than the second threshold, the term is assigned to the automatic filtering subset.

[0060] Terms from the directly adopted subset are directly included in the terminology training corpus. These terms have already been evaluated as conforming to domain standards and require no further processing. Terms from the manually reviewed subset undergo review by a human review unit. This review process is conducted by domain experts, whose task is to confirm whether these terms meet the standards and are suitable for inclusion in the training corpus. Terms that pass the review are added to the training corpus; those that fail are removed from the candidate terminology pool. Terms from the automatically filtered subset have low overall scores and are therefore automatically removed from the training corpus. Finally, all selected terms are included in the final terminology training corpus, which will serve as the foundational data for model training.

[0061] Furthermore, based on the comparison between the comprehensive score and the hierarchical decision threshold, a threshold comparison result is generated. The candidate term set is then divided into a direct adoption subset, a manually reviewed subset, and an automatically filtered subset according to the threshold comparison result. The method includes: The candidate term set is traversed and compared with the hierarchical decision threshold: S1: When the comprehensive score of the candidate term set is greater than or equal to the first threshold, a first candidate term is extracted and assigned to the direct adoption subset; S2: When the comprehensive score of the candidate term set is less than the first threshold but greater than or equal to the second threshold, a second candidate term is extracted and assigned to the manual review subset; S3: When the comprehensive score of the candidate term set is less than the second threshold, a third candidate term is extracted and assigned to the automatic filtering subset.

[0062] If the overall score of a candidate term is higher than or equal to the first threshold, it means that the quality of the term has been sufficiently guaranteed and it is directly included in the direct adoption subset. These terms are judged to meet the core standards of the domain and do not require further verification.

[0063] If the overall score of a candidate term falls between the first and second thresholds, it indicates that the term is not perfect in some aspects but still has a certain degree of credibility. Further manual review is required. These terms are then included in the manual review subset and await further confirmation from domain experts to determine whether they meet the standards for domain terms. Only if the experts confirm their validity after review will they be included in the training corpus.

[0064] If the overall score of a candidate term is lower than the second threshold, it indicates that its quality is low, it does not conform to the core features of domain terms, or it is noisy data. These terms are included in the automatic filtering subset and will be removed from the candidate term set and will not be included in the subsequent training corpus construction.

[0065] This hierarchical screening method helps improve the quality of training data and ensures that the terms ultimately included in the training conform to domain standards, thereby improving the accuracy and reliability of the term recognition model.

[0066] Example 2, based on the same inventive concept as the terminology corpus generation method based on in-depth document content mining in the previous examples, such as... Figure 2 As shown in the embodiment of this application, a terminology corpus generation system based on deep mining of document content is provided. The system includes: The candidate term set construction module 10 is used to acquire scientific and technological literature data, preprocess the scientific and technological literature data, and construct a candidate term set; the multidimensional analysis module 20 is used to traverse the candidate term set to perform multidimensional analysis and extract a multidimensional feature set, which includes syntactic and semantic features, statistical features, and domain knowledge features; the weighted calculation module 30 is used to perform dynamic weighted calculation based on the syntactic and semantic features, the statistical features, and the domain knowledge features to obtain a comprehensive score; and the filtering module 40 is used to backtrack to the candidate term set based on the comprehensive score to filter and generate term training corpus.

[0067] Furthermore, the candidate term set construction module 10 is used to perform the following steps: Acquire scientific and technological literature data, perform quality screening on the scientific and technological literature data, and construct a basic dataset; extract text data based on the basic dataset, traverse the text data to perform text cleaning, and generate standard text data, which includes title data, abstract data, and keyword data; label the title data, abstract data, and keyword data according to multiple data sources to determine multiple candidate terms; construct an authoritative terminology database to merge and deduplicate the multiple candidate terms, and construct the candidate terminology set.

[0068] Furthermore, the multidimensional analysis module 20 is used to perform the following operation steps: Syntactic analysis is performed on the candidate term set to extract syntactic position features. Based on these features, it is determined whether the candidate term set appears in the subject, object, or predicate position, and a syntactic position feature score is calculated. Lexical structure features are extracted from the candidate term set, and part-of-speech tagging is performed based on these features. A lexical structure feature score is calculated based on the lexical structure features. Step analysis is performed on the candidate term set to extract step type features. Based on these features, it is determined whether the candidate term set appears in the research question sentence, research object sentence, or research background sentence, and a step type feature score is calculated. The syntactic position feature score, lexical structure feature score, and step type feature score are weighted and fused to generate the syntactic-semantic features.

[0069] Furthermore, the multidimensional analysis module 20 is used to perform the following operation steps: Based on the part-of-speech tagging (POS) results, multiple POS combinations are determined, including double noun combinations and noun-adjective combinations. When the POS result is a double noun combination or a noun-adjective combination, a first sub-score is generated; when the POS result is not a double noun combination or a noun-adjective combination, a second sub-score is generated, wherein the first sub-score is greater than the second sub-score. Based on the candidate term set, POS tagging is performed; when the recognition result is a noun phrase, a third sub-score is generated; when the recognition result is not a noun phrase, a fourth sub-score is generated, wherein the third sub-score is greater than the fourth sub-score. Based on the candidate term set, segment length is identified, and a length sub-score is calculated. The first or second sub-score, the third or fourth sub-score, and the length sub-score are weighted and summed to generate the lexical structure feature score.

[0070] Furthermore, the candidate term set construction module 10 is used to perform the following steps: Based on the multiple data sources, an authority score is generated to produce a source authority score. Terminology matching is performed according to the source authority score to construct an authoritative domain thesaurus. The candidate term set is matched against the authoritative domain thesaurus, and a thesaurus matching score is calculated. Based on the thesaurus matching score, it is determined whether the candidate term set originates from the authoritative domain thesaurus. When the thesaurus matching score is greater than or equal to a preset thesaurus matching threshold, the candidate term set is determined to originate from the authoritative domain thesaurus, and a score-adding operation is performed on the candidate term set to generate domain knowledge features. When the thesaurus matching score is less than the preset thesaurus matching threshold, the candidate term set is determined not to originate from the authoritative domain thesaurus, and a score-subtracting operation is performed on the candidate term set to generate domain knowledge features.

[0071] Furthermore, the weighted calculation module 30 is used to perform the following operation steps: Based on the syntactic and semantic features, the statistical features, and the domain knowledge features, score allocation calculations are performed to determine the syntactic and semantic feature scores, statistical feature scores, and domain knowledge feature scores. A corpus size parameter is defined to perform dynamic weight analysis on the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores, generating multiple dynamic weight coefficients. These multiple dynamic weight coefficients include a first weight coefficient, a second weight coefficient, and a third weight coefficient, wherein the first weight coefficient, the second weight coefficient, and the third weight coefficient correspond to the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores. The syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores are weighted and fused according to the first weight coefficient, the second weight coefficient, and the third weight coefficient to calculate a comprehensive score.

[0072] Furthermore, the weighted calculation module 30 is used to perform the following operation steps: Based on the corpus size parameters, a target domain is determined for term formation analysis to generate a term formation regularity index; a term distribution analysis is performed based on the target domain to generate a term distribution significance index; when the term formation regularity index is higher than a first index threshold, the first weight coefficient is increased, and the second and third weight coefficients are decreased to generate a first dynamic weight coefficient; when the term distribution significance index is higher than a second index threshold, the second weight coefficient is increased, and the first and third weight coefficients are decreased to generate a second dynamic weight coefficient; when the corpus size is less than a preset size threshold, the third weight coefficient is increased, and the first and second weight coefficients are decreased to generate a third dynamic weight coefficient.

[0073] Furthermore, the filtering module 40 is used to perform the following operation steps: A hierarchical decision threshold is preset, including a first threshold and a second threshold, wherein the first threshold is greater than the second threshold. A threshold comparison result is generated by comparing the comprehensive score with the hierarchical decision threshold. The candidate terminology set is divided into a directly adopted subset, a manually reviewed subset, and an automatically filtered subset according to the threshold comparison result. The directly adopted subset is included in the terminology training corpus. The manually reviewed subset is transmitted to a manually reviewed unit for review and confirmation. Based on the confirmation result, the manually reviewed subset is included in the terminology training corpus. The automatically filtered subset is removed and not included in the terminology training corpus, thus generating the terminology training corpus.

[0074] Furthermore, the filtering module 40 is used to perform the following operation steps: Traverse the set of candidate terms and compare them with the hierarchical decision threshold: S1: When the overall score of the candidate term set is greater than or equal to the first threshold, extract the first candidate term and assign it to the direct adoption subset; S2: When the overall score of the candidate term set is less than the first threshold but greater than or equal to the second threshold, extract the second candidate term and assign it to the manual review subset; S3: When the overall score of the candidate term set is less than the second threshold, extract the third candidate term and assign it to the automatic filtering subset.

[0075] Through the foregoing detailed description of the terminology corpus generation method based on in-depth mining of document content, those skilled in the art can clearly understand the terminology corpus generation system based on in-depth mining of document content in this embodiment. Since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and relevant parts can be referred to the method section.

[0076] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for generating terminology corpus based on in-depth mining of document content, characterized in that, The method includes: Acquire scientific and technological literature data, preprocess the scientific and technological literature data, and construct a candidate term set; The candidate term set is traversed for multidimensional analysis to extract a multidimensional feature set, which includes syntactic and semantic features, statistical features, and domain knowledge features. The process of constructing syntactic and semantic features includes: Syntactic analysis is performed on the candidate term set to extract syntactic position features. Based on the syntactic position features, it is determined whether the candidate term set appears in the subject, object, or predicate position, and the syntactic position feature score is calculated. Lexical structure features are extracted from the candidate term set, part-of-speech tagging is performed based on the lexical structure features, and lexical structure feature scores are calculated based on the part-of-speech tagging results. Based on the candidate term set, step analysis is performed to extract step type features. Based on the step type features, it is determined whether the candidate term set appears in the research question sentence, research object sentence, or research background sentence, and the step type feature score is calculated. The syntactic position feature score, the lexical structure feature score, and the step type feature score are weighted and fused to generate the syntactic semantic feature; The calculation of lexical structure feature scores based on part-of-speech recognition results includes: Based on the part-of-speech recognition results, multiple part-of-speech combinations are determined, which may include double noun combinations and noun-adjective combinations. When the part-of-speech recognition result is the double noun combination or the noun-adjective combination, a first sub-score is generated; when the part-of-speech recognition result is not the double noun combination or the noun-adjective combination, a second sub-score is generated, wherein the first sub-score is greater than the second sub-score. Part-of-speech tagging is performed based on the candidate term set. When the tagging result is a noun phrase, a third sub-score is generated. When the tagging result is not a noun phrase, a fourth sub-score is generated. The third sub-score is greater than the fourth sub-score. Based on the candidate term set, segment length identification is performed, and length sub-scores are calculated. The first sub-score or the second sub-score, the third sub-score or the fourth sub-score, and the length sub-score are weighted and summed to generate the lexical structure feature score; The process of constructing domain knowledge features includes: An authoritative score is generated based on multiple data sources. Term matching is performed based on the source authority score to construct an authoritative domain thesaurus; The candidate term set is matched against the authoritative domain thesaurus, and the thesaurus matching score is calculated. The candidate term set is determined based on the word list matching score to determine whether it originates from the authoritative domain word list; When the word matching score is greater than or equal to the preset word matching threshold, it is determined that the candidate term set comes from an authoritative domain term database, and a scoring operation is performed on the candidate term set to generate domain knowledge features. When the word matching score is less than the preset word matching threshold, it is determined that the candidate term set does not come from the authoritative domain term library, and a deduction operation is performed on the candidate term set to generate domain knowledge features. A comprehensive score is obtained by dynamically weighting the syntactic and semantic features, the statistical features, and the domain knowledge features. Based on the comprehensive score, backtracking to the candidate term set for filtering, and generating term training corpus; The process involves backtracking to the candidate term set based on the comprehensive score to generate term training corpus, including: A preset hierarchical decision threshold is defined, which includes a first threshold and a second threshold, wherein the first threshold is greater than the second threshold. Based on the comparison between the comprehensive score and the hierarchical decision threshold, a threshold comparison result is generated, and the candidate term set is divided into a direct adoption subset, a manual review subset, and an automatic filtering subset according to the threshold comparison result; The directly adopted subset is included in the terminology training corpus, the manually reviewed subset is transmitted to the manual review unit for review and confirmation, the manually reviewed subset is included in the terminology training corpus according to the confirmation result, the automatically filtered subset is removed and not included in the terminology training corpus, and the terminology training corpus is generated.

2. The terminology corpus generation method based on in-depth document content mining as described in claim 1, characterized in that, The method includes acquiring scientific and technological literature data, preprocessing the data, and constructing a candidate term set. Acquire scientific and technological literature data, perform quality screening on the scientific and technological literature data, and construct a basic dataset; Based on the aforementioned basic dataset, text data is extracted, and the text data is traversed to perform text cleaning, generating standard text data, which includes title data, abstract data, and keyword data. The title data, the abstract data, and the keyword data are labeled according to multiple data sources to determine multiple candidate terms; An authoritative terminology database is constructed to merge and deduplicate the multiple candidate terms, thereby constructing the candidate terminology set.

3. The terminology corpus generation method based on in-depth document content mining as described in claim 1, characterized in that, A comprehensive score is obtained by dynamically weighting the syntactic and semantic features, the statistical features, and the domain knowledge features. The method includes: The scores are calculated based on the syntactic and semantic features, the statistical features, and the domain knowledge features to determine the scores for the syntactic and semantic features, the statistical features, and the domain knowledge features. By defining the corpus size parameters, dynamic weight analysis is performed on the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores to generate multiple dynamic weight coefficients. These multiple dynamic weight coefficients include a first weight coefficient, a second weight coefficient, and a third weight coefficient. The first weight coefficient, the second weight coefficient, and the third weight coefficient correspond to the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores. The syntactic and semantic feature scores, statistical feature scores, and domain knowledge feature scores are weighted and fused based on the first weighting coefficient, the second weighting coefficient, and the third weighting coefficient to calculate a comprehensive score.

4. The terminology corpus generation method based on in-depth document content mining as described in claim 3, characterized in that, The method involves defining corpus size parameters and performing dynamic weight analysis on the syntactic and semantic feature scores, the statistical feature scores, and the domain knowledge feature scores to generate multiple dynamic weight coefficients. Based on the corpus size parameters, the target domain is determined, term formation analysis is performed, and term formation rule indicators are generated. Analyze the term distribution based on the target domain and generate a significant index of term distribution. When the term formation regularity index is higher than the first index threshold, the first weight coefficient is increased, the second weight coefficient and the third weight coefficient are decreased to generate the first dynamic weight coefficient. When the significance index of the term distribution is higher than the second index threshold, the second weight coefficient is increased, the first weight coefficient and the third weight coefficient are decreased to generate a second dynamic weight coefficient. When the corpus size is less than a preset size threshold, the third weight coefficient is increased, and the first weight coefficient and the second weight coefficient are decreased to generate a third dynamic weight coefficient.

5. The terminology corpus generation method based on in-depth document content mining as described in claim 1, characterized in that, Based on a comparison between the comprehensive score and the hierarchical decision threshold, a threshold comparison result is generated. The candidate term set is then divided into a direct adoption subset, a manually reviewed subset, and an automatically filtered subset according to the threshold comparison result. The method includes: Traverse the set of candidate terms and compare them with the hierarchical decision threshold: S1: When the comprehensive score of the candidate term set is greater than or equal to the first threshold, extract the first candidate term and classify the first candidate term into the direct adoption subset; S2: When the comprehensive score of the candidate term set is less than the first threshold and greater than or equal to the second threshold, extract the second candidate term and classify the second candidate term into the manual review subset; S3: When the overall score of the candidate term set is less than the second threshold, extract the third candidate term and classify the third candidate term into the automatic filtering subset.

6. A terminology corpus generation system based on in-depth document content mining, characterized in that, The system is used to implement the terminology corpus generation method based on deep mining of document content as described in any one of claims 1-5, the system comprising: The candidate term set construction module is used to acquire scientific and technological literature data, preprocess the scientific and technological literature data, and construct a candidate term set. The multidimensional analysis module is used to traverse the candidate term set to perform multidimensional analysis and extract a multidimensional feature set, which includes syntactic and semantic features, statistical features, and domain knowledge features. The process of constructing syntactic and semantic features includes: Syntactic analysis is performed on the candidate term set to extract syntactic position features. Based on the syntactic position features, it is determined whether the candidate term set appears in the subject, object, or predicate position, and the syntactic position feature score is calculated. Lexical structure features are extracted from the candidate term set, part-of-speech tagging is performed based on the lexical structure features, and lexical structure feature scores are calculated based on the part-of-speech tagging results. Based on the candidate term set, step analysis is performed to extract step type features. Based on the step type features, it is determined whether the candidate term set appears in the research question sentence, research object sentence, or research background sentence, and the step type feature score is calculated. The syntactic position feature score, the lexical structure feature score, and the step type feature score are weighted and fused to generate the syntactic semantic feature; The calculation of lexical structure feature scores based on part-of-speech recognition results includes: Based on the part-of-speech recognition results, multiple part-of-speech combinations are determined, which may include double noun combinations and noun-adjective combinations. When the part-of-speech recognition result is the double noun combination or the noun-adjective combination, a first sub-score is generated; when the part-of-speech recognition result is not the double noun combination or the noun-adjective combination, a second sub-score is generated, wherein the first sub-score is greater than the second sub-score. Part-of-speech tagging is performed based on the candidate term set. When the tagging result is a noun phrase, a third sub-score is generated. When the tagging result is not a noun phrase, a fourth sub-score is generated. The third sub-score is greater than the fourth sub-score. Based on the candidate term set, segment length identification is performed, and length sub-scores are calculated. The first sub-score or the second sub-score, the third sub-score or the fourth sub-score, and the length sub-score are weighted and summed to generate the lexical structure feature score; The process of constructing domain knowledge features includes: An authoritative score is generated based on multiple data sources. Term matching is performed based on the source authority score to construct an authoritative domain thesaurus; The candidate term set is matched against the authoritative domain thesaurus, and the thesaurus matching score is calculated. The candidate term set is determined based on the word list matching score to determine whether it originates from the authoritative domain word list; When the word matching score is greater than or equal to the preset word matching threshold, it is determined that the candidate term set comes from an authoritative domain term database, and a scoring operation is performed on the candidate term set to generate domain knowledge features. When the word matching score is less than the preset word matching threshold, it is determined that the candidate term set does not come from the authoritative domain term library, and a deduction operation is performed on the candidate term set to generate domain knowledge features. The weighted calculation module is used to perform dynamic weighted calculation based on the syntactic and semantic features, the statistical features, and the domain knowledge features to obtain a comprehensive score. The filtering module is used to filter the candidate term set back from the comprehensive score and generate term training corpus. The process involves backtracking to the candidate term set based on the comprehensive score to generate term training corpus, including: A preset hierarchical decision threshold is defined, which includes a first threshold and a second threshold, wherein the first threshold is greater than the second threshold. Based on the comparison between the comprehensive score and the hierarchical decision threshold, a threshold comparison result is generated, and the candidate term set is divided into a direct adoption subset, a manual review subset, and an automatic filtering subset according to the threshold comparison result; The directly adopted subset is included in the terminology training corpus, the manually reviewed subset is transmitted to the manual review unit for review and confirmation, the manually reviewed subset is included in the terminology training corpus according to the confirmation result, the automatically filtered subset is removed and not included in the terminology training corpus, and the terminology training corpus is generated.

Images