Method, device, medium and program product for constructing a professional lexicon

Abstract

The application provides a professional vocabulary construction method and device, medium and program product. Professional corpus and general corpus are obtained, and word extraction processing is performed on the professional corpus and general corpus to obtain an initial vocabulary and a general vocabulary. The professional part-of-speech association degree of each candidate word in the initial vocabulary is determined according to the general vocabulary, and a seed word set is extracted from the initial vocabulary according to the professional part-of-speech association degree. In the initial vocabulary, each seed word in the seed word set is taken as a seed point, and the initial vocabulary is clustered based on the seed point corresponding to each seed word and a preset clustering algorithm to obtain a candidate word set. The professional vocabulary is determined according to a preset word selection condition and the candidate word set. The technical problem of how to quickly construct a professional vocabulary without data accumulation or artificial annotation is solved. The technical effect of not needing artificial annotation, improving the construction efficiency and word selection accuracy, and reducing the construction cost is achieved.

Description

Technical Field

[0001] This application relates to the field of natural language processing, and in particular to a method, apparatus, medium, and program product for constructing a professional lexicon. Background Technology

[0002] Words are the foundation of natural language processing, possessing a richer expressive power in semantic representation than characters. Words also exhibit specialized characteristics; different professions have different vocabulary systems, with significant differences existing between fields such as military, medical, public security, and finance. Different specialized thesauruses can support different application scenarios.

[0003] Currently, the primary challenge in developing professional text services is the need to invest significant manpower in manual annotation to create professional lexicons, or to manually annotate a certain amount of data and then use deep learning or machine learning to train a professional lexicon building model to expand the vocabulary of the professional lexicon.

[0004] In many scenarios, manually labeling and accumulating words is costly and inefficient. This makes cold start a critical technical problem that needs to be solved: how to quickly build a professional thesaurus without data accumulation or manually labeled data. Summary of the Invention

[0005] This application provides a method, apparatus, medium, and program product for constructing a professional thesaurus, in order to solve the technical problem of how to quickly construct a professional thesaurus when there is no data accumulation or no manually annotated data.

[0006] Firstly, this application provides a method for constructing a professional thesaurus, including:

[0007] Acquire specialized and general corpora, perform word extraction processing on the specialized and general corpora to obtain an initial lexicon and a general lexicon;

[0008] The degree of professional part-of-speech association of each candidate word in the initial lexicon is determined based on the general lexicon, and a seed word set is extracted from the initial lexicon based on the degree of professional part-of-speech association. Each seed word in the seed word set is a candidate word.

[0009] In the initial vocabulary, each seed word in the seed word set is used as a seed point. Based on the seed point corresponding to each seed word and the preset clustering algorithm, the initial vocabulary is clustered to obtain the candidate word set.

[0010] A professional thesaurus is determined based on preset word selection criteria and a set of words to be selected.

[0011] In one possible design, the seed words in the seed word set are words from the initial vocabulary and not words from the general vocabulary; and / or,

[0012] The probability of retrieving a seed word in the initial thesaurus is greater than the probability of retrieving a seed word in the general thesaurus.

[0013] In one possible design, the degree of professional part-of-speech association for each candidate word in the initial lexicon is determined based on a general lexicon, and a seed word set is extracted from the initial lexicon based on the degree of professional part-of-speech association, including:

[0014] The degree of professional part-of-speech relevance of the candidate words is determined based on the first total word count, the second total word count, the first number of candidate words in the initial lexicon, and the second number of candidate words in the general lexicon. The first total word count is the total number of words in the initial lexicon, and the second total word count is the total number of words in the general lexicon.

[0015] If the degree of professional part-of-speech relevance is greater than the preset threshold, then the candidate word is determined as the seed word.

[0016] In one possible design, the degree of professional part-of-speech association of the candidate words is determined based on the first total word count, the second total word count, the first number of candidate words in the initial lexicon, and the second number of candidate words in the general lexicon, including:

[0017] Calculate the first product of the first total word count and the second quantity;

[0018] Calculate the second product of the second total word count and the first word count;

[0019] The degree of professional part-of-speech relevance includes the quotient of the second product and the first product.

[0020] In one possible design, in the initial vocabulary, each seed word in the seed word set is used as a seed point. Based on the seed point corresponding to each seed word and a preset clustering algorithm, the initial vocabulary is clustered to obtain a candidate word set, including:

[0021] Calculate the semantic distance between each candidate word and each seed word;

[0022] If the semantic distance is less than the preset distance threshold, the candidate word and the corresponding seed word are combined to form a first candidate word set. Each seed word is given a first candidate word set, and the candidate word set includes multiple first candidate word sets.

[0023] In one possible design, after adding the candidate word to the first candidate word set corresponding to the seed word, it also includes:

[0024] Based on the semantic distance, the first K first candidate words in the first candidate word set corresponding to each seed word are taken as the second candidate words. The semantic distance between the first K first candidate words and the corresponding seed words is less than the semantic distance between the other candidate words and the seed words.

[0025] The K second candidate words are combined into a second candidate word set. The candidate word set also includes multiple second candidate word sets, that is, the second candidate word set corresponding to each seed word, where K is a positive integer.

[0026] In one possible design, after combining the K second candidate words into a second candidate word set, it also includes:

[0027] Based on the number of words contained in each first candidate word set, select M first candidate word sets and combine them into a third candidate word set;

[0028] Among them, the number of words in each of the M first candidate word sets is greater than the number of words in the first candidate word sets other than the M first candidate word sets; the candidate word sets include the third candidate word set, and M is a positive integer.

[0029] The purpose of the third candidate word set is to avoid excessive elimination from the second candidate word set and to replenish some words.

[0030] In one possible design, after selecting M first candidate word sets based on the number of words contained in each first candidate word set and forming a third candidate word set, the following steps are also included:

[0031] The seed word set, each of the second candidate word sets, and the third candidate word set are combined into a candidate word set, and duplicate words in the candidate word set are removed.

[0032] Optionally, the preset word selection conditions include at least one of word frequency conditions and part-of-speech conditions, wherein the word frequency threshold in the word frequency conditions corresponds to the size of the professional corpus, and the part-of-speech conditions correspond to the professional fields in the professional corpus.

[0033] Secondly, this application provides a professional thesaurus automatic construction device, comprising:

[0034] The acquisition module is used to acquire specialized and general corpora.

[0035] Processing module, used for:

[0036] Word extraction processing is performed on specialized and general corpora to obtain an initial lexicon and a general lexicon;

[0037] The degree of professional part-of-speech association of each candidate word in the initial lexicon is determined based on the general lexicon, and a seed word set is extracted from the initial lexicon based on the degree of professional part-of-speech association. Each seed word in the seed word set is a candidate word.

[0038] In the initial lexicon, each seed word in the seed word set is used as a seed point. Based on the seed point corresponding to each seed word and the preset clustering algorithm, the initial lexicon is semantically clustered to obtain the candidate word set. The professional lexicon is determined according to the preset word selection conditions and the candidate word set.

[0039] In one possible design, the seed words in the seed word set are words from the initial vocabulary and not words from the general vocabulary; and / or,

[0040] The probability of retrieving a seed word in the initial thesaurus is greater than the probability of retrieving a seed word in the general thesaurus.

[0041] In one possible design, the processing module is used for:

[0042] The degree of professional part-of-speech relevance of the candidate words is determined based on the first total word count, the second total word count, the first number of candidate words in the initial lexicon, and the second number of candidate words in the general lexicon. The first total word count is the total number of words in the initial lexicon, and the second total word count is the total number of words in the general lexicon.

[0043] If the degree of professional part-of-speech relevance is greater than the preset threshold, then the candidate word is determined as the seed word.

[0044] In one possible design, the processing module is used for:

[0045] Calculate the first product of the first total word count and the second quantity;

[0046] Calculate the second product of the second total word count and the first word count;

[0047] The degree of professional part-of-speech relevance includes the quotient of the second product and the first product.

[0048] In one possible design, the processing module is used for:

[0049] Calculate the semantic distance between each candidate word and each seed word;

[0050] If the semantic distance is less than a preset distance threshold, the candidate word and the corresponding seed word are combined to form a first candidate word set. Each seed word is given a first candidate word set, and the candidate word set includes multiple first candidate word sets.

[0051] In one possible design, the processing module is also used for:

[0052] Based on the semantic distance, the first K first candidate words in the first candidate word set corresponding to each seed word are taken as the second candidate words. The semantic distance between the first K first candidate words and the corresponding seed words is less than the semantic distance between the other candidate words and the seed words.

[0053] The K second candidate words are combined into a second candidate word set. The candidate word set also includes multiple second candidate word sets, that is, the second candidate word set corresponding to each seed word, where K is a positive integer.

[0054] In one possible design, the processing module is also used for:

[0055] Based on the number of words contained in each first candidate word set, select M first candidate word sets and combine them into a third candidate word set;

[0056] Among them, the number of words in each of the M first candidate word sets is greater than the number of words in the first candidate word sets other than the M first candidate word sets; the candidate word sets include the third candidate word set, and M is a positive integer.

[0057] The purpose of the third candidate word set is to avoid excessive elimination from the second candidate word set and to replenish some words.

[0058] In one possible design, the processing module is also used for:

[0059] Based on the semantic distance and the first sorting order, each first candidate word in each first candidate word set is sorted, and the first sorting order includes sorting from smallest to largest.

[0060] The first candidate word in the top K positions is used as the second candidate word.

[0061] In one possible design, the processing module is also used for:

[0062] Based on the number of words contained in each first candidate word set and the second sorting order, the first candidate word sets are sorted. The second sorting order includes sorting from largest to smallest.

[0063] The first set of candidate words ranked in the top M positions is merged into the third set of candidate words, and the candidate word set includes the third set of candidate words.

[0064] In one possible design, the processing module is also used for:

[0065] The seed word set, each of the second candidate word sets, and the third candidate word set are combined into a candidate word set, and duplicate words in the candidate word set are removed.

[0066] Optionally, the preset word selection conditions include at least one of word frequency conditions and part-of-speech conditions, wherein the word frequency threshold in the word frequency conditions corresponds to the size of the professional corpus, and the part-of-speech conditions correspond to the professional fields in the professional corpus.

[0067] Thirdly, this application provides an electronic device, comprising:

[0068] Memory, used to store program instructions;

[0069] The processor is configured to invoke and execute program instructions in the memory to perform any of the possible methods for determining item storage information provided in the first aspect.

[0070] Fourthly, this application provides a storage medium storing a computer program for executing any of the possible professional thesaurus construction methods provided in the first aspect.

[0071] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the possible methods for constructing a specialized thesaurus provided in the first aspect.

[0072] This application provides a method, apparatus, medium, and program product for constructing a professional lexicon. It involves acquiring professional and general corpora, performing word extraction processing on the professional and general corpora to obtain an initial lexicon and a general lexicon; determining the degree of professional part-of-speech association for each candidate word in the initial lexicon based on the general lexicon, and extracting a seed word set from the initial lexicon based on the degree of professional part-of-speech association, where each seed word in the seed word set is a candidate word; clustering the initial lexicon using each seed word in the seed word set as a seed point and a preset clustering algorithm to obtain a candidate word set; and determining the professional lexicon based on preset word selection conditions and the candidate word set. This application automatically extracts professional vocabulary from professional and general corpora, eliminating the need for extensive data accumulation or manually annotated data. It automatically filters vocabulary based on the degree of professional part-of-speech association, thus enabling the rapid construction of a professional lexicon. It achieves the technical effect of quickly building a professional thesaurus without manual annotation, improving construction efficiency and word selection accuracy, and reducing construction costs. Attached Figure Description

[0073] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0074] Figure 1 This is a flowchart illustrating a method for constructing a specialized dictionary based on deep learning in conventional techniques.

[0075] Figure 2 A flowchart illustrating a method for constructing a professional thesaurus provided in this application;

[0076] Figure 3 A flowchart illustrating another professional thesaurus construction method provided for the implementation of this application;

[0077] Figure 4A schematic diagram of a professional thesaurus construction device provided in this application embodiment;

[0078] Figure 5 This is a schematic diagram of the structure of an electronic device provided in this application.

[0079] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0080] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort, including but not limited to combinations of multiple embodiments, are within the scope of protection of this application.

[0081] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0082] The following is an explanation of the technical terms used in this application:

[0083] 1. Words: Words are crucial linguistic units in a language system and the foundation of text processing. They possess richer expressive power than characters in semantic representation. Words exhibit specialized characteristics; different professions have different vocabulary systems. For example, there are significant differences between military, medical, public security, and financial professions. This specialized nature of words can be used for professional classification, labeling, differentiation, keyword extraction, and feature word extraction. It can also support the construction of knowledge graph systems such as ontology and instance systems. Furthermore, it has led to tasks such as extracting professional vocabulary, extracting semantic relationships between professional words, expanding professional vocabulary, and constructing professional vocabulary knowledge bases. Related professional vocabulary mining technology is a key technology in natural language processing and an essential capability for supporting business implementation and application. Different professional lexicons can support different applications. Professional lexicons can be subdivided into six types: professional feature lexicons, professional stop word lexicons, professional sentiment lexicons, professional abstract lexicons, professional semantic lexicons, and professional event lexicons.

[0084] 2. BERT (Bidirectional Encoder Representations from Transformers): A pre-trained language model that uses bidirectional encoder representations developed using the Transformer framework. Pre-trained BERT means that the model can be fine-tuned through an additional output layer, making it widely applicable for building state-of-the-art models.

[0085] 3. LSTM (Long Short-Term Memory): This is a type of recurrent neural network designed to address the long-term dependency problem inherent in general recurrent neural networks, and it can perform better on longer sequences.

[0086] Currently, the primary challenge in developing professional text processing services is the cold start problem, namely, how to quickly build a professional thesaurus without any existing data. This is a technical issue that professional text processing services must address.

[0087] Currently, professional lexicons can be constructed using deep learning-based methods. This method requires a certain amount of labeled data to be accumulated in the professional lexicon. Through modules such as data labeling, model training, professional vocabulary extraction, and filtering, the professional lexicon can be expanded.

[0088] Figure 1 This is a flowchart illustrating a method for constructing a specialized dictionary based on deep learning, as described in related technologies. Figure 1 As shown, the specific steps of this method include:

[0089] S101. Data annotation. Manually annotating texts in a specialized field using a certain amount of accumulated professional vocabulary.

[0090] S102. Model Training. Generally, entity recognition tasks are used to train the model, and models such as LSTM and BERT can be selected.

[0091] S103. Extract information from the data to be identified. That is, identify the technical terms in the data to be identified.

[0092] S104. Vocabulary Filtering. This involves filtering the specialized vocabulary from step three using methods such as confidence level and word frequency.

[0093] S105. Add the selected professional terms to the professional terminology database to expand the professional terminology database.

[0094] However, the inventors of this application have discovered the following technical problems with the above-mentioned method for constructing specialized dictionaries based on deep learning:

[0095] 1. A certain level of specialized vocabulary is required;

[0096] 2. Manual annotation by domain experts is required;

[0097] 3. The model must have high recognition accuracy; otherwise, it will pollute the professional lexicon.

[0098] Among other related technologies for building professional lexicons, rule-based methods have high accuracy and strong targeting, but the rules are generally professionally specific and lack transferability and universality. Moreover, the complexity of manually writing and maintaining rules is relatively high, and they are more often used as an additional module combined with other methods. Statistical methods are more flexible, but they require a certain accumulation of professional vocabulary, manual annotation by domain experts, and high recognition accuracy of the model; otherwise, they will pollute the professional lexicon.

[0099] To solve the above problems, the inventive concept of this application is as follows:

[0100] This paper proposes a method and apparatus for automatically constructing a specialized thesaurus. Through new word discovery, seed word selection, semantic computation, and word frequency + part-of-speech tagging, the automated construction of a domain-specific dictionary is achieved, offering advantages such as high accuracy, simple structure, and high efficiency. It can save significant manpower and resources, enabling the construction of a relevant domain-specific dictionary at low cost. The seed word selection and semantic clustering computation in this application can improve the quality of the domain-specific thesaurus; a high-quality domain-specific dictionary can be obtained without manual annotation or specialized knowledge; and it possesses good transferability, universality, and scalability.

[0101] This professional thesaurus construction method can be applied to various scenarios and systems of language and text processing, scenarios and systems of professional thesaurus construction and management, etc. The main entities executing this professional thesaurus construction method include processors, servers, etc., used for language and text processing.

[0102] The following details the professional thesaurus construction method provided in this application:

[0103] Figure 2 This is a flowchart illustrating a method for constructing a professional thesaurus as provided in an embodiment of this application. Figure 2 As shown, the specific steps include:

[0104] S201. Obtain specialized and general corpora, perform word extraction processing on the specialized and general corpora to obtain the initial lexicon and the general lexicon.

[0105] In this step, the same word may appear multiple times in both the initial lexicon and the general lexicon. The specialized corpus includes semantic data from one or more specialized fields.

[0106] In this embodiment, the word extraction process includes: new word discovery and stop word removal.

[0107] New word discovery: Users import specialized and general corpora from the internet, databases, or big data centers. Then, new word discovery tools (such as those based on mutual information and / or left / right entropy) are used to discover new words in both the specialized and general corpora. The results are then stored in corresponding databases: the specialized corpora are stored in the first database, and the general corpora are stored in the second database.

[0108] It should be noted that specialized corpora include text collections from at least one specialized field, such as economics, finance, military, medicine, semiconductors, computer science, etc., containing a large number of specialized terms. General corpora include: publicly available text collections, such as press releases, public datasets, online media data, etc.

[0109] Stop word removal: The stop word database is used to filter all words in the new word discovery results, i.e., the first and second databases. After filtering, an initial lexicon and a general lexicon are obtained. Its functions include: first, reducing the number of invalid words and improving the overall speed of the device; second, improving the accuracy of semantic calculation in the following text, i.e., semantic clustering in S203.

[0110] S202. Determine the degree of professional part-of-speech association of each candidate word in the initial lexicon based on the general lexicon, and extract a seed word set from the initial lexicon based on the degree of professional part-of-speech association.

[0111] In this step, the degree of professional part-of-speech relevance refers to the degree to which a word is specific to a certain field. For example, a professional term for a certain field has a high degree of professional part-of-speech relevance in that field. Conversely, if a word can be applied to various fields, then this word is called a general term, and its degree of professional part-of-speech relevance in a certain field is low.

[0112] The selection criteria for seed words in the seed word set include: the seed words in the seed word set are words from the initial vocabulary and not words from the general vocabulary; and / or,

[0113] The probability of retrieving a seed word in the initial thesaurus is greater than the probability of retrieving a seed word in the general thesaurus.

[0114] It should be noted that seed words possess special attributes specific to at least one professional field. They are rarely involved in other professional fields and have a low probability of appearing in general corpora. However, they are not completely impossible to find. They may be mentioned in some news reports or popular science texts, but the probability of their appearance is not high.

[0115] This application does not require manual annotation of seed words. Instead, it automatically searches for seed words by comparing the initial vocabulary and the general vocabulary, thereby saving a lot of manpower and greatly improving the efficiency of seed word set construction.

[0116] By comparing the words in the initial thesaurus and the general thesaurus, seed words for the initial thesaurus can be selected, thus identifying high-quality seed words and improving the quality of the final professional thesaurus.

[0117] In one possible implementation, the seed word can also be determined in the following way:

[0118] The degree of professional part-of-speech relevance of the candidate words is determined based on the first total word count, the second total word count, the first number of candidate words in the initial lexicon, and the second number of candidate words in the general lexicon. The first total word count is the total number of words in the initial lexicon, and the second total word count is the total number of words in the general lexicon.

[0119] If the degree of professional part-of-speech relevance is greater than the preset threshold, then the candidate word is determined as the seed word.

[0120] It should be noted that the specific principles and terminology of this implementation method can be found in [reference needed]. Figure 3 The embodiments shown are not described in detail here.

[0121] S203. In the initial lexicon, each seed word in the seed word set is used as a seed point. Based on the seed point corresponding to each seed word and the preset clustering algorithm, the initial lexicon is clustered to obtain the candidate word set. The professional lexicon is determined according to the preset word selection conditions and the candidate word set.

[0122] This step specifically includes:

[0123] Calculate the semantic distance between each candidate word and each seed word;

[0124] If the semantic distance is less than the preset distance threshold, the candidate word and the corresponding seed word are combined to form a first candidate word set. Each seed word is given a first candidate word set, and the candidate word set includes multiple first candidate word sets.

[0125] For example, each candidate word in the initial lexicon and each seed word in the seed word set are vectorized to determine each candidate word vector and each seed word vector; the semantic distance between each candidate word vector and each seed word vector is calculated, and it is determined whether the semantic distance is greater than a preset distance threshold; if so, the candidate word is added to the first candidate word set corresponding to the seed word.

[0126] It should be noted that the candidate word set includes multiple first candidate word sets. Semantic distance includes: Euclidean distance, cosine distance, etc.

[0127] In this embodiment, the number of seed words is equal to the number of words in the first candidate word set.

[0128] Optionally, after S202 is completed, the seed word set can be removed from the initial word set to avoid repeatedly including seed words in the first candidate word set, which would affect computational efficiency.

[0129] Optionally, the preset word selection conditions include at least one of word frequency conditions and part-of-speech conditions, wherein the word frequency threshold in the word frequency conditions corresponds to the size of the professional corpus, and the part-of-speech conditions correspond to the professional fields in the professional corpus.

[0130] For example, word frequency conditions include: adding a word to a professional vocabulary database when the frequency of its occurrence exceeds a preset frequency threshold.

[0131] The part-of-speech criteria include: if the semantics expressed by a word correspond to a professional field, then it is added to the professional lexicon.

[0132] The solution proposed in this embodiment can be adapted to other tasks or application scenarios through fine-tuning, and other excellent solutions can also be embedded into the implementation logic. Therefore, this application has good portability, versatility, and scalability. For example, based on this embodiment, by adjusting thresholds, quality detection of new word discovery can be performed to improve the quality of new word discovery.

[0133] This application provides a method for constructing a professional lexicon. It involves acquiring professional and general corpora, and then performing word extraction processing on each to determine an initial lexicon and a general lexicon, where the same word can appear multiple times in both. The method then determines the degree of professional part-of-speech association for each candidate word in the initial lexicon based on the general lexicon, and extracts a seed word set from the initial lexicon based on this association. Semantic clustering is then performed in the initial lexicon using each seed word in the seed word set as a cluster center to determine a candidate word set. Finally, the professional lexicon is determined based on preset word selection conditions and the candidate word set. This method solves the technical problem of how to quickly construct a professional lexicon without data accumulation or manually annotated data. It achieves the technical effect of quickly constructing a professional lexicon without manual annotation, improving construction efficiency and word selection accuracy, and reducing construction costs.

[0134] Figure 3 A flowchart illustrating another method for constructing a specialized thesaurus provided for the implementation of this application. (For example...) Figure 3 As shown, this professional thesaurus construction method can be applied to various scenarios and systems of language and text processing, scenarios and systems of professional thesaurus construction and management, etc. The specific steps include:

[0135] S301. Obtain specialized corpora and general corpora, and perform word extraction processing from the specialized corpora and general corpora respectively to determine the initial lexicon and the general lexicon.

[0136] In this step, the same word may appear multiple times in both the initial lexicon and the general lexicon. The specialized corpus includes semantic data from one or more specialized fields.

[0137] S302. Determine the degree of professional part-of-speech association of the candidate words based on the first total word count, the second total word count, the first number of candidate words in the initial lexicon, and the second number of candidate words in the general lexicon.

[0138] In this step, the first total word count is the total number of words in the initial lexicon, and the second total word count is the total number of words in the general lexicon. Specifically, the first product of the first total word count and the second total word count is calculated; the second product of the second total word count and the first total word count is calculated; the degree of professional part-of-speech relevance includes the quotient of the second product and the first product.

[0139] It should be noted that, in this embodiment, the degree of professional part-of-speech relevance of the candidate words can be divided into two cases:

[0140] The first type consists of words from the initial lexicon, but not words from the general lexicon.

[0141] Specifically, if a candidate word has a second quantity greater than zero, it is considered not to be the first seed word in this case, and proceeds to the second level of professional judgment, namely, whether it is a second seed word.

[0142] The set C1 consisting of all the first seed words can be calculated using formula (1):

[0143] C1=AA∩B (1)

[0144] Where A represents the initial lexicon and B represents the general lexicon.

[0145] The second approach is that the probability of retrieving a seed word in the initial thesaurus is greater than the probability of retrieving a seed word in the general thesaurus.

[0146] Specifically, in this case, the degree of professional part-of-speech relevance Z of a candidate word i i It can be expressed by formula (2):

[0147]

[0148] Among them, l A and l B These represent the first total word count and the second total word count, respectively. i and B i These represent the first quantity and the second quantity, respectively.

[0149] S303. If the degree of professional part-of-speech relevance is greater than the preset threshold, then the candidate word is determined as the seed word.

[0150] In this embodiment, the first seed word in C1 in step S302 has the highest degree of professional part-of-speech association. For the second seed word, a preset threshold α needs to be set. By comparing the degree of professional part-of-speech association Z with the preset threshold α, it can be determined whether the candidate word is a second seed word. The set C2 formed by the second seed words can be represented by formula (3):

[0151] C2={i∈A∩B|Z i >α} (3)

[0152] Finally, by finding the union of sets C1 and C2 and removing duplicate words, we can obtain the seed word set C.

[0153] Optionally, seed words from the seed word set C can be removed from the initial dictionary A to avoid affecting the accuracy and efficiency of subsequent semantic calculations.

[0154] Next, we will proceed with semantic computation, the specific computation methods and rules of which are as follows:

[0155] S304. Vectorize each candidate word in the initial lexicon and each seed word in the seed word set to determine the vectors of each candidate word and each seed word.

[0156] In this step, the words in the initial vocabulary A and the seed vocabulary C are vectorized. The vectorization methods include word2vec, glove, etc.

[0157] S305. Calculate the semantic distance between each candidate word vector and each seed word vector, and determine whether the semantic distance is less than the preset distance threshold.

[0158] In this step, if yes, then execute S306; otherwise, continue calculating the next semantic distance.

[0159] Optional semantic distances include: Euclidean distance, cosine distance, etc.

[0160] S306. Add the candidate word to the first candidate word set corresponding to the seed word.

[0161] In this embodiment, assuming that there are n seed words in the seed word set C, the seed word set C can be represented as {C1, C2, ..., C...} n}

[0162] For each seed word C in the seed word set C j (1≤j≤n):

[0163] First, extract the first candidate word A1 from the initial vocabulary A, and calculate C. j The semantic distance d(C) from A1 j A1). If d(C j If A1) < β, then add word A1 to the seed word C. j The first candidate word is selected from the centered vocabulary. Otherwise, the second candidate word A2 is taken from the initial vocabulary A, and C is calculated. j The semantic distance d(C) from A2 j (A2). It should be noted that β represents the first preset distance threshold.

[0164] For ease of explanation in the following calculations, we assume here that A1 satisfies the condition, then the seed word C is used. j The related word C appears in the central set. j1 Next, retrieve the second word A2 from the domain dictionary A. At this point, it is necessary to calculate the values ​​of the candidate word A2 and C. j C j1 semantic distance d(C) j ,A2) and d(C j1 (A2).

[0165] Then, determine the magnitudes of these two semantic distances relative to the first preset distance threshold β and the second preset distance threshold γ. If d(C j A2) < β and d(C j1 If A2) < γ, then add the candidate word A2 to the seed word C. j The first candidate set centered on the word A2 is denoted as C. j2 Otherwise, take the third candidate word A3 from the initial word library A and repeat the above calculation.

[0166] Similarly, each candidate word in the initial vocabulary A must satisfy the condition of C. j The semantic distance is less than the first preset distance threshold β, and is related to the seed word C. j Each word C in the central set jm Only when the semantic distance of all terms is less than the second preset distance threshold γ, is it sufficient to add the seed word C. j The process of iterating through the first candidate word set centered on the initial lexicon A continues until all words in A have been traversed. Then, it iterates through the first candidate word set corresponding to the next seed word, repeating this process until all first candidate word sets corresponding to all seed words are constructed. This semantic computation method and rule can improve the computational efficiency of the entire process and ensure the quality of the specialized lexicon.

[0167] S307. Based on the semantic distance, take the first K first candidate words in the first candidate word set corresponding to each seed word as the second candidate words, and combine the K second candidate words into a second candidate word set.

[0168] In this step, to further improve the quality of the professional terminology database, it is necessary to conduct further screening from another dimension, building upon the level of professionalism in S306. Therefore, a preset distance requirement different from the level of professionalism needs to be set.

[0169] In one possible implementation, based on the semantic distance, the top K candidate words in the first candidate word set corresponding to each seed word are selected as the second candidate words, specifically including:

[0170] Based on the semantic distance and the first sorting order, each first candidate word in each first candidate word set is sorted.

[0171] That is, based on the semantic distance between each first candidate word in each first candidate word set and its corresponding seed word, and the first sorting order, each first candidate word in each first candidate word set is sorted. The first sorting order includes sorting from smallest to largest.

[0172] The first candidate word in the top K positions is used as the second candidate word.

[0173] For example, consider a first candidate word set {C}m1 C m2 , ..., C mn The corresponding seed word is C. j Then, according to the semantic distance d(C) mi C j The size of the first candidate word set is used to arrange each first candidate word in ascending order, and the first k first candidate words are selected as the second candidate words, which are then combined to form the second candidate set.

[0174] Optionally, the first sorting order can also be to first sort from smallest to largest, then extract the odd-numbered or even-numbered positions, divide all odd-numbered and even-numbered positions into two columns, and then recombine these two columns into one column. This transforms the linear semantic distance dimension into a non-linear semantic distance dimension, making the selection of candidate words in the professional thesaurus more comprehensive and of higher quality. It is understood that those skilled in the art can design the first sorting order according to the actual scenario, and this application does not impose any limitations.

[0175] S308. Select M first candidate word sets according to the number of words contained in each first candidate word set and combine them into a third candidate word set.

[0176] In this step, the number of words in each of the M first candidate word sets is greater than the number of words in the first candidate word sets other than the M first candidate word sets.

[0177] Specifically, based on the number of words contained in each first candidate word set and the second sorting order, the first candidate word sets are sorted, and the top M first candidate word sets are combined into a third candidate word set. The second sorting order includes sorting from largest to smallest. To avoid the problem of excessive elimination in S305 to S307, it is necessary to replenish some candidate words, hence this step was designed.

[0178] Specifically, each first candidate word set is sorted from largest to smallest according to the total number of first candidate words contained in it, i.e., the number of words contained in the first candidate word set. Then, all the first candidate word sets ranked in the top M positions are extracted and merged into the third candidate word set.

[0179] It should be noted that the second sorting order can also be other sorting methods. For example, firstly, sort from largest to smallest, then extract multiple first candidate word sets at preset intervals, such as every 2 to 5 words, and then select M words from the remaining or extracted first candidate word sets as the third candidate word set. Those skilled in the art can design the second sorting order according to the actual scenario, and this application does not limit it.

[0180] S304 to S308 pertain to semantic computing. By designing novel semantic computing rules, potential specialized terms centered on seed words are selected. Further screening yields a second and a third candidate word set.

[0181] S309. Combine the seed word set, each second candidate word set, and the third candidate word set into a candidate word set, and remove duplicate words from the candidate word set.

[0182] S310. Determine the professional thesaurus based on the preset word selection conditions and the set of words to be selected.

[0183] In this step, the preset word selection conditions include at least one of word frequency conditions and part-of-speech conditions. The word frequency threshold in the word frequency conditions corresponds to the size of the professional corpus, and the part-of-speech conditions correspond to the professional fields in the professional corpus.

[0184] It should be noted that the word frequency and part-of-speech filtering section plays a significant role in improving the quality of the final professional thesaurus.

[0185] Specifically, this application uses a combination of word frequency and part-of-speech tagging for selection. Words in the candidate word set must simultaneously meet both frequency and part-of-speech criteria to be added to the specialized thesaurus. The word frequency threshold needs to be set based on the size of the domain corpus; the part-of-speech requirement depends on the domain of the corpus. For example, if the data in the corpus belongs to the financial category, it is generally required that the part of speech be a noun.

[0186] This application provides a method for constructing a professional thesaurus. Through new word discovery, seed word selection, semantic calculation, and word frequency + part-of-speech filtering, it automates the construction of a domain-specific dictionary, offering advantages such as high accuracy, simple structure, and high efficiency. It can save significant manpower and resources, enabling the construction of a relevant domain-specific dictionary at low cost. The seed word selection and semantic clustering calculations in this application can improve the quality of the domain-specific thesaurus; a high-quality domain-specific dictionary can be obtained without manual annotation or specialized knowledge; and it possesses good transferability, universality, and scalability.

[0187] Figure 4 This is a schematic diagram of a professional thesaurus construction device provided in an embodiment of this application. The professional thesaurus automatic construction device 400 can be implemented through software, hardware, or a combination of both.

[0188] like Figure 4 As shown, the specialized lexicon automatic construction device 400 includes:

[0189] Module 401 is used to acquire specialized and general corpora.

[0190] Processing module 402 is used for:

[0191] Word extraction processing is performed on specialized and general corpora to obtain an initial lexicon and a general lexicon;

[0192] The degree of professional part-of-speech association of each candidate word in the initial lexicon is determined based on the general lexicon, and a seed word set is extracted from the initial lexicon based on the degree of professional part-of-speech association. Each seed word in the seed word set is a candidate word.

[0193] In the initial lexicon, each seed word in the seed word set is used as a seed point. Based on the seed point corresponding to each seed word and the preset clustering algorithm, the initial lexicon is semantically clustered to obtain the candidate word set. The professional lexicon is determined according to the preset word selection conditions and the candidate word set.

[0194] In one possible design, the seed words in the seed word set are words from the initial vocabulary and not words from the general vocabulary; and / or,

[0195] The probability of retrieving a seed word in the initial thesaurus is greater than the probability of retrieving a seed word in the general thesaurus.

[0196] In one possible design, processing module 402 is used for:

[0197] The degree of professional part-of-speech relevance of the candidate words is determined based on the first total word count, the second total word count, the first number of candidate words in the initial lexicon, and the second number of candidate words in the general lexicon. The first total word count is the total number of words in the initial lexicon, and the second total word count is the total number of words in the general lexicon.

[0198] If the degree of professional part-of-speech relevance is greater than the preset threshold, then the candidate word is determined as the seed word.

[0199] In one possible design, processing module 402 is used for:

[0200] Calculate the first product of the first total word count and the second quantity;

[0201] Calculate the second product of the second total word count and the first word count;

[0202] The degree of professional part-of-speech relevance includes the quotient of the second product and the first product.

[0203] In one possible design, processing module 402 is used for:

[0204] Calculate the semantic distance between each candidate word and each seed word;

[0205] If the semantic distance is less than the preset distance threshold, the candidate word and the corresponding seed word are combined to form a first candidate word set. Each seed word is given a first candidate word set, and the candidate word set includes multiple first candidate word sets.

[0206] In one possible design, processing module 402 is also used for:

[0207] Based on the semantic distance, the first K first candidate words in the first candidate word set corresponding to each seed word are taken as the second candidate words. The semantic distance between the first K first candidate words and the corresponding seed words is less than the semantic distance between the other candidate words and the seed words.

[0208] The K second candidate words are combined into a second candidate word set. The candidate word set also includes multiple second candidate word sets, that is, the second candidate word set corresponding to each seed word, where K is a positive integer.

[0209] In one possible design, processing module 402 is also used for:

[0210] Based on the number of words contained in each first candidate word set, select M first candidate word sets and combine them into a third candidate word set;

[0211] Among them, the number of words in each of the M first candidate word sets is greater than the number of words in the first candidate word sets other than the M first candidate word sets; the candidate word sets include the third candidate word set, and M is a positive integer.

[0212] The purpose of the third candidate word set is to avoid excessive elimination from the second candidate word set and to replenish some words.

[0213] In one possible design, processing module 402 is also used for:

[0214] Based on the semantic distance and the first sorting order, each first candidate word in each first candidate word set is sorted, and the first sorting order includes sorting from smallest to largest.

[0215] The first candidate word in the top K positions is used as the second candidate word.

[0216] In one possible design, processing module 402 is also used for:

[0217] Based on the number of words contained in each first candidate word set and the second sorting order, the first candidate word sets are sorted. The second sorting order includes sorting from largest to smallest.

[0218] The first set of candidate words ranked in the top M positions is merged into the third set of candidate words, and the candidate word set includes the third set of candidate words.

[0219] In one possible design, processing module 402 is also used for:

[0220] The seed word set, each of the second candidate word sets, and the third candidate word set are combined into a candidate word set, and duplicate words in the candidate word set are removed.

[0221] Optionally, the preset word selection conditions include at least one of word frequency conditions and part-of-speech conditions, wherein the word frequency threshold in the word frequency conditions corresponds to the size of the professional corpus, and the part-of-speech conditions correspond to the professional fields in the professional corpus.

[0222] It is worth noting that, Figure 4 The apparatus provided in the illustrated embodiments can execute the methods provided in any of the above method embodiments. Their specific implementation principles, technical features, explanations of technical terms, and technical effects are similar and will not be repeated here.

[0223] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device 500 may include at least one processor 501 and a memory 502. Figure 5 The example shown is an electronic device using a processor.

[0224] The memory 502 is used to store programs. Specifically, the program may include program code, which includes computer operation instructions.

[0225] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0226] The processor 501 is used to execute computer execution instructions stored in the memory 502 to implement the methods described in the above embodiments.

[0227] The processor 501 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0228] Optionally, the memory 502 can be either standalone or integrated with the processor 501. When the memory 502 is a device independent of the processor 501, the electronic device 500 may further include:

[0229] Bus 503 is used to connect the processor 501 and the memory 502. The bus can be an industry standard architecture (ISA) bus, a peripheral component (PCI) bus, or an extended industry standard architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc., but this does not mean there is only one bus or one type of bus.

[0230] Optionally, in a specific implementation, if the memory 502 and the processor 501 are integrated on a single chip, the memory 502 and the processor 501 can communicate through an internal interface.

[0231] This application also provides a computer-readable storage medium, which may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a disk, or an optical disk. Specifically, the computer-readable storage medium stores program instructions, which are used in the methods described in the above-mentioned method embodiments.

[0232] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the methods described in the above-described method embodiments.

[0233] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0234] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for constructing a professional lexicon, characterized by, The method comprises the following steps: obtaining professional corpus and general corpus, and performing word extraction processing on the professional corpus and the general corpus to obtain an initial vocabulary and a general vocabulary; calculating a first product of a first total word quantity and a second quantity, the first total word quantity being a total number of words in the initial vocabulary; the second quantity being a number of each candidate word in the initial vocabulary in the general vocabulary; calculating a second product of a second total word quantity and a first quantity, the second total word quantity being a total number of words in the general vocabulary, and the first quantity being a number of the candidate word in the initial vocabulary; determining a professional word association degree of each candidate word based on a quotient of the second product and the first product; if the professional word association degree of the candidate word is greater than a preset threshold, determining the candidate word as a seed word to obtain a seed word set; in the initial vocabulary, taking each seed word in the seed word set as a seed point, performing clustering processing on the initial vocabulary based on the seed point corresponding to each seed word and a preset clustering algorithm to obtain a candidate word set; determining a professional vocabulary according to a preset word selection condition and the candidate word set.

2. The professional dictionary construction method of claim 1, wherein, the seed word in the seed word set is a word in the initial vocabulary and not a word in the general vocabulary; and / or, a first probability of searching for the seed word in the initial vocabulary is greater than a second probability of searching for the seed word in the general vocabulary.

3. The professional dictionary construction method of claim 1, wherein, the step of performing clustering processing on the initial vocabulary based on the seed point corresponding to each seed word and the preset clustering algorithm in the initial vocabulary, taking each seed word in the seed word set as a seed point, comprises the following steps: respectively calculating a semantic distance between each candidate word and each seed word; if the semantic distance between the candidate word and the seed word is less than a preset distance threshold, the candidate word and the corresponding seed word form a first candidate word set, and each seed word obtains one first candidate word set.

4. The professional dictionary construction method of claim 3, wherein, after the candidate word is added to the first candidate word set corresponding to the seed word, the method further comprises the following steps: according to the size of the semantic distance, the first K first candidate words in the first candidate word set corresponding to each seed word are taken as second candidate words, the semantic distance between the first K first candidate words and the corresponding seed word is less than the semantic distance between other candidate words and the seed word except the first K first candidate words; combining the K second candidate words into one second candidate word set, the candidate word set further comprises: a second candidate word set corresponding to each seed word, and K is a positive integer.

5. The professional dictionary construction method of claim 4, wherein, after the K second candidate words are combined into one second candidate word set, the method further comprises the following steps: selecting M first candidate word sets according to the number of words contained in each first candidate word set to form a third candidate word set; wherein the number of words in each first candidate word set in the M first candidate word sets is greater than the number of words in a first candidate word set other than the M first candidate word sets; the candidate word set comprises the third candidate word set, and M is a positive integer.

6. The professional dictionary construction method of claim 5, wherein, After the M first candidate word sets are selected according to the number of words contained in each of the first candidate word sets and combined into a third candidate word set, the method further comprises: combining the seed word set, each of the second candidate word sets and the third candidate word set into the candidate word set and removing duplicate words in the candidate word set.

7. The method of claim 1 to 6, wherein, The preset word selection condition comprises at least one of a word frequency condition and a part of speech condition, the word frequency threshold in the word frequency condition corresponds to the size of the professional corpus, and the part of speech condition corresponds to the professional field in the professional corpus.

8. An apparatus for automatically constructing a specialized vocabulary, characterized by comprising: The method comprises: acquiring a professional corpus and a general corpus; processing the professional corpus and the general corpus to obtain an initial word library and a general word library; determining a professional part of speech association degree of each candidate word in the initial word library according to the general word library, and extracting a seed word set from the initial word library according to the professional part of speech association degree, each seed word in the seed word set being a candidate word; respectively taking each seed word in the seed word set as a seed point in the initial word library, performing clustering processing on the initial word library based on the seed point corresponding to each seed word and a preset clustering algorithm to obtain a candidate word set, and determining a professional word library according to a preset word selection condition and the candidate word set; the processing module is specifically configured to calculate a first product of a first total word quantity and a second quantity, the first total word quantity being a total number of words in the initial word library, and the second quantity being a number of each candidate word in the initial word library in the general word library; calculate a second product of a second total word quantity and a first quantity, the second total word quantity being a total number of words in the general word library, and the first quantity being a number of the candidate word in the initial word library; determine the professional part of speech association degree of each candidate word based on a quotient of the second product and the first product; if the professional part of speech association degree of the candidate word is greater than a preset threshold, the candidate word is determined to be the seed word, and a seed word set is obtained. The method comprises:

9. An electronic device, comprising: a processor; and a memory for storing a computer program of the processor; wherein the processor is configured to execute the computer program to execute the professional word library construction method of any one of claims 1 to 7. The computer program is executed by the processor to implement the professional word library construction method of any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, ​

Images