Characteristic distillation-based concept similarity calculation method and system

Abstract

The invention discloses a concept similarity calculation method and system based on feature distillation. The method comprises the steps that S1, two original definitions in a term library are acquired; s2, performing word segmentation on the definition to obtain a word segmentation result; s3, calculating a first vector of the vocabulary according to a word segmentation result, S4, judging whether the vocabulary is an important vocabulary and a final vector which is not the important vocabulary is the first vector, if so, calculating a weighted vector of the important vocabulary and recording the weighted vector as a second vector, and adding the first vector and the second vector of the important vocabulary to obtain a final vector of the important vocabulary; s5, calculating a definition sentence vector; the step of calculating the weighted vector of the important vocabulary comprises the sub-steps of searching a plurality of definitions corresponding to the important vocabulary in a term library, executing S2-S5 to calculate a definition sentence vector corresponding to the important vocabulary, and adding products of the sentence vector of the plurality of definition sentences corresponding to the important vocabulary and the weight assignment to obtain the weighted vector of the important vocabulary; and S6, calculating the concept similarity according to the defined sentence vector. Through the method and the device, the accuracy of semantic similarity analysis is improved.

Description

Technical Field

[0001] The present application relates to the technical field of term base maintenance, and in particular to a concept similarity calculation method and system. Background Art

[0002] A term base is a database system specifically used to store, manage and retrieve professional terms. It plays a vital role in maintaining language accuracy and promoting interdisciplinary and cross-industry communication. A term base not only contains the name of the term, but also covers the detailed definition, usage, source, and relationship with other terms of each term. When building and maintaining a term base, it is particularly important to calculate the similarity of the term definitions. This process helps to identify and resolve duplication, ambiguity or conflict in the term definitions, and ensure the accuracy and consistency of the terms. At the same time, in terms of term resource integration, the semantic similarity calculation method can effectively identify and associate term resources from different sources, and achieve seamless integration of term resources. Therefore, it is very important to accurately calculate the similarity between term definitions in the term base.

[0003] When calculating sentence similarity in the prior art, word vectors are only generated through the FastText model, and the word vectors generated by FastText are deeply analyzed through the forward and backward sequence processing of the BiLSTM model to obtain sentence vectors. However, this method has certain limitations when processing data in the terminology library. When faced with professional terms or domain-specific vocabulary, the word vectors generated by the FastText model may not fully reflect the professionalism and domain relevance of these words. This limitation may cause the semantic results of the model to be inaccurate when processing definition sentences in the terminology library, especially when the definition sentences contain a large number of professional terms. This information has not been fully mined and used. Specifically, the hybrid model may not be able to distinguish the specific meanings and differences between terms with the same name in different fields, thereby affecting the accuracy of the analysis. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a method and system for calculating concept similarity based on feature distillation is provided to solve the problem that the prior art has low accuracy when calculating the similarity of sentences defined in a term library.

[0005] In a first aspect, a method for calculating concept similarity based on feature distillation is provided, the method comprising:

[0006] Step S1: obtaining two original definitions in the term library whose similarity is to be calculated;

[0007] Step S2: segmenting the definition to obtain a segmentation result;

[0008] Step S3: Calculate the word vector of each word according to the defined word segmentation result, and record it as the first vector of each word;

[0009] Step S4: Determine whether each word in the defined word segmentation result is an important word. If not, the final vector of the non-important word is the first vector. If yes, calculate the weighted vector of the important word, record it as the second vector, and add the first vector and the second vector of the important word to obtain the final vector of the important word.

[0010] Step S5: obtaining the final vector of each word in the defined word segmentation result, and calculating the defined sentence vector according to the final vector of each word in the defined word segmentation result;

[0011] The calculating of the weighted vector of the important vocabulary comprises: searching for multiple definitions corresponding to the important vocabulary in the term library, repeatedly executing steps S2 to S5 to calculate the sentence vector of the definition corresponding to the important vocabulary, and obtaining the weight assignment of the definition sentence of the important vocabulary according to the category relationship between the classification number of the definition corresponding to the important vocabulary and the classification number of the original definition where the important vocabulary is located, and adding the product of the sentence vectors of the multiple definition sentences corresponding to the important vocabulary and the corresponding weight assignment to obtain the weighted vector of the important vocabulary;

[0012] Step S6: Calculate the concept similarity based on the sentence vector of the original definition of the similarity to be calculated.

[0013] In the above scheme, optionally, after obtaining the two original definitions to be calculated for similarity in the terminology library, it also includes: judging whether there is referenced parsed content in the terminology library for the definition, and if so, obtaining the parsed content referenced by the definition in the terminology library, and replacing the part of the definition with the referenced parsed content.

[0014] In the above scheme, further optionally, after replacing the part of the definition with the referenced parsed content or after the definition does not have the referenced parsed content in the term library, it also includes: preprocessing the original definition to remove redundant punctuation marks and irrelevant characters in the text.

[0015] In the above scheme, further optionally, after replacing the part of the definition with the referenced parsed content or after the original definition does not have the referenced parsed content in the term library, it also includes: using a synonym dictionary to map each word in the definition after word segmentation to a standard form; the synonym dictionary contains a large number of mapping relationships between standard words and their synonyms.

[0016] In the above scheme, optionally, the definition is segmented using a jieba segmenter with a proper noun segmentation table pre-added; the proper noun segmentation table contains professional terms in a large number of fields.

[0017] In the above solution, further optionally, the FastText model is used to calculate the word vector of the vocabulary.

[0018] In the above scheme, optionally, the calculation of the defined sentence vector according to the final vector of each word in the defined word segmentation result includes: inputting the final vector of each word in the defined word segmentation result into the BiLSTM model, and calculating the sentence vector of the entire sentence.

[0019] In a second aspect, a concept similarity calculation system based on feature distillation is provided, the system comprising:

[0020] Definition acquisition module: used to obtain two original definitions in the term base whose similarity is to be calculated;

[0021] Word segmentation module: used to segment the definition to obtain a word segmentation result;

[0022] Word vector calculation module: used to calculate the word vector of each word according to the defined word segmentation results, and record it as the first vector of each word;

[0023] The final vector calculation module of the vocabulary: used to determine whether each vocabulary in the defined word segmentation result is an important vocabulary. If not, the final vector of the non-important vocabulary is the first vector. If yes, the weighted vector of the important vocabulary is calculated and recorded as the second vector. The first vector and the second vector of the important vocabulary are added to obtain the final vector of the important vocabulary.

[0024] Sentence vector calculation module: used to obtain the final vector of each word in the defined word segmentation result, and calculate the defined sentence vector according to the final vector of each word in the defined word segmentation result;

[0025] The weighted vector calculation module of important vocabulary: used to search for multiple definitions corresponding to important vocabulary in the terminology library, repeatedly execute the word segmentation module, the word vector calculation module, the final vector calculation module of vocabulary and the sentence vector calculation module to calculate the sentence vector of the definition corresponding to the important vocabulary, and obtain the weight assignment of the definition sentence of the important vocabulary according to the category relationship between the classification number of the classification information of the definition corresponding to the important vocabulary and the classification number of the classification information of the original definition where the important vocabulary is located, and add the product of the sentence vectors of the multiple definition sentences corresponding to the important vocabulary and the corresponding weight assignment to obtain the weighted vector of the important vocabulary;

[0026] Cosine similarity calculation module: used to calculate the cosine similarity according to the sentence vector of the original definition of the similarity to be calculated.

[0027] In a third aspect, a computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for calculating concept similarity based on feature distillation described in the first aspect is implemented.

[0028] In a fourth aspect, a computer program product includes a computer program / instruction, which, when executed by a processor, implements a method for calculating concept similarity based on feature distillation as described in the first aspect.

[0029] This application has at least the following beneficial effects:

[0030] When calculating the similarity of two definition sentences, the application first performs word segmentation on the definition sentence, and then uses the term library to find out whether each word in the word segmentation result belongs to an important word (term). If so, multiple definitions of important words in the term library are introduced, and the sentence vector of each definition of important words is calculated, and different weights are assigned according to the category relationship between the classification information of the definition of important words and the classification information of the original definition where the important words are located, so that the definition most relevant to the important words can get the largest weight, and finally the product of the sentence vectors of the multiple definitions of important words and the assigned weights is added to obtain the weighted vector of the important words, and then the word vector of the important words and the weighted vector are added to the final word vector of the important words. Thus, in the process of calculating the final word vector of the vocabulary, the definition information of the field is introduced, and the sentence vector defined in the specific field is more accurately reflected, thereby improving the accuracy and context understanding ability when processing field-specific data, thereby improving the accuracy and effectiveness of semantic similarity analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A specific flow chart of a method for calculating concept similarity based on feature distillation is provided for an embodiment of the present application;

[0032] Figure 2 A flowchart of steps for constructing a word segmentation table provided in one embodiment of the present application;

[0033] Figure 3 A schematic diagram of a weighted vector flow of important words provided in one embodiment of the present application;

[0034] Figure 4 A schematic diagram of a method for assigning weights to multiple definitions of important words provided in one embodiment of the present application;

[0035] Figure 5 An embodiment of the present application provides a flowchart of a method for calculating concept similarity based on feature distillation;

[0036] Figure 6A flowchart for constructing a synonym dictionary is provided for one embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0038] In the description of this application: unless otherwise specified, the meaning of "plurality" is two or more. The terms "first", "second", "third", etc. in this application are intended to distinguish the objects referred to, and do not have special meanings in terms of technical connotations (for example, they should not be understood as emphasizing the importance or order, etc.). Expressions such as "including", "comprising", "having", etc. also mean "not limited to" (certain units, components, materials, steps, etc.).

[0039] In one embodiment, Figure 1 and Figure 5 As shown, a method for calculating concept similarity based on feature distillation is provided, and the method includes:

[0040] Step S1: Obtain two original definitions in the term library whose similarity is to be calculated.

[0041] Step S2: Segment the definition to obtain a segmentation result.

[0042] In step S2, Chinese word segmentation is mainly processed using the Jieba word segmenter. The Jieba word segmenter is a widely used Chinese word segmentation tool that can split a continuous sequence of Chinese characters into independent words. In order to ensure the integrity of proper nouns and the accuracy of word segmentation, it is necessary to build and add a custom word segmentation table to the Jieba word segmenter. The proper noun word segmentation table contains a large number of professional terms in various fields. Generally speaking, when building a term library, the builder can build an analysis table based on his own term library, such as Figure 2 The following are the steps to construct a word segmentation table. The process of constructing a word segmentation table includes:

[0043] Extract key (term) elements: extract the terminology from the term base. For example, some terminology in the term base contains three columns of data: "Chinese terms", "other preferred Chinese terms" and "permitted Chinese terms". Therefore, extract all the term names contained in these three columns of data.

[0044] Term segmentation: Segmentation is performed during extraction to extract individual terms and avoid combining multiple terms into one.

[0045] Deduplication: To ensure the uniqueness of the word segmentation table, a deduplication operation is performed, all segmented words are stored in a set, and the natural deduplication property of the set is used to ensure the uniqueness of the words.

[0046] Of course, the word segmentation table constructed above is only an example. If the data classification contained in the terminology library is different from the classification contained above, the word segmentation table can be constructed according to its own classification, or the word segmentation table that has been constructed in the corresponding field can be used.

[0047] Step S3: Calculate the word vector of each word according to the defined word segmentation result, and record it as the first vector of each word.

[0048] In step S3, the pre-trained FastText model is used to calculate the word vector of each word after word segmentation. FastText is a word embedding technology that not only represents the whole word, but also incorporates the subword structure of the word to generate a richer and more accurate word vector. For each word in the word segmentation result of the term definition, the FastText model generates a fixed-length vector that effectively captures the semantic information of the word.

[0049] Step S4: Determine whether each word in the defined word segmentation result is an important word. If not, the final vector of the non-important word is the first vector. If so, calculate the weighted vector of the important word, recorded as the second vector, and add the first vector and the second vector of the important word to obtain the final vector of the important word.

[0050] In step S4, the term base is checked to determine whether the words in the text are important terms. Only words that exist in the term base are marked as important words. The words in the term base usually cover professional terms and keywords in the field, which often have important meanings and functions in the definition text.

[0051] When calculating the weighted vector of an important word, we usually look for the definition sentence corresponding to the word in the term library. If the definition sentence contains other important words, we also use the same method to calculate the vectors of these words. This calculation method is recursive, that is, by calculating the vectors of the definition sentences containing important words, we gradually obtain the vector representation of these words. This weighted method can effectively integrate domain characteristics into the text and ensure that important words are highlighted and accurately expressed.

[0052] Step S5: Obtain the final vector of each word in the defined word segmentation result, and calculate the defined sentence vector according to the final vector of each word in the defined word segmentation result.

[0053] In step S5, the word vector after word segmentation is input into the BiLSTM model to calculate the vector representation of the entire sentence. BiLSTM (Bidirectional Long Short-Term Memory Network) is a recursive neural network that can learn long-distance dependencies from sequence data. The BiLSTM model processes the word vector sequence through two LSTM networks, forward and backward, to capture the contextual information of the words in the sentence. Finally, the BiLSTM model generates a sentence vector, which is a semantic representation of the entire sentence.

[0054] The calculation of the weighted vector of the important words includes: Figure 3 As shown, it is a flowchart for calculating the vocabulary weighted vector, searching for multiple definitions corresponding to important vocabulary in the terminology library, repeating steps S2-S5 to calculate the sentence vector of the definition corresponding to the important vocabulary, and obtaining the weight assignment of the definition sentence of the important vocabulary according to the category relationship between the classification information of the definition corresponding to the important vocabulary and the classification information of the original definition where the important vocabulary is located, adding the product of the sentence vectors of the multiple definition sentences corresponding to the important vocabulary and the corresponding weight assignment to obtain the weighted vector of the important vocabulary.

[0055] That is to say, in the process of calculating the weighted vector of important words, after finding the definition sentence corresponding to the important words, it is necessary to determine whether the definition sentence corresponding to the important words still contains important words. If so, recalculate the weighted vector of the newly found important words until the definition found does not contain important words. For example, if A is an important word, and the definition of A is abcd, and c is found as an important word in the definition of abcd, then continue to look for the definition of c until the definition found does not contain important words.

[0056] like Figure 4 As shown, in the process of calculating the weighted vector of important words, the weight assignment for multiple definitions of important words includes the following:

[0057] First, determine whether the classification information of the definition corresponding to the important vocabulary is the same as the national standard number of the classification information of the original definition where the important vocabulary is located. If so, the weight is 1;

[0058] If not, compare and determine whether the national standard numbers are the same. If they are the same, the weight is 1;

[0059] If they are not the same, further determine the Chinese standard classification number. Since the international standard number is divided into three levels, XX.XX.XX, the comparison result needs further discussion: if all three levels are the same, the weight is 1; if all three levels are different, the weight is 0; if part of the three levels are the same, if the first two parts are the same and only the third part is different, the weight can be appropriately increased, such as 0.8 (such as 21.2.1 and 21.2.3); if only the first part is the same and the second part is different, the weight is relatively low, such as 0.3 (such as 21.1.X and 21.2.Y); if there is only one level and it is the same, the weight is 0.5 (such as 21 and 21.XY).

[0060] Of course, the above specific weight assignment method is only an example. In actual use, it is necessary to use the category relationship to determine the actual assignment method according to the actual application situation.

[0061] Step S6: Calculate the concept similarity based on the sentence vector of the original definition of the similarity to be calculated.

[0062] In step S6, cosine similarity is used to measure the similarity between two vectors. It reflects the similarity by calculating the cosine value of the angle between the two vectors. The range of cosine value is between -1 and 1. The closer the value is to 1, the more similar the two vectors are, which means the higher the similarity between the texts. i and L j , their cosine similarity is defined as:

[0063]

[0064] In the above-mentioned concept similarity calculation method based on feature distillation, when calculating the similarity of two definition sentences, the definition sentences are first segmented, and then the term library is used to find out whether each word in the segmentation result belongs to an important word (term). If so, multiple definitions of important words in the term library are introduced, and the sentence vector of each definition of important words is calculated, and different weights are assigned according to the category relationship between the classification information of the definition of important words and the classification information of the original definition where the important words are located, so that the definition most relevant to the important words can get the largest weight, and finally the product of the sentence vectors of the multiple definitions of important words and the assigned weights is added to obtain the weighted vector of the important words, and then the word vector of the important words and the weighted vector are added to obtain the final word vector of the important words. Therefore, in the process of calculating the final word vector of the vocabulary, the definition information of the field is introduced to more accurately reflect the sentence vector defined in the specific field, thereby improving the accuracy and context understanding ability when processing field-specific data, thereby improving the accuracy and effectiveness of semantic similarity analysis.

[0065] In one embodiment, after obtaining two original definitions in the terminology library whose similarity is to be calculated, the method further includes: determining whether there is referenced parsed content in the terminology library for the definition, and if so, obtaining the referenced parsed content in the terminology library, and replacing the referenced parsed part of the definition with the referenced parsed content.

[0066] In this embodiment, the purpose of reference parsing is to identify and parse the reference content in the text and replace it with the actual definition or explanation to ensure the integrity and coherence of the text. The Chinese definitions in the term library often contain references to other definitions, usually expressed in the form of "see <number>". Each number corresponds to a specific term. In order to perform accurate text similarity calculation, these references need to be parsed and replaced with the corresponding detailed definitions.

[0067] In one embodiment, after replacing the referenced parsed part of the definition with the referenced parsed content or after the referenced parsed content does not exist in the term library, the method further includes: preprocessing the original definition to remove redundant punctuation marks and irrelevant characters in the text to improve the quality and consistency of the data.

[0068] In one embodiment, after replacing the referenced parsed portion of the definition with the referenced parsed content or after the referenced parsed content does not exist in the term library for the original definition, the method further includes: mapping each word in the segmented definition to a standard form using a synonym dictionary; the synonym dictionary contains a large number of mapping relationships between standard words and their synonyms.

[0069] In this embodiment, vocabulary standardization is to summarize the diverse vocabulary forms in the text into a standard form. To this end, a synonym dictionary is created to map the vocabulary to its standard form. When building a synonym dictionary, it is necessary to cover not only common vocabulary but also proper nouns. Among them, common vocabulary mainly relies on existing synonym resources, while proper nouns can be created based on the information in the term library.

[0070] If the data in the term base includes two columns of terms, "Chinese terms" and "other preferred Chinese terms", then you can directly build based on these two columns of terms, such as Figure 6 As shown, the specific steps are:

[0071] Data preparation and definition: Extract data from the two columns "Chinese terms" and "Other preferred Chinese terms" in the term base. The "Chinese terms" column contains the standard form of each term, while the "Other preferred Chinese terms" column lists the corresponding synonyms. In the term base, these synonyms are usually separated by delimiters (such as double semicolons ";;").

[0072] Build a synonym mapping dictionary: record the mapping relationship between standard vocabulary and its synonyms. The key of the dictionary is the standard vocabulary, and the value is a set of all synonyms. You need to ensure that all synonyms are consistently mapped to the corresponding standard vocabulary to maintain consistency.

[0073] Verify and save the dictionary: Verify the constructed dictionary, check whether the synonym mapping of the standard vocabulary is accurate, ensure that there are no omissions or conflicts, and save the synonym dictionary in txt format for subsequent use and maintenance. By systematically creating a synonym dictionary, the synonyms in the text are uniformly processed, thereby improving the accuracy and consistency of subsequent text similarity comparisons.

[0074] Of course, the above is only an example of constructing a synonym dictionary. If there is no data division such as "Chinese terms" and "other preferred Chinese terms" in the terminology database, the synonym dictionary that has been constructed in the existing corresponding field can be used.

[0075] In one embodiment, a concept similarity calculation system based on feature distillation is provided, the system comprising:

[0076] Definition acquisition module: used to obtain two original definitions in the term base whose similarity is to be calculated;

[0077] Word segmentation module: used to segment the definition to obtain a word segmentation result;

[0078] Word vector calculation module: used to calculate the word vector of each word according to the defined word segmentation results, and record it as the first vector of each word;

[0079] The final vector calculation module of the vocabulary: used to determine whether each vocabulary in the defined word segmentation result is an important vocabulary. If not, the final vector of the non-important vocabulary is the first vector. If yes, the weighted vector of the important vocabulary is calculated and recorded as the second vector. The first vector and the second vector of the important vocabulary are added to obtain the final vector of the important vocabulary.

[0080] Sentence vector calculation module: used to obtain the final vector of each word in the defined word segmentation result, and calculate the defined sentence vector according to the final vector of each word in the defined word segmentation result;

[0081] The weighted vector calculation module of important vocabulary: used to search for multiple definitions corresponding to important vocabulary in the terminology library, repeatedly execute the word segmentation module, the word vector calculation module, the final vector calculation module of vocabulary and the sentence vector calculation module to calculate the sentence vector of the definition corresponding to the important vocabulary, and obtain the weight assignment of the definition sentence of the important vocabulary according to the category relationship between the classification number of the classification information of the definition corresponding to the important vocabulary and the classification number of the classification information of the original definition where the important vocabulary is located, and add the product of the sentence vectors of the multiple definition sentences corresponding to the important vocabulary and the corresponding weight assignment to obtain the weighted vector of the important vocabulary;

[0082] Cosine similarity calculation module: used to calculate the cosine similarity according to the sentence vector of the original definition of the similarity to be calculated.

[0083] The key of the present invention is to introduce additional domain knowledge to improve the accuracy and contextual understanding ability of the model when processing domain-specific data. In the terminology library, each term contains national standard number, international standard classification number and Chinese standard classification number information, which provides the domain background of the term. Therefore, consider weighting the sentence vector generated by the FastText model.

[0084] Compared with the prior art, the technical solution proposed in the present invention combines the classification information in the term library during the vector generation process, recalculates the words in the same field and assigns different weights. This weighted processing not only improves the accuracy of sentence vectors, but also highlights the importance of words in sentences, thereby improving the model's ability to recognize and analyze term definition sentences. The improved model can more accurately reflect the sentence vectors defined in a specific field, thereby improving the accuracy and effectiveness of semantic similarity analysis.

[0085] For the specific definition of a concept similarity calculation system based on feature distillation, please refer to the definition of a concept similarity calculation method based on feature distillation above, which will not be repeated here. Each module in the above-mentioned concept similarity calculation system based on feature distillation can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0086] In one embodiment, a computer device is provided, which may be a server, and includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above-mentioned concept similarity calculation method based on feature distillation is implemented.

[0087] In one embodiment, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, involves all or part of the process in the above-mentioned embodiment method.

[0088] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0089] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A concept similarity calculation method based on feature distillation, characterized in that: The method comprises: Step S1: obtaining two original definitions in the term library whose similarity is to be calculated; Step S2: segmenting the definition to obtain a segmentation result; Step S3: Calculate the word vector of each word according to the defined word segmentation result, and record it as the first vector of each word; Step S4: Determine whether each word in the defined word segmentation result is an important word. If not, the final vector of the non-important word is the first vector. If yes, calculate the weighted vector of the important word, record it as the second vector, and add the first vector and the second vector of the important word to obtain the final vector of the important word. Step S5: obtaining the final vector of each word in the defined word segmentation result, and calculating the defined sentence vector according to the final vector of each word in the defined word segmentation result; The calculating of the weighted vector of the important vocabulary comprises: searching for multiple definitions corresponding to the important vocabulary in the term library, repeatedly executing steps S2 to S5 to calculate the sentence vector of the definition corresponding to the important vocabulary, and obtaining the weight assignment of the definition sentence of the important vocabulary according to the category relationship between the classification number of the definition corresponding to the important vocabulary and the classification number of the original definition where the important vocabulary is located, and adding the product of the sentence vectors of the multiple definition sentences corresponding to the important vocabulary and the corresponding weight assignment to obtain the weighted vector of the important vocabulary; Step S6: Calculate the concept similarity based on the sentence vector of the original definition of the to-be-calculated similarity.

2. The method for calculating concept similarity based on feature distillation according to claim 1, characterized in that: After obtaining the two original definitions to be calculated for similarity in the terminology library, the method further includes: determining whether there is referenced parsed content in the terminology library for the definition, and if so, obtaining the referenced parsed content in the terminology library, and replacing the referenced parsed part of the definition with the referenced parsed content.

3. The method for calculating concept similarity based on feature distillation according to claim 2, characterized in that: After replacing the referenced parsed part of the definition with the referenced parsed content or after the referenced parsed content does not exist in the term library, the method further includes: preprocessing the original definition to remove redundant punctuation marks and irrelevant characters in the text.

4. The method for calculating concept similarity based on feature distillation according to claim 2, characterized in that: After replacing the referenced parsed part of the definition with the referenced parsed content or after the referenced parsed content does not exist in the term library for the original definition, the method further includes: mapping each word in the segmented definition to a standard form using a synonym dictionary; the synonym dictionary contains a large number of mapping relationships between standard words and their synonyms.

5. The method for calculating concept similarity based on feature distillation according to claim 1, characterized in that: The definition is segmented using the Jieba word segmenter with a proper noun segmentation table pre-added; the proper noun segmentation table contains a large number of professional terms in a certain field.

6. The method for calculating concept similarity based on feature distillation according to claim 1, characterized in that: Use the FastText model to calculate the word vectors of the vocabulary.

7. The method for calculating concept similarity based on feature distillation according to claim 1, characterized in that: The step of calculating the defined sentence vector according to the final vector of each word in the defined word segmentation result includes: inputting the final vector of each word in the defined word segmentation result into the BiLSTM model, and calculating the sentence vector of the entire sentence.

8. A concept similarity calculation system based on feature distillation, characterized in that: The system comprises: Definition acquisition module: used to obtain two original definitions in the term base whose similarity is to be calculated; Word segmentation module: used to segment the definition to obtain a word segmentation result; Word vector calculation module: used to calculate the word vector of each word according to the defined word segmentation results, and record it as the first vector of each word; The final vector calculation module of the vocabulary: used to determine whether each vocabulary in the defined word segmentation result is an important vocabulary. If not, the final vector of the non-important vocabulary is the first vector. If yes, the weighted vector of the important vocabulary is calculated and recorded as the second vector. The first vector and the second vector of the important vocabulary are added to obtain the final vector of the important vocabulary. Sentence vector calculation module: used to obtain the final vector of each word in the defined word segmentation result, and calculate the defined sentence vector according to the final vector of each word in the defined word segmentation result; The weighted vector calculation module of important vocabulary: used to search for multiple definitions corresponding to important vocabulary in the terminology library, repeatedly execute the word segmentation module, the word vector calculation module, the final vector calculation module of vocabulary and the sentence vector calculation module to calculate the sentence vector of the definition corresponding to the important vocabulary, and obtain the weight assignment of the definition sentence of the important vocabulary according to the category relationship between the classification number of the classification information of the definition corresponding to the important vocabulary and the classification number of the classification information of the original definition where the important vocabulary is located, and add the product of the sentence vectors of the multiple definition sentences corresponding to the important vocabulary and the corresponding weight assignment to obtain the weighted vector of the important vocabulary; Cosine similarity calculation module: used to calculate the cosine similarity according to the sentence vector of the original definition of the similarity to be calculated.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images