A professional term recognition method and device based on a natural language model

Abstract

This invention discloses a method and apparatus for identifying professional terms based on a natural language model, belonging to the field of natural language recognition technology. The method includes: determining identifiers for the relevant field; obtaining several keywords based on the identifiers; searching for M texts based on the keywords; selecting the K characters following the identifiers as candidate terms from the texts; matching the candidate terms against the texts, comparing their frequency with a preset threshold, and gradually decreasing the K value until a suitable professional term is determined; repeating the above operations to obtain the identification results for all professional terms. This invention provides a method for identifying professional terms based on a natural language model, solving the problem of identifying and building a professional term database for emerging industries.

Description

Technical Field

[0001] This invention relates to the field of natural language recognition technology, and in particular to a method and apparatus for recognizing specialized terms based on natural language models. Background Technology

[0002] To study the relevant knowledge of a particular profession, researchers often need to first identify the professional terminology. Only based on this established terminology can further research be conducted. However, for emerging industries, there is often a lack of substantive methods for identifying the professional terminology; instead, it is often determined by manual methods. Therefore, finding a suitable identification method to establish a professional terminology database for this industry is a pressing issue that needs to be addressed. Summary of the Invention

[0003] To address this problem, this invention provides a method for identifying specialized terms based on a natural language model, comprising:

[0004] (1) Determine the identifiers of the relevant field, and obtain several keywords based on the identifiers;

[0005] (2) Based on the keywords, M texts were obtained by searching;

[0006] (3) In the corpus, select the K characters after the marker word as candidate nouns, denoted as O;

[0007] (4) Match the candidate noun O in the corpus, count the number of times it appears in the corpus n1, and calculate... Where M1 represents the total number of segments in the M corpus;

[0008] (5) Compare p1 with the preset threshold p 1d If p1>p 1d If the combination of the identifier and the candidate noun O is deemed to be a suitable technical term, then proceed to step (6);

[0009] (6) Calculation Where q is the average number of characters in each segment of the corpus;

[0010] (7) Compare p2 with the preset threshold p 2d If p2>p 2d If the combination of the identifier and the candidate noun O is considered to be a suitable professional term, then K = K-1, and return to step (3);

[0011] (8) Repeat steps (3)-(7) to obtain the recognition results of all technical terms.

[0012] Furthermore, K characters following the identifier are selected as candidate nouns. When the selected K characters contain punctuation marks, K is the number of characters before the punctuation marks following the identifier.

[0013] Furthermore, the professional terms identified in step (5) are common professional terms in the several keyword fields.

[0014] Furthermore, p 1d The value range is between 0.2 and 0.3.

[0015] Furthermore, the professional term identified in step (7) is a professional term in a specific field among the several keyword fields.

[0016] Furthermore, p 2d The value ranges from 0.005 to 0.007.

[0017] This invention also provides a device for recognizing technical terms based on a natural language model, comprising:

[0018] Keyword acquisition module: used to determine the identifier words of the relevant field, and obtain several keywords based on the identifier words;

[0019] Candidate noun acquisition module: used to search for M texts based on the keywords; in the texts, select the K characters after the identifier as candidate nouns, denoted as O;

[0020] Candidate Noun Judgment Module: Used to match the candidate noun O in the corpus, count the number of times it appears in the corpus n1, and calculate... Where M1 is the total number of segments in the M texts; compare p1 with the preset threshold p 1d If p1>p 1d If the candidate noun O is found to be a suitable technical term, then the calculation is performed; otherwise, the calculation is performed. Where q is the average number of characters in each segment of the corpus; compare p2 with the preset threshold p 2d If p2>p 2d If the candidate noun O is found to be a suitable professional term, then K = K-1, and the candidate noun is retrieved again, and the above operation is repeated.

[0021] The present invention also provides a computer-readable storage medium, characterized in that it stores a computer program thereon, which, when executed, implements the professional term recognition method based on a natural language model as described in any of the preceding claims.

[0022] The present invention also provides a computer device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the natural language model-based professional terminology recognition method as described in any of the preceding claims.

[0023] The beneficial effects of this invention are:

[0024] This invention provides a mechanism for identifying technical terms based on natural language models and word frequency comparison, which can solve the problem of establishing a technical term database in emerging industries. Attached Figure Description

[0025] Figure 1 This is a flowchart illustrating the professional term recognition method based on a natural language model according to an embodiment of the present invention. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] This embodiment uses the identification of professional terms in the field of "dual carbon" as an example for introduction.

[0028] Example 1, as Figure 1 As shown, a method for identifying technical terms based on a natural language model includes:

[0029] S101. Determine the identifiers of the relevant field, and obtain several keywords based on the identifiers;

[0030] In this embodiment, we use "carbon" as the identifier. In our research on issues related to "carbon and carbon", we divide the entire "carbon and carbon" field into "carbon inventory", "carbon monitoring", "carbon analysis", "carbon optimization", "carbon asset management", "carbon consulting" and "carbon footprint". We can assume that all professional terms in the field of "carbon and carbon" will only involve the above seven areas. Therefore, all the "core", that is, the candidate terms, will only appear in the areas of "inventory", "monitoring", "analysis", "optimization", "asset management", "consulting" and "footprint".

[0031] S102. Based on the keywords, M texts are retrieved.

[0032] The search engine searches for and downloads M corpora for the keywords "investigation", "monitoring", "analysis", "optimization", "asset management", "consulting" and "footprint", respectively. M>1000 is a user-defined constant. There are a total of 7M corpora, with an average of q words per corpus, and a total of 7Mq words in the corpus.

[0033] S103. In the corpus, select the K characters following the identifier as candidate nouns, denoted as O;

[0034] After locating the target word "carbon", select the K characters following the target word, where K is obtained through verification. If the selected K characters contain punctuation marks, then K is the number of characters following the target word and before the punctuation marks.

[0035] S104. Match the candidate noun O in the corpus, count the number of times it appears in the corpus n1, and calculate... Where M1 represents the total number of segments in the M corpus;

[0036] The business significance of p1 is the proportion of candidate noun O appearing in all corpus segments. The larger the p1 is, the more likely O is to be a candidate noun.

[0037] S105. Compare p1 with the preset threshold p 1d If p1>p 1d If the combination of the identifier and the candidate noun O is deemed to be a suitable technical term, then step S106 is executed.

[0038] threshold p 1d Pre-set for users, based on experience, p 1d A value between 0.2 and 0.3 is more appropriate. The technical terms identified in this step are those common to several keyword fields.

[0039] S106, Calculation Where q is the average number of characters in each segment of the corpus;

[0040] The business significance of p2 is the proportion of candidate noun O in all the texts in the corpus. The larger the p2 is, the more likely O is to be a candidate noun.

[0041] S107. Compare p2 with the preset threshold p. 2d If p2>p 2d If the combination of the identifier and the candidate noun O is deemed to be a suitable technical term, then K = K-1, and the process returns to step S103.

[0042] threshold p 2d Pre-set for users, based on experience, p 2d It should be between 0.005 and 0.007. The technical terms identified in this step are technical terms from a specific field within the aforementioned keyword areas.

[0043] S108. Repeat steps S103-S107 to obtain the recognition results for all technical terms.

[0044] Select the K characters following the next identifier as candidate nouns, and repeat steps S103-S107 to identify the next technical term.

[0045] For the method in this patent, we always assume that the corpora and professional terms in each field are comparable, that is, the total length of the corpus, the total number of corpora, the average length of the corpus, and the density of professional terms are similar. If this assumption is not true, then the corresponding scenario is not applicable to this patent.

[0046] Example 2: The present invention also provides a device for recognizing technical terms based on a natural language model, comprising:

[0047] Keyword acquisition module: used to determine the identifier words of the relevant field, and obtain several keywords based on the identifier words;

[0048] Candidate noun acquisition module: used to search for M texts based on the keywords; in the texts, select the K characters after the identifier as candidate nouns, denoted as O;

[0049] Candidate Noun Judgment Module: Used to match the candidate noun O in the corpus, count the number of times it appears in the corpus n1, and calculate... Where M1 is the total number of segments in the M texts; compare p1 with the preset threshold p 1d If p1>p 1d If the candidate noun O is found to be a suitable technical term, then the calculation is performed; otherwise, the calculation is performed. Where q is the average number of characters in each segment of the corpus; compare p2 with the preset threshold p 2d If p2>p 2d If the candidate noun O is found to be a suitable professional term, then K = K-1, and the candidate noun is retrieved again, and the above operation is repeated.

[0050] In embodiment three, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed, implements the professional term recognition method based on natural language model as described in any of the preceding embodiments.

[0051] In embodiment four, the present invention also provides a computer device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the professional term recognition method based on the natural language model as described in any of the preceding embodiments.

[0052] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A method for professional term recognition based on a natural language model, characterized in that, include: (1) Determine the identifiers of the relevant field, and obtain several keywords based on the identifiers; (2) Based on the keywords, M texts were retrieved; (3) In the corpus, select the K characters following the marker word as candidate nouns, denoted as O; (4) Match the candidate noun O in the corpus and count the number of times it appears in the corpus. and calculate ,in This represents the total number of segments in the M-text corpus. (5) comparison with a preset threshold , if , it is considered that the combination of the identified word and the candidate noun O is a suitable professional noun, otherwise step (6) is performed; The professional terms identified in step (5) are common professional terms in the aforementioned keyword fields; (6) Calculate wherein is the average number of words per corpus segment; (7) comparison with a preset threshold , if , it is considered that the combination of the identifier and the candidate noun O is a suitable professional noun, otherwise , return to step (3); The professional term identified in step (7) is a professional term in a specific field among the several keyword fields; (8) Repeat steps (3)-(7) to obtain the recognition results of all technical terms.

2. The professional term recognition method based on a natural language model according to claim 1, characterized in that: K characters following the identifier are selected as candidate nouns. When the selected K characters contain punctuation marks, K is the number of characters before the punctuation marks following the identifier.

3. The professional term recognition method based on a natural language model according to claim 1, characterized in that: the value range of a is between 0.2 and 0.

3. 4.The professional term recognition method based on a natural language model according to claim 1, characterized in that: the value range of a is between 0.005 to 0.

007. 5.A professional term recognition apparatus based on a natural language model, characterized by comprising: include: Keyword acquisition module: used to determine the identifier words of the relevant field, and obtain several keywords based on the identifier words; Candidate noun acquisition module: used to search for M texts based on the keywords; In the corpus, the K characters following the marker word are selected as candidate nouns, denoted as O; Candidate Noun Judgment Module: Used to match the candidate noun O in the corpus and count the number of times it appears in the corpus. and calculate ,in The total number of segments in the M-text corpus; comparison With the preset threshold ,like If the candidate noun O is found to be a suitable technical term, then the calculation is performed; otherwise, the calculation is performed. ,in The average number of words in each corpus segment; the specialized terms identified here are those common to the aforementioned keyword domains; comparison. With the preset threshold ,like If so, then the candidate noun O is considered a suitable technical term; otherwise... The candidate nouns are retrieved again, and the above operation is repeated. The professional nouns identified here are professional nouns in a specific field among the several keyword fields.

6. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed, implements the natural language model-based professional terminology recognition method according to any one of claims 1 to 4.

7. A computer device, comprising: It includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the natural language model-based professional terminology recognition method according to any one of claims 1 to 4.

Images