A context-aware vocabulary recognition model training method and system

Abstract

The present invention discloses a context-aware vocabulary recognition model training method and system, which relates to the technical field of vocabulary definition recognition, and comprises the following steps: constructing a vocabulary word library and constructing a triple table based on vocabulary definitions; performing semantic recognition on sample articles using a semantic recognition model to obtain correct semantics, and analyzing the vocabulary associations between different words based on the correct semantics; obtaining the context of target vocabulary and extracting article segmentation of the context; analyzing the association ratio of vocabulary definitions based on article segmentation and the positions of article segmentation, then recognizing the vocabulary definitions of target vocabulary, and detecting whether the vocabulary recognition model is effective; the present invention is used to solve the problem that existing vocabulary definition recognition technology still recognizes vocabulary in a single field, resulting in deviation in judgment results and slightly insufficient vocabulary recognition accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of vocabulary definition recognition, and in particular to a context-aware vocabulary recognition model training method and system. Background Art

[0002] Lexical definition recognition technology refers to a natural language processing technology that aims to automatically identify and extract the definitional descriptions of specific words or terms from text. Its core goal is to analyze the context, determine whether the text contains a clear definition of a certain word, and accurately extract the content of the definition.

[0003] Existing vocabulary definition recognition technologies usually make judgments based on multiple keywords before and after the vocabulary. Chinese is profound and extensive. Some words in Chinese have multiple meanings. The meanings they represent are different in different contexts and fields. Existing vocabulary definition recognition technologies usually judge the definition of vocabulary based on the field to which the keywords belong. When analyzing by field, if the article is not a discussion of a single field, but exists in multiple fields, the judgment result will be biased, resulting in a slightly insufficient accuracy rate for vocabulary recognition. Today, the demand for AI intelligence is getting higher and higher. In AI question-answering, it is necessary to have an in-depth understanding of the vocabulary proposed by the user, rather than conventional error correction, that is, it is necessary to understand the specific meaning of the vocabulary. The meaning of expression, for example, in the patent application with publication number CN115563988A, a semantic recognition method for special vocabulary in the field of automotive finance is disclosed. This solution is to recognize the semantics of vocabulary in a single field, which is used to improve the intention understanding of AI intelligent voice in the automotive industry. However, the existing AI intelligent voice almost has no single-field question and answer, but multi-field or even comprehensive interactive question and answer. Therefore, the limitation of this method is too high, and AI intelligent voice usually does not carry a recognition model in each field. The existing vocabulary definition recognition technology also has the problem of recognizing vocabulary in a single field, resulting in deviations in the judgment results and slightly insufficient accuracy of vocabulary recognition. Summary of the Invention

[0004] The present invention aims to solve one of the technical problems in the prior art to at least a certain extent, by constructing a vocabulary thesaurus and constructing a triple table based on vocabulary definitions, then using a semantic recognition model to perform semantic recognition on sample articles to obtain correct semantics, and then constructing a vocabulary recognition model, analyzing the vocabulary association between different words based on the correct semantics, obtaining the context of the target vocabulary, performing word segmentation on the context, extracting article segments of the context, and analyzing the association ratio of the vocabulary definition based on the ambiguous association table of the target vocabulary and the article segments and the positions of the article segments, and finally identifying the vocabulary definition of the target vocabulary based on the association ratio, and then detecting whether the vocabulary recognition model is effective, so as to solve the problem that the existing vocabulary definition recognition technology still recognizes vocabulary in a single field, resulting in deviations in the judgment results and slightly insufficient accuracy of vocabulary recognition.

[0005] To achieve the above objectives, in a first aspect, the present application provides a context-aware vocabulary recognition model training method, comprising the following steps:

[0006] Build a vocabulary database and a triple table based on vocabulary definitions;

[0007] Use the semantic recognition model to perform semantic recognition on the sample article to obtain the correct semantics, and analyze the lexical relevance between different words based on the correct semantics;

[0008] Obtain the context of the target vocabulary, perform word segmentation on the context, and extract the article word segmentation of the context;

[0009] The correlation ratio of vocabulary definitions is analyzed based on the article segmentation and the position of the article segmentation. The vocabulary definition of the target vocabulary is then identified based on the correlation ratio, and the effectiveness of the vocabulary recognition model is tested.

[0010] Furthermore, constructing a vocabulary word library and constructing a triple table based on vocabulary definitions includes the following sub-steps:

[0011] Build a vocabulary database to include all existing words;

[0012] Obtaining a vocabulary definition of a word, which is the meaning of the word in a dictionary or encyclopedia. A word can have different vocabulary definitions.

[0013] Constructing a triple table, wherein the header of the triple table includes entity 1, relationship and entity 2;

[0014] Fill in the words into entity 1, fill in the definitions at the relationships, fill in the vocabulary definitions corresponding to entity 1 into entity 2, and fill different vocabulary definitions into different cells in entity 2.

[0015] Furthermore, the semantic recognition model is used to perform semantic recognition on the sample article to obtain the correct semantics. The lexical relevance analysis between different words based on the correct semantics includes the following sub-steps:

[0016] Use the semantic recognition model to perform semantic recognition on sample articles and obtain the correct semantics;

[0017] Build a vocabulary recognition model to analyze the lexical relevance between different words based on correct semantics.

[0018] Furthermore, the semantic recognition model is used to perform semantic recognition on the sample article. Obtaining the correct semantics includes the following sub-steps:

[0019] Mark words with non-unique lexical definitions as ambiguous words;

[0020] Periodicals, news reports, magazines, and online articles are collectively referred to as sample articles. For any ambiguous word, a total of the first number of sample articles are extracted from the Internet, and the sample articles must contain ambiguous words;

[0021] Mark the ambiguous words in the sample articles, identify the marked ambiguous words through the existing semantic recognition model, and annotate the ambiguous words based on the semantics of the words obtained by recognition;

[0022] The developer determines whether the semantics of the annotated words are correct. If so, a correct annotation signal is output; otherwise, an incorrect annotation signal is output.

[0023] Count the number of correctly labeled signals and incorrectly labeled signals, mark them as A and B respectively, calculate A / (A+B) to get the semantic model error rate;

[0024] The developer modifies the semantics of the words that output the wrongly labeled signal, and marks the semantics of the words that output the correctly labeled signal and the semantics of the words modified by the developer as correct semantics.

[0025] Furthermore, constructing a vocabulary recognition model to analyze the vocabulary relevance between different words based on correct semantics includes the following sub-steps:

[0026] Constructing a vocabulary recognition model; extracting keywords from sample articles based on keyword extraction technology and marking them as sample keywords, wherein the sample keywords do not include stop words;

[0027] Output the number of correctly labeled signals and incorrectly labeled signals in statistical journals, marked as A1 and B1 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical news reports, marked as A2 and B2 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical magazines, marked as A3 and B3 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical online articles, marked as A4 and B4 respectively;

[0028] Proofread journals, news reports, magazines, and online articles using document proofreading technology, count the number of words in journals, news reports, magazines, and online articles that need proofreading, and mark them as RJ, RN, RM, and RO respectively;

[0029] Count the number of words in journals, news reports, magazines, and online articles, labeled NWJ, NWN, NWM, and NWO respectively;

[0030] For any sample keyword, mark it as the target keyword, count the number of times the target keyword appears in all sample articles, mark it as the target number, and represent it with the symbol C;

[0031] By formula

[0032] Calculate the lexical correlation between the target keyword and the ambiguous word, where T is the lexical correlation and α is the reference coefficient;

[0033] Count the lexical correlations between each sample keyword and ambiguous words;

[0034] An ambiguous association table is constructed for each ambiguous word. Sample keywords that have lexical associations with the ambiguous word and the corresponding lexical associations are entered into the ambiguous association table. Different lexical definitions of the ambiguous word each occupy a column in the ambiguous association table, and the lexical association is recorded in the cell belonging to the lexical definition with the same semantics as the correct one.

[0035] Furthermore, the context of the target word is obtained and the context is segmented. The article segmentation of the context extraction includes the following sub-steps:

[0036] When identifying content, if an ambiguous word is detected, the ambiguous word is marked as the target word, the paragraph containing the target word is marked as the target paragraph, and the paragraph before the target paragraph and the paragraph after the target paragraph are marked as the previous paragraph and the following paragraph respectively;

[0037] The previous paragraph, target paragraph and following paragraph are segmented, i.e. keyword extraction, while stop words are removed and the keywords obtained by segmentation are marked as article segmentation.

[0038] Furthermore, the correlation ratio of vocabulary definitions is analyzed based on the article segmentation and the position of the article segmentation, and the vocabulary definition of the target vocabulary is identified based on the correlation ratio. The effectiveness of the vocabulary recognition model is tested, which includes the following sub-steps:

[0039] Based on the ambiguous association table of the target vocabulary, the association ratio of the vocabulary definition is analyzed based on the article segmentation and the position of the article segmentation;

[0040] The vocabulary definitions of the target vocabulary are identified based on the association ratio, and then the effectiveness of the vocabulary identification model is tested.

[0041] Furthermore, based on the ambiguous association table of the target vocabulary, analyzing the association ratio of the vocabulary definition based on the article segmentation and the position of the article segmentation includes the following sub-steps:

[0042] Find the ambiguous association table of the target vocabulary and mark it as the target association table;

[0043] Sort the article words and get P in the order from front to back. i , where i is a non-zero natural number and i is the serial number of P, and the P of the target vocabulary is searched. i , the P of the target vocabulary i The serial number i is marked as j;

[0044] For any vocabulary definition of the target vocabulary, search P i The vocabulary association in the target association table is marked as S i , if there is no lexical association, then S i is 0, and S j Set to 0;

[0045] By formula Calculate the relevance ratio of vocabulary definitions, where K is the relevance ratio and max(i) is the maximum value of i.

[0046] Furthermore, the vocabulary definition of the target vocabulary is identified based on the association ratio, and then the effectiveness of the vocabulary identification model is tested, which includes the following sub-steps:

[0047] Calculate the correlation ratios of different vocabulary definitions, find the maximum value, and mark it as the maximum ratio;

[0048] Mark the target vocabulary based on the vocabulary definition corresponding to the largest proportion;

[0049] The ambiguous words in the sample articles are identified by the vocabulary recognition model, and the semantic model error rate of the vocabulary recognition model is calculated, which is marked as the recognition model error rate;

[0050] The recognition model error rate is compared with the semantic model error rate. If the recognition model error rate is less than the semantic model error rate, a model valid signal is output; otherwise, a model invalid signal is output.

[0051] In a second aspect, the present application provides a context-aware vocabulary recognition model training system, comprising a vocabulary building module, a vocabulary association analysis module, an article word segmentation module, and a vocabulary recognition module; the vocabulary building module, the vocabulary association analysis module, and the article word segmentation module are respectively connected to the vocabulary recognition module data;

[0052] The vocabulary building module is used to build a vocabulary vocabulary and simultaneously build a triple table based on vocabulary definitions;

[0053] The vocabulary association analysis module is used to perform semantic recognition on the sample article using the semantic recognition model to obtain the correct semantics, and analyze the vocabulary association between different words based on the correct semantics;

[0054] The article segmentation module is used to obtain the context of the target vocabulary, perform segmentation processing on the context, and extract the article segmentation of the context;

[0055] The vocabulary recognition module is used to analyze the correlation ratio of vocabulary definitions based on article segmentation and the position of article segmentation, then identify the vocabulary definition of the target vocabulary based on the correlation ratio, and detect whether the vocabulary recognition model is effective.

[0056] The beneficial effects of the present invention are as follows: the present invention constructs a vocabulary word library, constructs a triple table based on vocabulary definitions, and then uses a semantic recognition model to perform semantic recognition on sample articles to obtain correct semantics, and then constructs a vocabulary recognition model to analyze the vocabulary associations between different words based on the correct semantics. The advantage is that the construction of the triple table is to facilitate the search for ambiguous words, and then when analyzing vocabulary associations, the vocabulary associations are corrected based on the rigor of the words used in different types of sample articles, that is, the results of document proofreading. Although it is impossible to analyze the associations between all words and the vocabulary definitions to be analyzed through the vocabulary association analysis of sample articles, it is possible to screen out most of the keywords that are associated with the vocabulary definitions to be analyzed. Then, combined with a large number of keywords in the context for analysis, the vocabulary definitions of the words can be determined, thereby improving the accuracy and comprehensiveness of vocabulary recognition.

[0057] The present invention obtains the context of the target vocabulary, performs word segmentation on the context, extracts the article word segments of the context, and analyzes the correlation ratio of the vocabulary definition based on the ambiguous association table of the target vocabulary and the article word segments and the positions of the article word segments. Finally, the vocabulary definition of the target vocabulary is identified based on the correlation ratio, and then whether the vocabulary recognition model is effective is detected. The advantage is that, under normal circumstances, the closer the words are to the vocabulary to be analyzed, the stronger the correlation effect on the vocabulary to be analyzed. Therefore, combining the analysis of the correlation ratio of the vocabulary definition with the position of the article word segments can effectively enhance the accuracy of vocabulary recognition in articles in different fields, thereby improving the precision and rationality of vocabulary recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a functional block diagram of the system of the present invention;

[0059] Figure 2 is a flowchart of the steps of the semantic recognition strategy of the present invention;

[0060] Figure 3 is a flow chart of the steps of the method of the present invention;

[0061] Figure 4 Schematic diagram of the structure of the electronic device of the present invention. DETAILED DESCRIPTION

[0062] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0063] Example 1, please refer to Figure 1 As shown, the present application provides a context-aware vocabulary recognition model training system, including a vocabulary building module, a vocabulary association analysis module, an article word segmentation module and a vocabulary recognition module; the vocabulary building module, the vocabulary association analysis module and the article word segmentation module are respectively connected to the vocabulary recognition module data;

[0064] The vocabulary building module is used to build a vocabulary vocabulary and construct a triple table based on the vocabulary definition;

[0065] The vocabulary building module is configured with a vocabulary building strategy, which includes:

[0066] Build a vocabulary database to include all existing words;

[0067] Obtain the vocabulary definition of a word. The vocabulary definition is the meaning of the word in the dictionary and encyclopedia. A word can have different vocabulary definitions.

[0068] Construct a triple table. The header of the triple table includes entity 1, relationship, and entity 2.

[0069] Fill the words into entity 1, fill in the definitions at all relationships, fill the vocabulary definitions corresponding to entity 1 into entity 2, and fill different vocabulary definitions into different cells in entity 2;

[0070] In actual applications, the triple table only stores words that have more than one lexical definition. Some data in the triple table is shown in Table 1 below:

[0071] Table 1 Partial data of triple table

[0072]

[0073] The triple table is constructed to make it easier to find ambiguous words later.

[0074] The vocabulary association analysis module is used to perform semantic recognition on the sample article using the semantic recognition model to obtain the correct semantics and analyze the vocabulary association between different words based on the correct semantics. The vocabulary association analysis module includes a semantic recognition unit and an association analysis unit.

[0075] See also Figure 2 As shown, the semantic recognition unit is used to perform semantic recognition on the sample article using the semantic recognition model to obtain the correct semantics;

[0076] The semantic recognition unit is configured with a semantic recognition strategy, which includes:

[0077] Mark words with non-unique lexical definitions as ambiguous words;

[0078] Periodicals, news reports, magazines, and online articles are collectively referred to as sample articles. For any ambiguous word, a total of the first number of sample articles are extracted from the Internet, and the sample articles must contain ambiguous words;

[0079] In practical applications, under normal circumstances, the publication of journals and news reports will be strictly reviewed, and their wording is very standardized. However, magazines and online articles are not standardized enough in wording, and there may be some incorrect words, which leads to correlation between words that do not have any correlation. Therefore, it is necessary to eliminate errors in the correlation analyzed in magazines and online articles, that is, the multiplication of (A3+B3) and (A4+B+) in the correlation analysis strategy; taking the ambiguous word apple as an example, the meaning of apple includes two meanings: Apple Company and Apple Fruit. Search for a first number of sample articles. In this embodiment, the first number is set to 400, and 100 different sample articles are searched respectively, and all 400 sample articles must contain the ambiguous word "apple";

[0080] Mark the ambiguous words in the sample articles, identify the marked ambiguous words through the existing semantic recognition model, and annotate the ambiguous words based on the semantics of the words obtained by recognition;

[0081] The developer determines whether the semantics of the annotated words are correct. If so, a correct annotation signal is output; otherwise, an incorrect annotation signal is output.

[0082] Count the number of correctly labeled signals and incorrectly labeled signals, mark them as A and B respectively, calculate A / (A+B) to get the semantic model error rate;

[0083] The developer modifies the semantics of the words that output the incorrectly labeled signal, and marks the semantics of the words that output the correctly labeled signal and the semantics of the words modified by the developer as the correct semantics;

[0084] In practical applications, the use of existing semantic recognition models to identify ambiguous words is both to reduce the workload of developers in the early stages of model training and to confirm the accuracy of the existing semantic recognition model in recognizing lexical semantics. A large amount of human resources is required in the early stages of training to manually judge and label the correct semantics. This is because AI intelligent question-answering technology requires an in-depth understanding of the text content input by users, including the meaning of each word. The meaning of these words changes with the influence of many factors such as usage scenarios, tone, and contextual coherence. The understanding process of AI intelligent question-answering technology is itself an anthropomorphic process, learning human cognition, emotions, and spoken language. This requires developers to invest a large amount of human resources in the early stages of model training to correct errors. Therefore, there is no waste of human resources here, which is essential for the training of semantic vocabulary recognition models. When counting the number of correctly labeled signals and incorrectly labeled signals, all sample articles of the first number are counted. When counting, refer to A1 to A4 and B1 to B4 in the association analysis strategy for statistics. The sum of A1 to A4 is A, and the sum of B1 to B4 is B. Figure 2 The statistical process of A and B is also shown in the figure. In this example, the statistical results of A and B are 1268 and 102 respectively, and the calculated semantic model error rate is 0.0745. The calculated result is rounded to four decimal places and saved as a percentage, that is, the semantic model error rate is 7.45%.

[0085] The association analysis unit is used to build a vocabulary recognition model and analyze the lexical associations between different words based on correct semantics;

[0086] The association analysis unit is configured with an association analysis strategy, which includes:

[0087] Build a vocabulary recognition model; extract keywords from sample articles based on keyword extraction technology and mark them as sample keywords. Sample keywords do not include stop words. For stop words, refer to the stop word library.

[0088] Output the number of correctly labeled signals and incorrectly labeled signals in statistical journals, marked as A1 and B1 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical news reports, marked as A2 and B2 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical magazines, marked as A3 and B3 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical online articles, marked as A4 and B4 respectively;

[0089] In actual applications, existing keyword extraction technology is used to extract sample keywords and remove stop words. In this embodiment, statistics show that A1 and B1 are 364 and 12 respectively, A2 and B2 are 331 and 10 respectively, A3 and B3 are 338 and 34 respectively, and A4 and B4 are 235 and 46 respectively.

[0090] Proofread journals, news reports, magazines, and online articles using document proofreading technology, count the number of words in journals, news reports, magazines, and online articles that need proofreading, and mark them as RJ, RN, RM, and RO respectively;

[0091] Count the number of words in journals, news reports, magazines, and online articles, labeled NWJ, NWN, NWM, and NWO respectively;

[0092] In actual applications, document proofreading technology is an existing technology and is used to correct errors in the words used in documents. However, document proofreading technology also has deviations, so error elimination is required. This has been explained in the aforementioned sample articles and will not be explained in detail in this embodiment. The statistics of RJ, RN, RM, and RO are 286, 328, 1226, and 1539, respectively. The number of words in journals, news reports, magazines, and online articles is the number of sample keywords extracted from journals, news reports, magazines, and online articles, excluding stop words. The statistics of NWJ, NWN, NWM, and NWO are 12868, 13426, 11729, and 12923, respectively. Since statistics are performed on all sample articles, the values ​​of RJ, RN, RM, RO, NWJ, NWN, NWM, and NWO are all relatively large.

[0093] For any sample keyword, mark it as the target keyword, count the number of times the target keyword appears in all sample articles, mark it as the target number, and represent it with the symbol C;

[0094] By formula

[0095]

[0096] Calculate the lexical correlation between the target keyword and the ambiguous word, where T is the lexical correlation and α is the reference coefficient;

[0097] Count the lexical correlations between each sample keyword and ambiguous words;

[0098] In actual applications, usually, the more the sample keywords and target keywords appear in the sample articles, the stronger their correlation is, so their relationship is positively correlated, and the value obtained by multiplying the two is larger. Therefore, setting a reference coefficient to reduce the value of T is convenient for data observation and has no practical significance. In this embodiment, α is set to 100000. Assuming that the target keyword is "mobile phone", the ambiguous word is "apple", and the vocabulary is defined as "company", the statistical result shows that the number of times "mobile phone" appears in the sample articles is 1322, that is, the target number C is 1322. The formula is used to calculate the vocabulary correlation between "mobile phone" and "apple" when it is defined as "company" to be 17.3563, and the calculation result is rounded to four decimal places;

[0099] Construct an ambiguous association table for each ambiguous word. Enter the sample keywords that have lexical associations with the ambiguous word and the corresponding lexical associations into the ambiguous association table. Different lexical definitions of the ambiguous word occupy a column in the ambiguous association table. The lexical association is recorded in the cell belonging to the lexical definition with the same semantics as the correct one.

[0100] In actual applications, taking the ambiguous association table of "apple" as an example, Table 2 below shows part of the data of the ambiguous association table of "apple" to reveal and illustrate the specific storage format of the ambiguous association table;

[0101] Table 2 Partial data of the ambiguous association table of “apple”

[0102]

[0103] Among them, if the cell contains 0, it means that the sample keyword has no correlation with the corresponding vocabulary definition, while there is a correlation of 0.0083 between mobile phone and the vocabulary definition of fruit. This is because the content described in the sample article is not conventional, especially online articles, which are imaginative and cover many fields. Therefore, there will be a small correlation between the two unrelated ones, but because the value is small, it has no effect on the final judgment.

[0104] The article segmentation module is used to obtain the context of the target vocabulary, perform word segmentation on the context, and extract the article segmentation of the context;

[0105] The article segmentation module is configured with an article segmentation strategy, which includes:

[0106] When identifying content, if an ambiguous word is detected, the ambiguous word is marked as the target word, the paragraph containing the target word is marked as the target paragraph, and the paragraph before the target paragraph and the paragraph after the target paragraph are marked as the previous paragraph and the following paragraph respectively;

[0107] Perform word segmentation on the previous paragraph, target paragraph, and following paragraph, i.e. extract keywords, remove stop words, and mark the keywords obtained from the word segmentation as article segmentation;

[0108] In practical applications, assuming that a three-paragraph article is analyzed, if the ambiguous word is in the second paragraph, the first paragraph will be marked as the previous paragraph, the second paragraph will be marked as the target paragraph, and the third paragraph will be marked as the following paragraph. If the ambiguous word is in the first paragraph, the first paragraph will be marked as the target paragraph, the second paragraph will be marked as the following paragraph, and there will be no previous paragraph; the word segmentation processing is performed using the existing Jieba word segmentation technology.

[0109] The vocabulary recognition module is used to analyze the correlation ratio of vocabulary definitions based on the article segmentation and the position of the article segmentation, then identify the vocabulary definition of the target vocabulary based on the correlation ratio, and detect whether the vocabulary recognition model is effective; the vocabulary recognition module includes a correlation ratio analysis unit and a vocabulary recognition unit;

[0110] The correlation ratio analysis unit is used to analyze the correlation ratio of vocabulary definitions based on the ambiguous correlation table of the target vocabulary and the position of the article segmentation words;

[0111] The correlation ratio analysis unit is configured with correlation ratio analysis strategies, which include:

[0112] Find the ambiguous association table of the target vocabulary and mark it as the target association table;

[0113] Sort the article words and get P in the order from front to back. i , where i is a non-zero natural number and i is the serial number of P, and the P of the target vocabulary is searched. i , the P of the target vocabulary i The serial number i is marked as j;

[0114] For any vocabulary definition of the target vocabulary, search P i The vocabulary association in the target association table is marked as S i , if there is no lexical association, then S i is 0, and S j Set to 0;

[0115] By formula Calculate the relevance ratio of vocabulary definitions, where K is the relevance ratio and max(i) is the maximum value of i;

[0116] In actual application, if the target word is "apple", the ambiguous association table of "apple" is the target association table. Due to the large amount of data in the ambiguous association table, it is impossible to show it in detail in this embodiment. Therefore, only the final calculation result of the association ratio is given; sort the article segmentation to get Pi , where 1≤i≤38, and the target word is located at P 12 , i.e. j=12; P i The distance to the target word. Generally speaking, the closer the distance to the target word, the greater the influence on the definition of the target word. First, we analyze the word definition of "apple" as "company". The calculated correlation ratio of the word definition "company" is 284, and the calculated result is rounded to an integer. Similarly, the calculated correlation ratio of the word definition of "fruit" is 38.

[0117] The vocabulary recognition unit is used to identify the vocabulary definition of the target vocabulary based on the association ratio, and then detect whether the vocabulary recognition model is effective;

[0118] The vocabulary recognition unit is configured with a vocabulary recognition strategy, which includes:

[0119] Calculate the correlation ratios of different vocabulary definitions, find the maximum value, and mark it as the maximum ratio;

[0120] Mark the target vocabulary based on the vocabulary definition corresponding to the largest proportion;

[0121] The ambiguous words in the sample articles are identified by the vocabulary recognition model, and the semantic model error rate of the vocabulary recognition model is calculated, which is marked as the recognition model error rate;

[0122] Compare the recognition model error rate with the semantic model error rate. If the recognition model error rate is less than the semantic model error rate, output a model valid signal; otherwise, output a model invalid signal.

[0123] In actual application, the comparison shows that 284 is greater than 38. Therefore, in the paragraph analyzed this time, the target vocabulary is defined as "company". j The corresponding word is labeled "Apple". The trained vocabulary recognition model is tested using the same sample articles. The recognition model error rate is calculated using the same steps as the semantic model error rate. The error rate of the recognition model is 2.68%, and the corresponding accuracy is 97.32%. The comparison shows that the error rate of the recognition model is smaller than the error rate of the semantic model, and the model is output as a valid signal, indicating that the vocabulary recognition model has improved the accuracy of vocabulary definition recognition.

[0124] Example 2, please refer to Figure 3 As shown, the present application provides a context-aware vocabulary recognition model training method, comprising the following steps:

[0125] Step S1: Build a vocabulary database and build a triple table based on the vocabulary definition. Step S1 includes the following sub-steps:

[0126] Step S101, constructing a vocabulary database to include all existing words;

[0127] Step S102: Obtain the vocabulary definition of the word. The vocabulary definition is the meaning of the word in the dictionary and encyclopedia. A word can have different vocabulary definitions.

[0128] Step S103: construct a triple table, the header of which includes entity 1, relationship, and entity 2;

[0129] Step S104: Fill the words into entity 1, fill in the definitions at all relationships, fill the vocabulary definitions corresponding to entity 1 into entity 2, and fill different vocabulary definitions into different cells in entity 2;

[0130] Step S2: Using a semantic recognition model to perform semantic recognition on the sample article, obtain correct semantics, and analyze the lexical relevance between different words based on the correct semantics. Step S2 includes the following sub-steps:

[0131] Step S201: Using a semantic recognition model to perform semantic recognition on a sample article to obtain correct semantics;

[0132] Step S201 includes the following sub-steps:

[0133] Step S201.1, marking words with non-unique lexical definitions as ambiguous words;

[0134] Step S201.2: Periodicals, news reports, magazines, and online articles are collectively referred to as sample articles. For any ambiguous word, a first number of sample articles are extracted from the Internet, and the sample articles must contain the ambiguous word.

[0135] Step S201.3: Mark the ambiguous words in the sample article, identify the marked ambiguous words using an existing semantic recognition model, and annotate the ambiguous words based on the semantics of the words obtained by the recognition;

[0136] Step S201.4: The developer determines whether the semantics of the annotated word are correct. If so, a correct annotation signal is output; otherwise, an incorrect annotation signal is output.

[0137] Step S201.5: Count and output the number of correctly labeled signals and incorrectly labeled signals, label them as A and B respectively, and calculate A / (A+B) to obtain the semantic model error rate;

[0138] Step S201.6: The developer modifies the semantics of the word that outputs the signal with the error mark, and marks the semantics of the word that outputs the signal with the correct mark and the semantics of the word after the developer's modification as the correct semantics;

[0139] Step S202: constructing a vocabulary recognition model to analyze the vocabulary relevance between different words based on correct semantics;

[0140] Step S202 includes the following sub-steps:

[0141] Step S202.1, constructing a vocabulary recognition model; extracting keywords from sample articles based on keyword extraction technology, marking them as sample keywords, the sample keywords do not include stop words, which can be referenced from a stop word library;

[0142] Step S202.2: Output the number of correctly labeled signals and incorrectly labeled signals in statistical journals, labeled A1 and B1 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical news reports, labeled A2 and B2 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical magazines, labeled A3 and B3 respectively; and output the number of correctly labeled signals and incorrectly labeled signals in statistical online articles, labeled A4 and B4 respectively;

[0143] Step S202.3: Proofread the journals, news reports, magazines, and online articles using document proofreading technology, count the number of words in the journals, news reports, magazines, and online articles that need proofreading, and mark them as RJ, RN, RM, and RO respectively;

[0144] Step S202.4, counting the number of words in journals, news reports, magazines, and online articles, marking them as NWJ, NWN, NWM, and NWO respectively;

[0145] Step S202.5: For any sample keyword, mark it as a target keyword, and count the number of times the target keyword appears in all sample articles, marking it as the target number, represented by symbol C;

[0146] Step S202.6, by formula

[0147] Calculate the lexical correlation between the target keyword and the ambiguous word, where T is the lexical correlation and α is the reference coefficient;

[0148] Step S202.7, counting the lexical relevance between each sample keyword and the ambiguous word;

[0149] Step S202.8: Construct an ambiguity association table for each ambiguous word. Enter the sample keywords that have lexical associations with the ambiguous word and their corresponding lexical associations into the ambiguity association table. Each lexical definition of the ambiguous word occupies a column in the ambiguity association table. Lexical associations are recorded in the cells corresponding to the lexical definitions with the same semantic meaning as the correct word.

[0150] Step S3, obtaining the context of the target vocabulary, performing word segmentation processing on the context, and extracting the article word segmentation of the context; Step S3 includes the following sub-steps:

[0151] Step S301: When identifying content, if an ambiguous word is detected, the ambiguous word is marked as a target word, the paragraph containing the target word is marked as a target paragraph, and the paragraph before the target paragraph and the paragraph after the target paragraph are marked as the previous paragraph and the following paragraph, respectively.

[0152] Step S302: perform word segmentation on the previous paragraph, target paragraph, and following paragraph, i.e., extract keywords, remove stop words, and mark the keywords obtained by word segmentation as article segmentation;

[0153] Step S4, analyzing the association ratio of vocabulary definitions based on the article segmentation and the position of the article segmentation, then identifying the vocabulary definition of the target vocabulary based on the association ratio, and detecting whether the vocabulary recognition model is effective; Step S4 includes the following sub-steps:

[0154] Step S401: Analyze the association ratio of vocabulary definitions based on the ambiguous association table of the target vocabulary and the article segmentation and the position of the article segmentation;

[0155] Step S401 includes the following sub-steps:

[0156] Step S401.1, search the ambiguous association table of the target vocabulary and mark it as the target association table;

[0157] Step S401.2, sort the article words and get the P i , where i is a non-zero natural number and i is the serial number of P, and the P of the target vocabulary is searched. i , the P of the target vocabulary i The serial number i is marked as j;

[0158] Step S401.3: Search for any vocabulary definition of the target vocabulary. i The vocabulary association in the target association table is marked as S i , if there is no lexical association, then S i is 0, and S j Set to 0;

[0159] Step S401.4, by formula Calculate the relevance ratio of vocabulary definitions, where K is the relevance ratio and max(i) is the maximum value of i;

[0160] Step S402, identifying the vocabulary definition of the target vocabulary based on the association ratio, and then detecting whether the vocabulary recognition model is effective;

[0161] Step S402 includes the following sub-steps:

[0162] Step S402.1, calculate the correlation ratios of different vocabulary definitions, find the maximum value among them, and mark it as the maximum ratio;

[0163] Step S402.2: annotate the target vocabulary based on the vocabulary definition corresponding to the largest proportion;

[0164] Step S402.3, identifying ambiguous words in the sample article using the vocabulary recognition model, and calculating the semantic model error rate of the vocabulary recognition model, which is marked as the recognition model error rate;

[0165] Step S402.4, compare the recognition model error rate with the semantic model error rate. If the recognition model error rate is smaller than the semantic model error rate, output a model valid signal; otherwise, output a model invalid signal.

[0166] Example 3, please refer to Figure 4 As shown, Figure 4 A schematic diagram of the structure of an electronic device is provided, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus. The memory stores computer-readable instructions, and the processor can call the instructions in the memory. When the computer-readable instructions are executed by the processor, the steps in a context-aware vocabulary recognition model training method are executed to achieve the following functions: constructing a vocabulary word library and simultaneously constructing a triple table based on vocabulary definitions; using a semantic recognition model to perform semantic recognition on sample articles to obtain correct semantics, and analyzing the lexical relevance between different words based on the correct semantics; obtaining the context of the target vocabulary and extracting article segmentation of the context; analyzing the relevance ratio of the vocabulary definition based on the article segmentation and the position of the article segmentation; then identifying the vocabulary definition of the target vocabulary, and testing whether the vocabulary recognition model is effective.

[0167] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0168] Example 4. The present application also provides a computer-readable storage medium. The present application provides a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the above context-aware vocabulary recognition model training method are executed to achieve the following functions: construct a vocabulary word library and construct a triple table based on the vocabulary definition; use a semantic recognition model to perform semantic recognition on sample articles to obtain correct semantics, and analyze the vocabulary association between different words based on the correct semantics; obtain the context of the target vocabulary and extract the article segmentation of the context; analyze the association ratio of the vocabulary definition based on the article segmentation and the position of the article segmentation, then identify the vocabulary definition of the target vocabulary, and detect whether the vocabulary recognition model is valid.

[0169] Through the description of the above embodiments, the embodiments of the present invention can be provided as methods, systems, or computer program products. Based on this understanding, the essence of the above technical solutions or the portion that contributes to the prior art can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for enabling a computer device (such as a personal computer, server, or network device) to execute the methods described in various embodiments or certain portions of the embodiments.

[0170] In the embodiments provided in this application, it should be understood that the disclosed system or method can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of systems, modules and units can be electrical, mechanical or other forms.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A context-aware vocabulary recognition model training method, characterized in that: The steps include: Build a vocabulary database and a triple table based on vocabulary definitions; Use the semantic recognition model to perform semantic recognition on the sample article to obtain the correct semantics, and analyze the lexical relevance between different words based on the correct semantics; Obtain the context of the target vocabulary, perform word segmentation on the context, and extract the article word segmentation of the context; Analyze the correlation ratio of vocabulary definitions based on the article segmentation and the position of the article segmentation, then identify the vocabulary definition of the target vocabulary based on the correlation ratio and test whether the vocabulary recognition model is effective; Building a vocabulary recognition model to analyze the lexical relevance between different words based on correct semantics includes the following sub-steps: Constructing a vocabulary recognition model; extracting keywords from sample articles based on keyword extraction technology and marking them as sample keywords, wherein the sample keywords do not include stop words; Output the number of correctly labeled signals and incorrectly labeled signals in statistical journals, marked as A1 and B1 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical news reports, marked as A2 and B2 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical magazines, marked as A3 and B3 respectively; output the number of correctly labeled signals and incorrectly labeled signals in statistical online articles, marked as A4 and B4 respectively; Proofread journals, news reports, magazines, and online articles using document proofreading technology, count the number of words in journals, news reports, magazines, and online articles that need proofreading, and mark them as RJ, RN, RM, and RO respectively; Count the number of words in journals, news reports, magazines, and online articles, labeled NWJ, NWN, NWM, and NWO respectively; For any sample keyword, mark it as the target keyword, count the number of times the target keyword appears in all sample articles, mark it as the target number, and represent it with the symbol C; By formula Calculate the lexical correlation between the target keyword and the ambiguous word, where T is the lexical correlation and α is the reference coefficient; Count the lexical correlations between each sample keyword and ambiguous words; Construct an ambiguous association table for each ambiguous word. Enter the sample keywords that have lexical associations with the ambiguous word and the corresponding lexical associations into the ambiguous association table. Different lexical definitions of the ambiguous word occupy a column in the ambiguous association table. The lexical association is recorded in the cell belonging to the lexical definition with the same semantics as the correct one. Based on the ambiguous association table of the target vocabulary, analyzing the association ratio of vocabulary definitions based on the article segmentation and the position of the article segmentation includes the following sub-steps: Find the ambiguous association table of the target vocabulary and mark it as the target association table; Sort the article words and get P in the order from front to back. i , where i is a non-zero natural number and i is the serial number of P, and the P of the target vocabulary is searched. i , the P of the target vocabulary i The serial number i is marked as j; For any vocabulary definition of the target vocabulary, search P i The vocabulary association in the target association table is marked as S i , if there is no lexical association, then S i is 0, and S j Set to 0; By formula Calculate the relevance ratio of vocabulary definitions, where K is the relevance ratio and max(i) is the maximum value of i.

2. A context-aware vocabulary recognition model training method according to claim 1, characterized in that: Building a vocabulary database and constructing a triple table based on vocabulary definitions includes the following sub-steps: Build a vocabulary database to include all existing words; Obtaining a vocabulary definition of a word, which is the meaning of the word in a dictionary or encyclopedia. A word can have different vocabulary definitions. Constructing a triple table, wherein the header of the triple table includes entity 1, relationship and entity 2; Fill in the words into entity 1, fill in the definitions at the relationships, fill in the vocabulary definitions corresponding to entity 1 into entity 2, and fill different vocabulary definitions into different cells in entity 2.

3. The context-aware vocabulary recognition model training method according to claim 2, characterized in that: Use the semantic recognition model to perform semantic recognition on the sample article to obtain the correct semantics. Analyzing the lexical relevance between different words based on the correct semantics includes the following sub-steps: Use the semantic recognition model to perform semantic recognition on sample articles and obtain the correct semantics; Build a vocabulary recognition model to analyze the lexical relevance between different words based on correct semantics.

4. A context-aware vocabulary recognition model training method according to claim 3, characterized in that: Using the semantic recognition model to perform semantic recognition on sample articles, obtaining the correct semantics includes the following sub-steps: Mark words with non-unique lexical definitions as ambiguous words; Periodicals, news reports, magazines, and online articles are collectively referred to as sample articles. For any ambiguous word, a total of the first number of sample articles are extracted from the Internet, and the sample articles must contain ambiguous words; Mark the ambiguous words in the sample articles, identify the marked ambiguous words through the existing semantic recognition model, and annotate the ambiguous words based on the semantics of the words obtained by recognition; The developer determines whether the semantics of the annotated words are correct. If so, a correct annotation signal is output; otherwise, an incorrect annotation signal is output. Count the number of correctly labeled signals and incorrectly labeled signals, mark them as A and B respectively, calculate A / (A+B) to get the semantic model error rate; The developer modifies the semantics of the words that output the wrongly labeled signal, and marks the semantics of the words that output the correctly labeled signal and the semantics of the words modified by the developer as correct semantics.

5. A context-aware vocabulary recognition model training method according to claim 4, characterized in that: Obtain the context of the target word and perform word segmentation on the context. The word segmentation of the article with context extraction includes the following sub-steps: When identifying content, if an ambiguous word is detected, the ambiguous word is marked as the target word, the paragraph containing the target word is marked as the target paragraph, and the paragraph before the target paragraph and the paragraph after the target paragraph are marked as the previous paragraph and the following paragraph respectively; The previous paragraph, target paragraph and following paragraph are segmented, i.e. keyword extraction, while stop words are removed and the keywords obtained by segmentation are marked as article segmentation.

6. A context-aware vocabulary recognition model training method according to claim 5, characterized in that: Analyzing the association ratio of vocabulary definitions based on the article segmentation and the position of the article segmentation, then identifying the vocabulary definition of the target vocabulary based on the association ratio, and testing whether the vocabulary recognition model is effective includes the following sub-steps: Based on the ambiguous association table of the target vocabulary, the association ratio of the vocabulary definition is analyzed based on the article segmentation and the position of the article segmentation; The vocabulary definitions of the target vocabulary are identified based on the association ratio, and then the effectiveness of the vocabulary identification model is tested.

7. A context-aware vocabulary recognition model training method according to claim 6, characterized in that: Identifying the vocabulary definitions of target words based on the association ratio and then testing whether the vocabulary recognition model is effective includes the following sub-steps: Calculate the correlation ratios of different vocabulary definitions, find the maximum value, and mark it as the maximum ratio; Annotate the target vocabulary based on the vocabulary definition corresponding to the largest proportion; The ambiguous words in the sample articles are identified by the vocabulary recognition model, and the semantic model error rate of the vocabulary recognition model is calculated, which is marked as the recognition model error rate; The recognition model error rate is compared with the semantic model error rate. If the recognition model error rate is less than the semantic model error rate, a model valid signal is output; otherwise, a model invalid signal is output.

8. A context-aware vocabulary recognition model training system, used to implement the context-aware vocabulary recognition model training method according to any one of claims 1 to 7, characterized in that: It includes a vocabulary building module, a vocabulary association analysis module, an article word segmentation module and a vocabulary recognition module; the vocabulary building module, the vocabulary association analysis module and the article word segmentation module are respectively connected to the vocabulary recognition module data; The vocabulary building module is used to build a vocabulary vocabulary and simultaneously build a triple table based on vocabulary definitions; The vocabulary association analysis module is used to perform semantic recognition on the sample article using the semantic recognition model to obtain the correct semantics, and analyze the vocabulary association between different words based on the correct semantics; The article segmentation module is used to obtain the context of the target vocabulary, perform segmentation processing on the context, and extract the article segmentation of the context; The vocabulary recognition module is used to analyze the correlation ratio of vocabulary definitions based on article segmentation and the position of article segmentation, then identify the vocabulary definition of the target vocabulary based on the correlation ratio, and detect whether the vocabulary recognition model is effective.

Images