Character ambiguity elimination method and system based on artificial intelligence

Abstract

The invention relates to the field of data processing, in particular to a character ambiguity elimination method and system based on artificial intelligence. The method comprises the following steps: constructing a lexicon, and dividing all reference words into strong correlation words and weak correlation words of target words; removing unintentional words in the answer sentences to obtain effective answer sentences, calculating a distribution coefficient of any word in the effective answer sentences, and constructing word vectors of the effective answer sentences according to the distribution coefficient; clustering the word vectors of all the effective answers to obtain a plurality of clusters, traversing any cluster to obtain reciprocals of the number of the strongly related words in the marked question corresponding to each effective answer, and taking a mean value of all the reciprocals as an ambiguous value of the cluster; and traversing to obtain an ambiguity value of each cluster, constructing an ambiguity value sequence of the contact word to the target word, and calculating ambiguity evaluation to complete ambiguity elimination. According to the technical scheme, the accuracy of text understanding can be improved, and recognition errors caused by character ambiguity are reduced.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and more specifically, to a method and system for eliminating word ambiguity based on artificial intelligence. Background Art

[0002] Disambiguation of text is of great significance in ensuring accurate information transmission, avoiding misunderstandings and disputes, enhancing information value and maintaining language norms. In communication, it enables the recipient of information to accurately understand the intention and ensure smooth and efficient communication. In formal occasions such as law and business, disambiguation can clarify the rights and obligations of all parties, reduce legal risks and maintain the order of cooperation. For knowledge dissemination and news reporting, it helps to accurately convey the core content and enhance the authority and credibility of information. From the perspective of language development, disambiguation encourages people to follow grammatical rules and vocabulary collocation norms, maintain the stability and consistency of language, promote the healthy development of language, and enable it to better serve human communication needs.

[0003] Existing dialogue models mainly rely on context to make inferences and judgments when dealing with ambiguous words. However, in many cases, the meaning of ambiguous words changes depending on the professional field or specific context. In this case, the model lacks a deep understanding and association of vocabulary in different professional fields, resulting in an inability to accurately identify the actual meaning of ambiguous words. The answers output by the model may deviate from the actual needs of users and even be misleading. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes a text ambiguity elimination method and system based on artificial intelligence.

[0005] In a first aspect, the present invention discloses a method for eliminating text ambiguity based on artificial intelligence, comprising: constructing a word library, the word library includes professional words and non-problematic words, taking words other than professional words and non-problematic words in a question sentence as reference words, and dividing all reference words into strong related words and weak related words of a target word, wherein the target word is any professional word; taking any strongly related word as a connecting word, obtaining a marked question sentence containing both the target word and the connecting word, and obtaining an answer sentence corresponding to each marked question sentence, eliminating non-problematic words in the answer sentence to obtain an effective answer sentence, and calculating any strong related word in the effective answer sentence. The distribution coefficient of words is used to construct the word vector of the effective answer sentence according to the distribution coefficient; the word vectors of all effective answer sentences are clustered to obtain several clusters, one cluster corresponds to one type of effective answer sentence, and for any cluster, the reciprocal of the number of strongly related words in the marked question corresponding to each effective answer sentence is obtained by traversing, and the mean of all reciprocals is used as the ambiguity value of the cluster; the ambiguity value of each cluster is obtained by traversing, and the ambiguity value sequence of the connecting word to the target word is constructed, and the ambiguity evaluation is calculated to complete the ambiguity elimination; wherein, the ambiguity evaluation satisfies the relationship: , Indicates connecting words Target word The ambiguous evaluation of Indicates connecting words Target word The ambiguous value sequence of Indicates the number of elements, represents the minimum ambiguous value in the ambiguous value sequence, represents the maximum ambiguous value in the ambiguous value sequence, Represents the normalization function.

[0006] Preferably, the method of dividing all reference words into strongly related words and weakly related words of the target word includes: taking any reference word as the target reference word, calculating the association index between the target reference word and the target word, traversing to obtain the association index between each reference word and the target word, arranging all the association indexes from large to small, and constructing an association index sequence; setting a split point for the association index sequence, dividing the association index sequence into a first subsequence and a second subsequence, and calculating the sequence classification accuracy, iterating the position of the split point, stopping the iteration when the sequence classification accuracy is maximized, and obtaining the final first subsequence and second subsequence; taking the first subsequence and the second subsequence with a larger mean of the association index as a strongly related word, and taking the first subsequence and the second subsequence with a smaller mean of the association index as a weakly related word.

[0007] Preferably, the correlation index satisfies the relationship: , Reference word With target word The correlation index, Indicates that it also contains reference words and target word The number of questions Indicates that it contains the target word The number of questions Indicates contains the target word The number of all professional words in the question.

[0008] Preferably, the sequence classification accuracy satisfies the relationship: , represents the sequence classification accuracy, represents the proportion of the first subsequence in the correlation index sequence, Indicates the proportion of the second subsequence in the correlation index sequence, and denote the first subsequence and the second subsequence respectively, represents the standard deviation, Represents the mean.

[0009] Preferably, the initial position of the segmentation point is between the first correlation index and the second correlation index in the correlation index sequence, the iteration direction is from left to right, and the iteration step length is one correlation index.

[0010] Preferably, the calculation of the distribution coefficient of any word in the valid answer sentence includes: For the same valid answer sentence, obtain the proportion of any word in the valid answer sentence and the total number of words in the valid answer sentence respectively, and take the product of the normalized total number of words and the proportion of any word in the valid answer sentence as the distribution coefficient.

[0011] In a second aspect, the present invention discloses an artificial intelligence-based text ambiguity elimination system, comprising: a processor; and a memory, wherein the memory stores computer instructions, and when the computer instructions are executed by the processor, the system executes the above-mentioned artificial intelligence-based text ambiguity elimination method.

[0012] Beneficial effects of the present invention: The present invention constructs a vocabulary of professional words and non-meaning words, and judges ambiguity by combining the strong and weak correlation of reference words. The connection between the target word and its related words is quantified, and the word vectors of the question tags and the effective answer sentences are used for analysis, which can accurately identify and eliminate irrelevant information and extract valuable lexical features. In particular, based on the calculation and clustering method of word distribution coefficients, the answers can be systematically divided into multiple categories, each category representing a different semantic direction, thereby helping to accurately determine the meaning of the target word.

[0013] By analyzing the ambiguity values ​​of each cluster and combining it with the precise calculation of the correlation index, the ambiguity can be effectively quantitatively evaluated. This method further improves the classification accuracy and segmentation effect by introducing multiple means such as statistical standard deviation and mean difference for optimization. This refined ambiguity elimination mechanism can significantly reduce the understanding deviation caused by unclear semantics in practical applications. It is especially suitable for fields that require high-precision semantic understanding, such as natural language processing, intelligent question-answering systems, and machine translation, and ultimately improves user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The above and other objects, features and advantages of the exemplary embodiments of the present invention will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present invention are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein: Figure 1 It is a flow chart of a method for eliminating text ambiguity based on artificial intelligence according to an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0016] It should be understood that when the terms "first", "second", etc. are used in the claims, descriptions, and drawings of the present invention, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the description and claims of the present invention indicate the presence of the described features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components, and / or their collections.

[0017] The present invention provides a method for eliminating word ambiguity based on artificial intelligence. Figure 1 As shown, a method for eliminating text ambiguity based on artificial intelligence includes steps S1 to S4, which are described in detail below.

[0018] S1, construct a vocabulary, which includes professional words and non-problematic words. The words in the question except professional words and non-problematic words are used as reference words, and all reference words are divided into strong related words and weak related words of the target word.

[0019] It should be noted that some words in the text may be ambiguous, and the degree of ambiguity may vary. In practical applications, some words with specific meanings are usually unambiguous and can be used as effective reference words. These words can provide clear guidance for semantic understanding because of their fixed semantic scope and less context dependence.

[0020] However, some highly ambiguous words may have different interpretations in different contexts, so special care should be taken when using them. When constructing a vocabulary, the weight of such words should be lowered to reduce the interference of ambiguity on semantic analysis.

[0021] In one embodiment, when constructing a vocabulary, those skilled in the art will divide the vocabulary into two categories: professional words and non-pronunciation words. Professional words refer to terms with clear and fixed meanings in a specific field, such as proper nouns and terms in the fields of medicine, law, engineering, etc. These words are essential for understanding and processing texts in the field. Non-pronunciation words refer to those words that have no actual semantic carrying function, such as prepositions, modal particles, conjunctions, etc. They are mainly used to connect sentences or express tone, but do not play a substantial role in the core meaning of the text.

[0022] Take the words in the question except professional words and non-problematic words as reference words, take any professional word as the target word, take any reference word as the target reference word, calculate the association index between the target reference word and the target word, and the association index satisfies the relationship: , Reference word With target word The correlation index, Indicates that it also contains reference words and target word The number of questions Indicates that it contains the target word The number of questions Indicates contains the target word The number of all professional words in the question.

[0023] The frequency of reference words can be constrained. The larger the value of Reference words in question sentences The higher the frequency of occurrence, the more likely the reference word Target word The higher the correlation.

[0024] Reference word Target word The uniqueness constraint of The larger the value of is, the fewer other professional words there are in the question, so the reference word The relevance of relevance.

[0025] The relevance index combines the frequency of the reference word in the questions where the target word appears and the number of other professional words involved in these questions. By constraining the frequency of reference words, it ensures that only reference words that frequently appear in questions related to the target word are given a high relevance. At the same time, by considering the number of professional words in the questions, it avoids the interference of other professional words in the calculation of the relevance between the reference word and the target word, ensuring that the relevance truly reflects the closeness between the reference word and the target word.

[0026] Traverse to obtain the association index of each reference word and the target word, arrange all the association indexes from large to small, and construct an association index sequence. Set a split point for the association index sequence, and divide the association index sequence into the first subsequence and the second subsequence. And calculate the sequence classification accuracy, iterate the position of the split point, stop the iteration when the sequence classification accuracy is the largest, and obtain the final first subsequence and second subsequence; the words with the larger mean of the association index in the first subsequence and the second subsequence are regarded as strongly related words, and the words with the smaller mean of the association index are regarded as weakly related words.

[0027] The initial position of the split point is between the first correlation index and the second correlation index in the correlation index sequence, the iteration direction is from left to right, and the iteration step length is one correlation index.

[0028] The sequence classification accuracy satisfies the relationship: , represents the sequence classification accuracy, represents the proportion of the first subsequence in the correlation index sequence, Indicates the proportion of the second subsequence in the correlation index sequence, and denote the first subsequence and the second subsequence respectively, represents the standard deviation, Represents the mean.

[0029] The classification accuracy not only considers the dispersion of the association index within the subsequence (reflected by the standard deviation) and the mean difference between the subsequences, but also makes the difference between the two subsequences more significant through proportion adjustment. Maximizing the classification accuracy helps to find the most appropriate split point in practical applications, ensuring that each subsequence can effectively reflect the actual association of the vocabulary, and finally distinguishing between strongly related words and weakly related words.

[0030] Exemplarily, the sequence of associated indices is The initial position of the split point is and Between is the first subsequence, is the second subsequence, and the sequence classification accuracy at this time is calculated; after iterating the split point position once, the first subsequence is , the second subsequence becomes , calculate the sequence classification accuracy again. Then iterate the segmentation point position until the calculated sequence classification accuracy is the maximum, and obtain the strongly related words and weakly related words.

[0031] In another embodiment, the strongly related words and the weakly related words of the target word are selected by those skilled in the art.

[0032] S2, take any strongly related word as a connecting word, obtain the marked question sentence containing both the target word and the connecting word, and obtain the answer sentence corresponding to each marked question sentence, remove the meaningless words in the answer sentence to obtain the effective answer sentence, calculate the distribution coefficient of any word in the effective answer sentence, and construct the word vector of the effective answer sentence according to the distribution coefficient.

[0033] In one embodiment, weakly related words of the target word are removed because the weakly related words are loosely connected to the target word and cannot effectively improve the semantic relevance of the target word. Keeping these weakly related words may introduce noise, resulting in unnecessary interference in the subsequent ambiguous analysis, affecting the accuracy of the results.

[0034] For the same valid answer sentence, obtain the proportion of any word in the valid answer sentence and the total number of words in the valid answer sentence respectively, and take the product of the normalized total number of words and the proportion of any word in the valid answer sentence as the distribution coefficient.

[0035] The proportion of any word in a valid answer also indicates the probability of the word appearing, which can illustrate the importance of the word in the valid answer. It is also limited by the total number of words to prevent the situation where the total number of words in the valid answer is small, resulting in a high probability of the word appearing, which affects the judgment of the importance of the word.

[0036] For each valid answer, we first calculate the proportion of any word in the answer and multiply it by the normalized value of the total number of words in the entire answer to get a distribution coefficient. This coefficient can reflect the relative importance and distribution characteristics of the word in the sentence, avoiding the contextual influence that may be ignored by simple frequency calculation. In this way, we can accurately identify the key words that dominate the valid answer.

[0037] S3, cluster the word vectors of all valid answer sentences to obtain several clusters. For any cluster, traverse to obtain the inverse of the number of strongly related words in the marked question corresponding to each valid answer sentence, and take the mean of all inverses as the ambiguity value of the cluster.

[0038] In one embodiment, preferably, the clustering algorithm uses k-means clustering, sets the initial k value to 1, and uses the silhouette coefficient method to adjust the initial k value to obtain the optimal k value to obtain multiple clusters, and one cluster corresponds to one type of valid answer sentence.

[0039] S4, traverse to obtain the ambiguity value of each cluster, construct the ambiguity value sequence of the connecting word to the target word, and calculate the ambiguity evaluation to complete the ambiguity elimination.

[0040] In one embodiment, the ambiguous evaluation satisfies the relationship: , Indicates connecting words Target word The ambiguous evaluation of Indicates connecting words Target word The ambiguous value sequence of Indicates the number of elements, represents the minimum ambiguous value in the ambiguous value sequence, represents the maximum ambiguous value in the ambiguous value sequence, Represents the normalization function.

[0041] Ambiguity evaluation combines the sequence of ambiguity values ​​of the connecting words to the target words and the normalization of the sequence. This processing method can not only standardize the ambiguity value to adapt it to different ranges of lexical relations, but also ensure that the evaluation results are more accurate and stable by reducing the impact of the minimum ambiguity value. This method can effectively distinguish and quantify the degree of ambiguity in lexical relations, which helps to improve the accuracy of semantic understanding in practical applications, especially in polysemous word disambiguation in natural language processing, information retrieval and question-answering systems, which can better eliminate ambiguity and ensure the accuracy and reliability of text processing.

[0042] Traverse and obtain the ambiguous evaluation of all strongly related words.

[0043] It should be noted that one professional word corresponds to several strongly related words, that is, it corresponds to multiple ambiguous evaluations. When a question is input, first obtain all the professional words in the question, and obtain the ambiguous evaluation of any professional word and each strongly related word. For any strongly related word, the strongly related word in the question will have the same number of ambiguous evaluations as the number of professional words in the question. Calculate the mean of the ambiguity evaluation of the strongly related word as the total ambiguity coefficient of the strongly related word in the question, set the ambiguity coefficient threshold, and when the total ambiguity coefficient of the strongly related word is greater than the ambiguity coefficient threshold, it is considered that the ambiguity of the strongly related word in the question is large, and then eliminate the strongly related word in the question interpretation to complete the ambiguity elimination.

[0044] An embodiment of the present invention also discloses an artificial intelligence-based text ambiguity elimination system, including a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, an artificial intelligence-based text ambiguity elimination method according to the present invention is implemented.

[0045] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface, and their configuration and functions are known in the art, so they will not be described in detail here.

[0046] In the present invention, the aforementioned memory may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device. For example, a computer-readable storage medium may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory RRAM (Resistive Random Access Memory), a dynamic random access memory DRAM (Dynamic Random Access Memory), a static random access memory SRAM (Static Random-Access Memory), an enhanced dynamic random access memory EDRAM (Enhanced Dynamic Random Access Memory), a high-bandwidth memory HBM (High-Bandwidth Memory), a hybrid memory cube HMC (Hybrid Memory Cube), etc., or any other medium that can be used to store the required information and can be accessed by an application, a module or both. Any such computer storage medium may be part of a device or accessible or connectable to a device.

[0047] Although this specification has shown and described a number of embodiments of the present invention, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art will conceive of many modifications, changes and alternatives without departing from the ideas and spirit of the present invention. It should be understood that in the practice of the present invention, various alternatives to the embodiments of the present invention described herein may be employed.

[0048] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for eliminating word ambiguity based on artificial intelligence, characterized in that: include: Construct a word library, which includes professional words and non-problematic words. Use the words in the question except professional words and non-problematic words as reference words. Divide all reference words into strong related words and weak related words of the target word. The target word is any professional word. Take any strongly related word as a connecting word, obtain the marked question sentence containing both the target word and the connecting word, and obtain the answer sentence corresponding to each marked question sentence, remove the meaningless words in the answer sentence to obtain the effective answer sentence, calculate the distribution coefficient of any word in the effective answer sentence, and construct the word vector of the effective answer sentence according to the distribution coefficient; Cluster the word vectors of all valid answer sentences to obtain several clusters. One cluster corresponds to one type of valid answer sentence. For any cluster, traverse to obtain the reciprocal of the number of strongly related words in each valid answer sentence corresponding to the marked question, and take the mean of all reciprocals as the ambiguity value of the cluster. Traverse to obtain the ambiguity value of each cluster, construct the ambiguity value sequence of the connecting word to the target word, and calculate the ambiguity evaluation to complete the ambiguity elimination; Among them, the ambiguous evaluation satisfies the relationship: , Indicates connecting words Target word The ambiguous evaluation of Indicates connecting words Target word The ambiguous value sequence of Indicates the number of elements, represents the minimum ambiguous value in the ambiguous value sequence, represents the maximum ambiguous value in the ambiguous value sequence, Represents the normalization function.

2. The method for eliminating word ambiguity based on artificial intelligence according to claim 1, characterized in that: The method of dividing all reference words into strongly related words and weakly related words of the target word includes: Take any reference word as the target reference word, calculate the association index between the target reference word and the target word, traverse to obtain the association index between each reference word and the target word, arrange all the association indexes from large to small, and construct an association index sequence; Set a split point for the correlation index sequence, divide the correlation index sequence into a first subsequence and a second subsequence, calculate the sequence classification accuracy, iterate the position of the split point, stop the iteration when the sequence classification accuracy is the maximum, and obtain the final first subsequence and second subsequence; The words with larger average values ​​of correlation index in the first subsequence and the second subsequence are regarded as strongly related words, and the words with smaller average values ​​of correlation index are regarded as weakly related words.

3. The method for eliminating word ambiguity based on artificial intelligence according to claim 2, characterized in that: The correlation index satisfies the relationship: , Reference word With target word The correlation index, Indicates that it also contains reference words and target word The number of questions Indicates that it contains the target word The number of questions Indicates contains the target word The number of all professional words in the question.

4. The method for eliminating word ambiguity based on artificial intelligence according to claim 2, characterized in that: The sequence classification accuracy satisfies the relationship: , represents the sequence classification accuracy, Indicates the proportion of the first subsequence in the correlation index sequence, Indicates the proportion of the second subsequence in the correlation index sequence, and denote the first subsequence and the second subsequence respectively, represents the standard deviation, Represents the mean.

5. The method for eliminating word ambiguity based on artificial intelligence according to claim 2, characterized in that: The initial position of the segmentation point is between the first correlation index and the second correlation index in the correlation index sequence, the iteration direction is from left to right, and the iteration step length is one correlation index.

6. The method for eliminating word ambiguity based on artificial intelligence according to claim 1, characterized in that: The calculation of the distribution coefficient of any word in the valid answer sentence includes: For the same valid answer sentence, obtain the proportion of any word in the valid answer sentence and the total number of words in the valid answer sentence respectively, and take the product of the normalized total number of words and the proportion of any word in the valid answer sentence as the distribution coefficient.

7. A text ambiguity elimination system based on artificial intelligence, characterized in that: include: Processor; and A memory storing computer instructions, wherein when the computer instructions are executed by a processor, the system executes a text ambiguity elimination method based on artificial intelligence according to any one of claims 1 to 6.

Images