An artificial intelligence-based method and system for resolving word ambiguity

Abstract

The present application relates to the field of data processing, more particularly, the present application relates to a kind of based on artificial intelligence's method and system for eliminating the ambiguity of character.The method comprises: constructing the word bank, all reference words are divided into the strong correlation word and weakly correlated word of target word;Eliminate the unintended word in answer sentence to obtain effective answer sentence, the distribution coefficient of any word in effective answer sentence is calculated, and the word vector of effective answer sentence is constructed according to distribution coefficient;The word vector of all effective answer sentences is clustered to obtain several clustering clusters, for any clustering cluster, the number of strong correlation word in the corresponding marked question sentence of each effective answer sentence is obtained by traversal, and the mean of all reciprocals is taken as the ambiguity value of clustering cluster;Ambiguity value of each clustering cluster is obtained by traversal, the ambiguity value sequence of target word is constructed by contact word, and ambiguity evaluation is calculated, and ambiguity elimination is completed.The technical scheme of the present application can improve the precision of text understanding and reduce the recognition error caused by the ambiguity of character.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing. More particularly, the present application relates to an artificial intelligence-based method and system for resolving ambiguity in text. BACKGROUND

[0002] Resolving ambiguity in text is of great significance in ensuring accurate information transmission, avoiding misunderstandings and disputes, enhancing information value, and maintaining language standards. In communication, it enables the receiver to accurately understand the intent, ensuring smooth and efficient communication. In formal settings such as legal and business fields, resolving ambiguity can clarify the rights and obligations of all parties, reduce legal risks, and maintain cooperation order. For knowledge dissemination and news reporting, it helps to accurately convey core content and enhance the authority and credibility of information. From the perspective of language development, resolving ambiguity promotes people to follow grammar rules and vocabulary pairing standards, maintains language stability and consistency, and promotes the healthy development of language, making it better serve human communication needs.

[0003] Existing dialogue models mainly rely on context to speculate and judge when dealing with ambiguous text. However, in many cases, the meaning of ambiguous words can change depending on the professional field or specific context. In such cases, the model lacks in-depth understanding and association of different professional field vocabulary, leading to inaccurate identification of the actual meaning of ambiguous words. The model's output may deviate from the user's actual needs, even causing misinformation. SUMMARY

[0004] To solve the above technical problems, the present application proposes an artificial intelligence-based method and system for resolving ambiguity in text.

[0005] In a first aspect, the present application discloses an artificial intelligence-based method for resolving ambiguity in text, comprising: constructing a vocabulary, the vocabulary including professional words and non-intentional words, taking words in a question sentence other than professional words and non-intentional words as reference words, dividing all reference words into strong related words and weak related words of a target word, the target word being any professional word; taking any strong related word as a connecting word, obtaining a marked question sentence containing the target word and the connecting word, and obtaining an answer sentence corresponding to each marked question sentence, eliminating non-intentional words in the answer sentence to obtain an effective answer sentence, calculating the distribution coefficient of any word in the effective answer sentence, and constructing a word vector of the effective answer sentence according to the distribution coefficient; clustering the word vectors of all effective answer sentences to obtain a plurality of clustering clusters, one clustering cluster corresponding to one type of effective answer sentence, for any clustering cluster, traversing to obtain the reciprocal of the number of strong related words in each effective answer sentence corresponding to the marked question sentence, taking the mean of all reciprocals as the ambiguity value of the clustering cluster; traversing to obtain the ambiguity value of each clustering cluster, constructing an ambiguity value sequence of the connecting word to the target word, and calculating an ambiguity evaluation to complete ambiguity resolution; wherein the ambiguity evaluation satisfies the relationship:

[0006] , Connective words For target words Ambiguous evaluations Connective words For target words ambiguous value sequence, Indicates the number of elements. This represents the smallest ambiguous value in the sequence of ambiguous values. This represents the maximum ambiguity value in the sequence of ambiguities. This represents the normalization function.

[0007] Preferably, the step of classifying all reference words into strongly related and weakly related words of the target word includes: taking any reference word as the target reference word, calculating the correlation index between the target reference word and the target word, traversing to obtain the correlation index between each reference word and the target word, arranging all correlation indices from largest to smallest to construct a correlation index sequence; setting a split point on the correlation index sequence to divide the correlation index sequence into a first subsequence and a second subsequence, calculating the sequence classification accuracy, iterating the position of the split point, stopping the iteration when the sequence classification accuracy is maximum, and obtaining the final first subsequence and second subsequence; taking the words with the larger average correlation index in the first subsequence and the second subsequence as strongly related words, and taking the words with the smaller average correlation index as weakly related words.

[0008] Preferably, the correlation index satisfies the following relationship:

[0009] , Indicates reference word With target words The correlation index, Indicates that it contains reference words. and target words The number of questions Indicates the presence of target words The number of questions Indicates the first A collection containing target words The number of all technical terms in the question.

[0010] Preferably, the sequence classification accuracy satisfies the following relationship:

[0011] , Indicates the accuracy of sequence classification. This indicates the proportion of the first subsequence in the correlation index sequence. This indicates the proportion of the second subsequence in the correlation index sequence. and These represent the first subsequence and the second subsequence, respectively. a standard deviation, a mean value.

[0012] Preferably, the initial position of the split point is between the first and second association indexes in the association index sequence, the iteration direction is from left to right, and the iteration step is one association index.

[0013] Preferably, the calculation of the distribution coefficient of any word in the effective answer sentence comprises:

[0014] For the same effective answer sentence, the proportion of any word in the effective answer sentence and the total number of words in the effective answer sentence are obtained respectively, and the product of the normalized total number of words and the proportion of any word in the effective answer sentence is taken as the distribution coefficient.

[0015] In a second aspect, the present application discloses a text ambiguity elimination system based on artificial intelligence, comprising: a processor; and a memory storing computer instructions, when the computer instructions are run by the processor, the system executes the above-mentioned text ambiguity elimination method based on artificial intelligence.

[0016] The present application has the following beneficial effects:

[0017] The present application can accurately identify and eliminate irrelevant information and extract valuable word features by constructing a word library of professional words and unintended words, judging ambiguity in combination with the strong and weak relevance of reference words, quantifying the connection between the target word and its related words, and analyzing the tags of the question and the word vectors of the effective answer.

[0018] By analyzing the ambiguity values of each clustering cluster and combining the accurate calculation of the association index, the ambiguity can be effectively quantified and evaluated. This method optimizes the classification accuracy and segmentation effect by introducing statistical standard deviation, mean difference and other means, further improving the classification accuracy and segmentation effect. This fine ambiguity elimination mechanism can significantly reduce the understanding deviation caused by unclear semantics in practical applications, and is especially suitable for natural language processing, intelligent question and answer systems and machine translation fields that require high-precision semantic understanding, and ultimately improves the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0019] The above and other objects, features and advantages of the exemplary embodiments of the present application will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0020] Figure 1is a flowchart of a text ambiguity elimination method based on artificial intelligence according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0022] It should be understood that when the claims, the specification and the drawings of the present application use the terms "first", "second", etc., they are only used to distinguish different objects, and are not used to describe a specific sequence. The terms "include" and "contain" used in the specification and claims of the present application indicate the presence of the described features, whole, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, whole, steps, operations, elements, components and / or sets thereof.

[0023] The present application provides a text ambiguity elimination method based on artificial intelligence. As shown in Figure 1 The text ambiguity elimination method based on artificial intelligence includes steps S1-S4, which will be described in detail below.

[0024] S1, construct a word library, the word library includes professional words and unintentional words, the words in the question sentence except the professional words and the unintentional words are taken as reference words, and all the reference words are divided into strong related words and weak related words of the target word.

[0025] It should be noted that some words in the text may have ambiguity, and the degree of ambiguity may be different. In actual application, some words with specific meaning usually do not have ambiguity and can be used as effective reference words. Because of the fixed semantic range and less context dependence, these words can provide clear guidance for semantic understanding.

[0026] However, some words with strong polysemy may have different interpretations in different contexts, which need special attention when used, and when constructing the word library, the weight of these words should be reduced to reduce the interference of ambiguity on semantic analysis.

[0027] In one embodiment, when building the vocabulary, those skilled in the art will divide the vocabulary into two categories: professional words and non-intentional words. Professional words refer to terms with clear and fixed meanings in a specific field, such as professional names and terms in the fields of medicine, law, engineering, etc. These words are essential for understanding and processing texts in the field. Non-intentional words refer to words that do not have actual semantic bearing, such as prepositions, mood words, conjunctions, etc. They are mainly used to connect sentences or express mood, but do not have substantial effect on the core meaning of the text.

[0028] The words in the question sentence other than professional words and non-intentional words are taken as reference words, any professional word is taken as a target word, and any reference word is taken as a target reference word. The association index of the target reference word and the target word is calculated, and the association index satisfies the relationship:

[0029] , represents the association index of the reference word and the target word , represents the number of question sentences containing both the reference word and the target word , represents the number of question sentences containing the target word , represents the number of professional words in the th question sentence containing the target word .

[0030] The frequency of the appearance of the reference word can be constrained, The greater the value of , the higher the frequency of the appearance of the reference word in the question sentence containing the target word , indicating that the reference word has a higher association with the target word .

[0031] The uniqueness constraint of the reference word to the target word , The greater the value of , the fewer the other professional words in the question sentence, indicating that the association of the reference word is reflected in the association with the target word .

[0032] The association index combines the frequency of the reference word in the target word appearing question and the number of other professional words involved in these questions. By restricting the frequency of the reference word, it is ensured that only the reference word that frequently appears in the relevant question of the target word will be given a higher association. At the same time, by considering the number of professional words in the question, the interference of other professional words on the association calculation of the reference word and the target word is avoided, and it is ensured that the association truly reflects the closeness of the reference word and the target word.

[0033] The association index of each reference word and the target word is obtained by traversing, and all association indexes are arranged from large to small to construct an association index sequence. The association index sequence is divided into a first sub-sequence and a second sub-sequence by setting a split point. The sequence classification accuracy is calculated, and the position of the split point is iterated, and when the sequence classification accuracy is maximum, the iteration is stopped, and the final first sub-sequence and second sub-sequence are obtained; The larger the mean of the association index in the first sub-sequence and the second sub-sequence is regarded as the strong correlation word, and the smaller the mean of the association index is regarded as the weak correlation word.

[0034] The initial position of the split point is between the first association index and the second association index in the association index sequence, the iteration direction is from left to right, and the iteration step is an association index.

[0035] The sequence classification accuracy satisfies the relationship:

[0036] , The sequence classification accuracy is represented by The proportion of the first sub-sequence in the association index sequence is represented by The proportion of the second sub-sequence in the association index sequence is represented by And The first sub-sequence and the second sub-sequence are represented by The standard deviation is represented by The mean is represented by

[0037] The classification accuracy not only considers the dispersion of the association index in the sub-sequence (reflected by the standard deviation) and the mean difference between the sub-sequences, but also adjusts the proportion to make the difference between the two sub-sequences more significant. Maximizing the classification accuracy helps to find the most suitable split point in practical applications, ensuring that each sub-sequence can effectively reflect the actual association of the vocabulary, and finally distinguishing strong correlation words and weak correlation words.

[0038] For example, the association index sequence is The initial position of the split point is between and , that is, The first sub-sequence is The second sub-sequence is calculated at this time, and after iterating the position of the split point once, the first sub-sequence is , the second subsequence becomes The sequence classification accuracy at this time is calculated again. Then the position of the segmentation point is iterated until the calculated sequence classification accuracy is maximum, and strong related words and weak related words are obtained.

[0039] In another embodiment, the strong related words and weak related words of the target word are selected by a person skilled in the art.

[0040] S2, taking any strong related word as a conjunction word, obtaining a tagged question sentence containing both the target word and the conjunction word, and obtaining an answer sentence corresponding to each tagged question sentence, eliminating irrelevant words in the answer sentence to obtain an effective answer sentence, calculating the distribution coefficient of any word in the effective answer sentence, and constructing a word vector of the effective answer sentence according to the distribution coefficient.

[0041] In one embodiment, the weak related words of the target word are eliminated because the weak related words of the target word are eliminated because the weak related words are loosely connected with the target word, and cannot effectively improve the semantic relevance of the target word. Retaining these weak related words may introduce noise, causing unnecessary interference in subsequent ambiguity analysis, affecting the accuracy of the results.

[0042] For the same effective answer, the proportion of any word in the effective answer and the total number of words in the effective answer are obtained respectively, and the product of the normalized total number of words and the proportion of any word in the effective answer is taken as the distribution coefficient.

[0043] The proportion of any word in the effective answer also indicates the probability of the word appearing, which can indicate the importance of the word in the effective answer, and is limited by the total number of words, preventing the total number of words in the effective answer from being too small, resulting in a high probability of the word appearing, affecting the judgment of the importance of the word.

[0044] For each effective answer, first calculate the proportion of any word in the answer, and multiply it by the normalized value of the total number of words in the entire answer, to obtain a distribution coefficient. This coefficient can reflect the relative importance and distribution characteristics of the word in the sentence, avoiding the context influence that may be ignored by simple frequency calculation. Through this method, the key words that dominate in the effective answer can be accurately identified.

[0045] S3, clustering the word vectors of all effective answers to obtain a plurality of clustering clusters, for any clustering cluster, traversing to obtain the reciprocal of the number of strong related words in each effective answer corresponding to the tagged question, and taking the mean of all reciprocals as the ambiguity value of the clustering cluster.

[0046] In one embodiment, preferably, the clustering algorithm uses k-means clustering, sets the initial k value to 1, adjusts the initial k value using the silhouette coefficient method to obtain the optimal k value, to obtain a plurality of clustering clusters, and each clustering cluster corresponds to a type of effective answer sentence.

[0047] S4, traversing to obtain the ambiguity value of each clustering cluster, constructing the ambiguity value sequence of the target word for the connection word, and calculating the ambiguity evaluation to complete ambiguity elimination.

[0048] In one embodiment, the ambiguity evaluation satisfies the relationship:

[0049] , denotes the ambiguity evaluation of the connection word to the target word , denotes the ambiguity value sequence of the connection word to the target word , denotes the number of elements, denotes the minimum ambiguity value in the ambiguity value sequence, denotes the maximum ambiguity value in the ambiguity value sequence, denotes the normalization function.

[0050] The ambiguity evaluation combines the ambiguity value sequence of the connection word to the target word and the normalization processing of the sequence. This processing method not only standardizes the ambiguity value, making it suitable for different ranges of lexical relationships, but also ensures that the evaluation result is more accurate and stable by reducing the influence of the minimum ambiguity value. Through this method, the ambiguity degree in lexical relationships can be effectively distinguished and quantified, which helps to improve the precision of semantic understanding in practical applications, especially in polysemy disambiguation, information retrieval and question answering systems in natural language processing, which can better eliminate ambiguity and ensure the accuracy and reliability of text processing.

[0051] Traverse to obtain the ambiguity evaluation of all strong related words.

[0052] It should be noted that one professional word corresponds to several strong related words, i.e. corresponds to multiple ambiguity evaluations. When a question sentence is input, first obtain all professional words in the question sentence, and obtain the ambiguity evaluation of any one professional word and each strong related word. For any one strong related word, the strong related word in the question sentence will have the same number of ambiguity evaluations as the number of professional words in the question sentence, calculate the average ambiguity evaluation of the strong related word, as the total ambiguity coefficient of the strong related word in the question sentence, set the ambiguity coefficient threshold, when the total ambiguity coefficient of the strong related word is greater than the ambiguity coefficient threshold, it is considered that the ambiguity of the strong related word in the question sentence is large, then the strong related word is eliminated in the question interpretation, and the ambiguity elimination is completed.

[0053] The embodiment of the application further discloses a character ambiguity elimination system based on artificial intelligence, comprising a processor and a memory, and the memory stores computer program instructions.

[0054] The system further comprises a communication bus and a communication interface and other components well known to those skilled in the art, and the arrangement and functions of the components are known in the art, and thus will not be described here.

[0055] In the present application, the aforementioned memory can be any tangible medium that contains or stores a program, which can be used by or in conjunction with an instruction execution system, device or apparatus. For example, the computer readable storage medium can be any appropriate magnetic storage medium or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random-Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc., or any other medium that can be used to store desired information and can be accessed by an application, a module or both. Any such computer storage medium can be part of a device or accessible or connectable to the device.

[0056] Although the present specification has shown and described a number of embodiments of the application, it will be apparent to those skilled in the art that many modifications, variations, and alternatives to the embodiments described herein can be made in the light of the teachings herein. It is therefore to be understood that changes can be made in the form, details, and / or arrangement of the embodiments of the application without departing from the spirit and intent of the application. It is intended that the application be practiced otherwise than as specifically described herein.

[0057] The above are the preferred embodiments of the present application, which do not limit the protection scope of the present application, and thus: any equivalent changes made in the structure, shape, principle of the present application should be covered within the protection scope of the present application.

Claims

1. An artificial intelligence-based method for disambiguating words, characterized by, The method comprises the following steps: constructing a word library, the word library comprising professional words and unintended words, taking words in a question sentence other than the professional words and the unintended words as reference words, and dividing all the reference words into strong related words and weak related words of a target word, which comprises: taking any reference word as a target reference word, calculating the correlation index of the target reference word and the target word, obtaining the correlation index of each reference word and the target word through traversal, arranging all the correlation indexes from large to small to construct a correlation index sequence; setting a segmentation point for the correlation index sequence, dividing the correlation index sequence into a first sub-sequence and a second sub-sequence, and calculating sequence classification accuracy, iterating the position of the segmentation point, and stopping iteration when the sequence classification accuracy is maximum to obtain the final first sub-sequence and second sub-sequence; calculating the average of the correlation indexes of the first sub-sequence and the second sub-sequence respectively, in response to the average of the correlation indexes of the first sub-sequence being greater than the average of the correlation indexes of the second sub-sequence, taking the reference word corresponding to each correlation index in the first sub-sequence as a strong related word, and taking the reference word corresponding to each correlation index in the second sub-sequence as a weak related word; in response to the average of the correlation indexes of the first sub-sequence being less than the average of the correlation indexes of the second sub-sequence, taking the reference word corresponding to each correlation index in the second sub-sequence as a strong related word, and taking the reference word corresponding to each correlation index in the first sub-sequence as a weak related word, wherein the target word is any professional word; taking any strong related word as a connecting word, obtaining a marked question sentence containing the target word and the connecting word, obtaining an answer sentence corresponding to each marked question sentence, eliminating unintended words in the answer sentence to obtain an effective answer sentence, calculating the distribution coefficient of any word in the effective answer sentence, and constructing a word vector of the effective answer sentence according to the distribution coefficient; performing clustering on the word vectors of all effective answer sentences to obtain a plurality of clustering clusters, one clustering cluster corresponding to one type of effective answer sentence, for any clustering cluster, obtaining the reciprocal of the number of strong related words in the corresponding marked question sentence of each effective answer sentence through traversal, and taking the average of all reciprocals as the ambiguity value of the clustering cluster; traversing the ambiguity value of each clustering cluster to construct an ambiguity value sequence of the connecting word to the target word, and calculating an ambiguity evaluation to complete ambiguity elimination; wherein the ambiguity evaluation satisfies the relationship: , indicates a conjunction word ambiguity value of a target word , indicates a conjunction word ambiguity value of a target word , indicates the number of elements indicates the minimum ambiguity value in the ambiguity value sequence indicates the maximum ambiguity value in the ambiguity value sequence indicates a normalization function 2.The method of claim 1, wherein, the correlation index satisfies the relationship: , represents the reference word , represents the association index of the target word , represents the number of questions containing both the reference word and the target word , represents the number of questions containing the target word , represents the number of professional words in the th question containing the target word 3.The method of claim 1, wherein, the sequence classification accuracy satisfies the relationship: denotes sequence classification accuracy, denotes the proportion of the first sub-sequence in the correlation index sequence, denotes the proportion of the second sub-sequence in the correlation index sequence, and denote the first sub-sequence and the second sub-sequence, respectively, denotes the standard deviation, denotes the mean.​ 4.The method of claim 1, wherein, the initial position of the segmentation point is between the first correlation index and the second correlation index in the correlation index sequence, the iteration direction is from left to right, and the iteration step is one correlation index.

5. The method of claim 1, wherein the method further comprises: The calculation of the distribution coefficient of any word in the effective answer sentence comprises: for the same effective answer sentence, obtaining the proportion of any word in the effective answer sentence and the total number of words in the effective answer sentence respectively, and taking the product of the normalized total number of words and the proportion of any word in the effective answer sentence as the distribution coefficient.

6. An artificial intelligence based disambiguation system for words, characterized in that, The system comprises: a processor; and a memory storing computer instructions, which, when executed by the processor, cause the system to perform the artificial intelligence-based text ambiguity elimination method according to any one of claims 1-5.

Images