Word retrieval method, device, equipment and storage medium based on binary mutual information

Abstract

The present invention relates to the field of artificial intelligence, and discloses a word retrieval method, device, equipment and storage medium based on binary mutual information, which are used to increase the accuracy of extracting keywords, and then retrieve effective message responses. The method of the present invention includes: obtaining a target question text sent by a target user; segmenting the target question text to obtain multiple candidate words, each candidate word is unique; calling a preset corpus to determine the word frequency TF and inverse document frequency IDF of multiple target words in multiple candidate words; calculating the binary mutual information of each target word in multiple target words according to a preset formula; obtaining the adjustment factor of each target word in multiple target words according to a preset algorithm; calculating the weight value of each target word according to the adjustment factor of each target word and the binary mutual information of each target word; determining the keyword of the target question text according to the weight value of each target word, and retrieving the corresponding answer according to the keyword.

Description

Technical Field

[0001] The present invention relates to the technical field of keyword matching, and in particular to a word retrieval method, device, equipment and storage medium based on binary mutual information. Background Art

[0002] With the development of information technology, people can increasingly search and obtain the information they need online. However, quickly locating the information they need within the vast web is crucial. Therefore, information retrieval technology has emerged, and one of the key supporting technologies is keyword extraction. Currently, the most widely used keyword extraction technology is the term frequency–inverse document frequency (TF-IDF) algorithm. The basic principle of the TF-IDF algorithm is to calculate the ranking of words based on their number of occurrences and their frequency weights, and select the top few words as keywords.

[0003] In existing solutions, the TF-IDF algorithm's weight calculation relies too much on word frequency and fails to consider the impact of word distribution across different documents on its weight. This results in inaccurate information extraction and the inability to retrieve effective message responses. Summary of the Invention

[0004] The present invention provides a word retrieval method, apparatus, device and storage medium based on binary mutual information, which are used to add an adjustment factor to adjust the weight value of word frequency, and use the mutual information factor to eliminate the influence of different document distributions on the weight value, reduce the dependence of the TF-IDF algorithm on word frequency, improve the accuracy of the weight value, increase the accuracy of keyword extraction, and thus retrieve effective message responses.

[0005] A first aspect of an embodiment of the present invention provides a word retrieval method based on binary mutual information, comprising: obtaining a target question text sent by a target user, the target question text being used to indicate obtaining an answer corresponding to the target question text; segmenting the target question text to obtain multiple candidate words, each candidate word being unique; calling a preset corpus to determine the word frequency TF and inverse document frequency IDF of multiple target words in the multiple candidate words; calculating the binary mutual information of each target word in the multiple target words according to a preset formula; obtaining an adjustment factor of each target word in the multiple target words according to a preset algorithm; obtaining a weight value of each target word according to the adjustment factor of each target word and the binary mutual information of each target word; determining keywords of the target question text according to the weight value of each target word, and retrieving corresponding answers based on the keywords.

[0006] Optionally, in a first implementation method of the first aspect of the embodiment of the present invention, the calling of a preset corpus to determine the term frequency TF and inverse document frequency IDF of multiple target words among the multiple candidate words includes: performing stop word filtering processing on the multiple candidate words to obtain multiple target words; calling a preset corpus to determine the term frequency TF of each target word among the multiple target words; calling a preset corpus to determine the inverse document frequency IDF of each target word among the multiple target words.

[0007] Optionally, in a second implementation method of the first aspect of the embodiment of the present invention, calling a preset corpus to determine the word frequency TF of each target word among multiple target words includes: obtaining a preset corpus and determining the target corpus documents in the preset corpus; determining the number of occurrences T of each target word in the target corpus documents, and generating the word frequency TF of each target word.

[0008] Optionally, in a third implementation method of the first aspect of the embodiment of the present invention, calling a preset corpus to determine the inverse document frequency IDF of each target word in multiple target words includes: obtaining a preset corpus and determining the total number M of corpus documents in the preset corpus; determining the number of documents Wi containing the first target word in the M corpus documents, where i is a positive integer and the first target word is any one of the multiple target words; calling a first preset formula, the number of documents Wi and the total number M to generate the inverse document frequency IDF of the first target word, the first preset formula being IDF=log2(M / Wi+1); generating the inverse document frequency IDF for each target word.

[0009] Optionally, in a fourth implementation of the first aspect of the embodiment of the present invention, the binary mutual information of each target word in the multiple target words is calculated according to a preset formula, including: selecting any one target word in the multiple target words as a candidate target word; determining the count of occurrences of the candidate target word in two consecutive corpus documents; Get a first count; determine the count of the candidate target word appearing in two sequential corpus documents As the second count; calculate the ratio of the first count and the second count to obtain a first ratio p(X|w i ,w i+k ,w i+1 ); According to the first ratio p(X|w i ,w i+k ,w i+1 ) Determine the binary mutual information mi(x,w i ,w i+k ,w i+1 )=log2p(X|wi ,w i+k ,w i+1 ); Generate binary mutual information of other target words in the multiple target words to obtain binary mutual information of each target word.

[0010] Optionally, in a fifth implementation of the first aspect of the embodiment of the present invention, obtaining the adjustment factor of each of the multiple target words according to a preset algorithm includes: determining the current business scenario based on the target question text; dividing the multiple target words into common sense words and key words based on the current business scenario; setting the adjustment factor corresponding to the common sense words to a negative number, and setting the adjustment factor corresponding to the key words to a positive number; and generating an adjustment factor for each target word.

[0011] Optionally, in a sixth implementation of the first aspect of the embodiment of the present invention, the step of calculating the weight value of each target word based on the adjustment factor of each target word and the binary mutual information of each target word includes:

[0012] Select a target word from multiple target words as the second target word, and determine the adjustment factor μ corresponding to the second target word. x and binary mutual information mi(x); the weight value f(x) of the second target word is calculated according to the preset formula, and the preset calculation formula is: f(x)=mi(x)*TF*IDF+μ x ; Calculate the weight values ​​of other target words in the multiple target words to obtain the weight value of each target word.

[0013] The second aspect of an embodiment of the present invention provides a word retrieval device based on binary mutual information, including: a first acquisition unit, used to acquire a target question text sent by a target user, and the target question text is used to indicate the answer corresponding to the target question text; a word segmentation unit, used to segment the target question text to obtain multiple candidate words, each candidate word is unique; a call determination unit, used to call a preset corpus to determine the word frequency TF and inverse document frequency IDF of multiple target words in the multiple candidate words; a first calculation unit, used to calculate the binary mutual information of each target word in the multiple target words according to a preset formula; a second acquisition unit, used to acquire the adjustment factor of each target word in the multiple target words according to a preset algorithm; a second calculation unit, used to calculate the weight value of each target word according to the adjustment factor of each target word and the binary mutual information of each target word; a determination retrieval unit, used to determine the keywords of the target question text according to the weight value of each target word, and retrieve the corresponding answer based on the keyword.

[0014] Optionally, in a first implementation method of the second aspect of the embodiment of the present invention, the calling determination unit includes: a filtering module, used to perform stop word filtering processing on the multiple candidate words to obtain multiple target words; a first determination module, used to call a preset corpus to determine the word frequency TF of each target word in the multiple target words; and a second determination module, used to call a preset corpus to determine the inverse document frequency IDF of each target word in the multiple target words.

[0015] Optionally, in a second implementation of the second aspect of the embodiment of the present invention, the first determination module is specifically used to: obtain a preset corpus and determine the target corpus document in the preset corpus; determine the number of occurrences T of each target word in the target corpus document, and generate a word frequency TF for each target word.

[0016] Optionally, in a third implementation method of the second aspect of the embodiment of the present invention, the second determination module is specifically used to: obtain a preset corpus and determine the total number M of corpus documents in the preset corpus; determine the number Wi of documents containing the first target word in the M corpus documents, where i is a positive integer and the first target word is any one of the multiple target words; call a first preset formula, the number of documents Wi and the total number M to generate an inverse document frequency IDF for the first target word, and the first preset formula is IDF=log2(M / Wi+1); generate an inverse document frequency IDF for each target word.

[0017] Optionally, in a fourth implementation of the second aspect of the embodiment of the present invention, the first computing unit is specifically configured to: select any one of the multiple target words as a candidate target word; determine the count of occurrences of the candidate target word in two consecutive corpus documents; Get a first count; determine the count of the candidate target word appearing in two sequential corpus documents As the second count; calculate the ratio of the first count and the second count to obtain a first ratio p(X|w i ,w i+k ,w i+1 ); According to the first ratio p(X|w i ,w i+k ,w i+1 ) Determine the binary mutual information mi(x,w i ,w i+k ,w i+1 )=log2p(X|w i ,w i+k ,w i+1 ); Generate binary mutual information of other target words in the multiple target words to obtain binary mutual information of each target word.

[0018] Optionally, in a fifth implementation of the second aspect of the embodiment of the present invention, the second acquisition unit is specifically used to: determine the current business scenario based on the target question text; divide the multiple target words into common sense words and key words based on the current business scenario; set the adjustment factor corresponding to the common sense words to a negative number, and set the adjustment factor corresponding to the key words to a positive number; and generate an adjustment factor for each target word.

[0019] Optionally, in a sixth implementation of the second aspect of the embodiment of the present invention, the second computing unit is specifically configured to:

[0020] Select a target word from multiple target words as the second target word, and determine the adjustment factor μ corresponding to the second target word. x and binary mutual information mi(x); the weight value f(x) of the second target word is calculated according to the preset formula, and the preset calculation formula is: f(x)=mi(x)*TF*IDF+μ x ; Calculate the weight values ​​of other target words in the multiple target words to obtain the weight value of each target word.

[0021] A third aspect of an embodiment of the present invention provides a word retrieval device based on binary mutual information, comprising a memory, a processor, and a computer program stored on the memory and runnable on the processor. When the processor executes the computer program, the word retrieval method based on binary mutual information described in any of the above embodiments is implemented.

[0022] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the word retrieval method based on binary mutual information described in any of the above embodiments.

[0023] In the technical solution provided by the embodiment of the present invention, a target question text sent by a target user is obtained, and the target question text is used to indicate the answer corresponding to the target question text; the target question text is segmented to obtain multiple candidate words, each candidate word is unique; a preset corpus is called to determine the term frequency TF and inverse document frequency IDF of multiple target words in the multiple candidate words; the binary mutual information of each target word in the multiple target words is calculated according to a preset formula; an adjustment factor of each target word in the multiple target words is obtained according to a preset algorithm; a weight value of each target word is calculated based on the adjustment factor of each target word and the binary mutual information of each target word; the keywords of the target question text are determined based on the weight value of each target word, and the corresponding answer is retrieved based on the keyword. In the embodiment of the present invention, an adjustment factor is added to adjust the weight value of the term frequency, and the mutual information factor is used to eliminate the influence of different document distribution conditions on the weight value, reduce the dependence of the TF-IDF algorithm on the term frequency, improve the accuracy of the weight value, increase the accuracy of keyword extraction, and thus retrieve effective message responses. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of an embodiment of a word retrieval method based on binary mutual information in an embodiment of the present invention;

[0025] Figure 2 Schematic diagram of another embodiment of a word retrieval method based on binary mutual information in an embodiment of the present invention;

[0026] Figure 3 Schematic diagram of an embodiment of a word search device based on binary mutual information in an embodiment of the present invention;

[0027] Figure 4 Schematic diagram of another embodiment of a word search device based on binary mutual information in an embodiment of the present invention;

[0028] Figure 5 Schematic diagram of an embodiment of a word retrieval device based on binary mutual information in an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The present invention provides a word retrieval method, apparatus, device and storage medium based on binary mutual information, which are used to add an adjustment factor to adjust the weight value of word frequency, and use the mutual information factor to eliminate the influence of different document distributions on the weight value, reduce the dependence of the TF-IDF algorithm on word frequency, improve the accuracy of the weight value, increase the accuracy of keyword extraction, and thus retrieve effective message responses.

[0030] In order to enable those skilled in the art to better understand the solutions of the present invention, the embodiments of the present invention will be described below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0031] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatus.

[0032] See also Figure 1 , a flowchart of a word search method based on binary mutual information provided by an embodiment of the present invention specifically includes:

[0033] 101. Obtain a target question text sent by a target user, where the target question text is used to indicate obtaining an answer corresponding to the target question text.

[0034] The server obtains the target question text sent by the target user, and the target question text is used to indicate the answer corresponding to the target question text.

[0035] For example, in a chat customer service robot, the target user asks a question "What are the benefits of Fubaobao insurance?" The server needs to retrieve the relevant benefits of "Fubaobao" and use the retrieved benefits as the answer to the question.

[0036] It is understandable that the execution subject of the present invention may be a word search device based on binary mutual information, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0037] 102. Segment the target question text to obtain multiple candidate words, each candidate word is unique.

[0038] The server segments the target question text to obtain multiple candidate words, each of which is unique. In this embodiment, the server uses a preset segmentation algorithm to segment the target question text. The preset segmentation algorithms include a string matching-based segmentation algorithm, a comprehension-based segmentation algorithm, and a statistics-based segmentation algorithm.

[0039] It should be noted that (1) for the word segmentation algorithm based on string matching, it matches the Chinese character string to be analyzed with the entries in a "sufficiently large" machine dictionary according to a certain strategy. If a certain string is found in the dictionary, the match is successful (a word is recognized). According to the different scanning directions, the string matching word segmentation method can be divided into forward matching and reverse matching; according to the different length priority matching situations, it can be divided into maximum (longest) matching and minimum (shortest) matching; according to whether it is combined with the part-of-speech tagging process, it can be divided into a simple word segmentation method and an integrated method combining word segmentation and tagging. (2) For the word segmentation algorithm based on understanding, it achieves the effect of word recognition by letting the computer simulate human understanding of the sentence. Its basic idea is to perform syntactic and semantic analysis while segmenting the words, and use syntactic information and semantic information to deal with ambiguous phenomena. It usually includes three parts: word segmentation subsystem, syntactic semantic subsystem, and general control part. Under the coordination of the general control part, the word segmentation subsystem can obtain syntactic and semantic information about words, sentences, etc. to judge the ambiguity of word segmentation, that is, it simulates the process of human understanding of sentences. (3) For the word segmentation algorithm based on statistics, from a formal point of view, a word is a stable combination of characters. Therefore, in the context, the more times adjacent characters appear at the same time, the more likely they are to form a word. Therefore, the frequency or probability of adjacent co-occurrence of characters can better reflect the credibility of the word. The frequency of the combination of adjacent co-occurring characters in the corpus can be counted and their mutual occurrence information can be calculated. The mutual occurrence information reflects the closeness of the combination relationship between Chinese characters. When the closeness is higher than a certain threshold, it can be considered that this character group may constitute a word. It is understandable that other word segmentation methods can also be used, and the specific ones are not limited here.

[0040] For example, in a chat customer service robot, the target user asked a question, "What are the benefits of Fubaobao insurance?" After word segmentation, the words "Fubaobao", "insurance", "benefits", "which", "have" and so on were obtained.

[0041] 103. Call a preset corpus to determine the term frequency (TF) and inverse document frequency (IDF) of multiple target words in the multiple candidate words.

[0042] The server uses a pre-set corpus to determine the term frequency (TF) and inverse document frequency (IDF) of multiple target terms among the multiple candidate terms. Specifically, the server performs stop word filtering on the multiple candidate terms to obtain multiple target terms; the server uses the pre-set corpus to determine the term frequency (TF) of each target term among the multiple target terms; and the server uses the pre-set corpus to determine the inverse document frequency (IDF) of each target term among the multiple target terms.

[0043] 104. Calculate the binary mutual information of each target word in the plurality of target words according to a preset formula.

[0044] The server calculates the binary mutual information of each target word among multiple target words according to a preset formula. Specifically, it includes:

[0045] (1) The server selects any one target word from multiple target words as a candidate target word.

[0046] (2) The server determines the count of occurrences of the candidate target word in two consecutive corpus documents Get the first count.

[0047] in, For the target word X in two consecutive documents w i ,w i+1 Occurrence count, for example: documents w1, w2, w3 all contain word A, then

[0048] (3) The server determines the count of occurrences of the candidate target word in two sequential corpus documents As the second count.

[0049] in, For the target word X in two sequential documents w i ,w i+k Occurrence count, for example: documents w1, w2, w3 all contain word A, then

[0050] (4) The server calculates the ratio of the first count to the second count and obtains the first ratio p(X|w i ,w i+k ,w i+1 );

[0051] in, It is the ratio of the count of word X appearing in two consecutive adjacent documents to the count of word A appearing in two sequential documents. For example, w1 and w2 are consecutive documents, and w1 and w3 are non-consecutive documents; w1, w2 and w1, w3 are sequential documents, and w2, w1 is non-sequential document.

[0052] (5) The server calculates the value of the first ratio p(X|w i ,w i+k ,w i+1 ) Determine the binary mutual information mi(x,w) of the candidate target words i ,w i+k ,w i+1 )=log2p(X|w i ,w i+k ,w i+1 );

[0053] For example, for keywords A and B, there are 10 corpus documents w1, w2, w3, ..., w10 , where the corpus document w1 contains x1 keywords A and y1 keywords B; the corpus document w2 contains x2 keywords A and y2 keywords B; the corpus document w 10 Contains y3 keywords B. The binary mutual information factors of keywords A and B are calculated as mi(A) and mi(B) respectively. The specific calculation process is as follows:

[0054]

[0055]

[0056] (6) The server generates binary mutual information of other target words in the multiple target words to obtain the binary mutual information of each target word.

[0057] 105. Obtain an adjustment factor for each target word among the multiple target words according to a preset algorithm.

[0058] The server obtains an adjustment factor for each of the multiple target terms based on a preset algorithm. Specifically, the server determines the current business scenario based on the target question text; the server classifies the multiple target terms into slang terms and key terms based on the current business scenario; the server sets the adjustment factor corresponding to the slang terms to a negative number and sets the adjustment factor corresponding to the key terms to a positive number; and the server generates an adjustment factor for each target term.

[0059] For example, if you want to lower the weight of the catchphrase "hello," you can initialize it with a μ value less than 0 (the default value is -1). If you want to increase the weight of the keyword "Fu Bao Bao," you can initialize it with a μ value greater than 0 (the default value is 1). This means you can initialize a weighted bag of words: ["hello": -1, "Fu Bao Bao": 1].

[0060] It should be noted that μ can also be adjusted. For example, the keyword A extracted by the word weight calculated by the BOOST algorithm is compared with the preset keyword B. If A is related to B, then μ is adjusted. a Fine-tune to get μ a =μ a +w, where w can be set to μ a 1 / 100 of the initial value. If word A is not related to word B, then μ a Fine-tune to get μ a =μ a -w.

[0061] 106. Calculate the weight value of each target word according to the adjustment factor of each target word and the binary mutual information of each target word.

[0062] The server calculates the weight value of each target word based on the adjustment factor of each target word and the binary mutual information of each target word. Specifically, the server selects a target word as the second target word from multiple target words, and determines the adjustment factor μ x corresponding to the second target word and the binary mutual information mi(x); the server calculates the weight value f(x) of the second target word according to a preset formula, and the preset calculation formula is: f(x) = mi(x) * TF * IDF + μ x ; the server calculates the weight values of other target words among multiple target words to obtain the weight value of each target word.

[0063] For example, set "Fubaobao" as word A and "Rights and Interests" as word B. If calculated according to the existing TF-IDF algorithm, it can be seen that in the corpus w1, TF * IDF A <TF * IDF B , then the weight of "Rights and Interests" is greater than the weight of "Fubaobao", but the word "Fubaobao" is more representative. However, due to the concentrated distribution in a certain corpus and the strong dependence of the TF-IDF algorithm on word frequency, the weight of "Fubaobao" is weakened. Therefore, it is necessary to adjust the weight value of the word.

[0064] For example, for "Fubaobao", assume "Fubaobao" is word A. To increase the weight value of word A, the adjustment factor needs to be set greater than 0. Here, set μ A = 1, then the calculation process of the weight value is as follows:

[0065] For example, for "Rights and Interests", assume "Rights and Interests" is word B. To decrease the weight value of word B, the adjustment factor needs to be set less than 0. Here, set μ A = -1, then the calculation process of the weight value is as follows: Therefore, f(A) > f(B), that is, f(Fubaobao) > f(Rights and Interests).

[0066] 107. Determine the keywords of the target problem text according to the weight value of each target word, and retrieve the corresponding answers according to the keywords.

[0067] The server determines the keywords of the target problem text according to the weight value of each target word, and retrieves the corresponding answers according to the keywords.

[0068] It can be understood that one keyword corresponds to one or more answers. For example, if the keyword is "Rate of Return", the corresponding answers can be: "5%", "10%", or "20%", and can also be other values. If the keyword is "Scenario", the corresponding answers can be: "Loan", "Regular Financial Management", or "Mortgage", and can also be other scenario types.

[0069] In the embodiment of the present invention, an adjustment factor is added to adjust the weight value of the word frequency, and the mutual information factor is used to eliminate the influence of different document distributions on the weight value, thereby reducing the dependence of the TF-IDF algorithm on the word frequency, improving the accuracy of the weight value, and increasing the accuracy of keyword extraction, thereby retrieving effective message responses.

[0070] See also Figure 2 Another flow chart of the word search method based on binary mutual information provided by an embodiment of the present invention specifically includes:

[0071] 201. Obtain a target question text sent by a target user, where the target question text is used to indicate obtaining an answer corresponding to the target question text.

[0072] The server obtains the target question text sent by the target user, and the target question text is used to indicate the answer corresponding to the target question text.

[0073] For example, in a chat customer service robot, the target user asks a question "What are the benefits of Fubaobao insurance?" The server needs to retrieve the relevant benefits of "Fubaobao" and use the retrieved benefits as the answer to the question.

[0074] It is understandable that the execution subject of the present invention may be a word search device based on binary mutual information, or a terminal or a server, which is not limited here. The embodiment of the present invention is described by taking a server as the execution subject as an example.

[0075] 202. Segment the target question text to obtain multiple candidate words, each candidate word is unique.

[0076] The server segments the target question text to obtain multiple candidate words, each of which is unique. In this embodiment, the server uses a preset segmentation algorithm to segment the target question text. The preset segmentation algorithms include a string matching-based segmentation algorithm, a comprehension-based segmentation algorithm, and a statistics-based segmentation algorithm.

[0077] It should be noted that: (1) For the word segmentation algorithm based on string matching, it matches the Chinese character string to be analyzed with the entries in a "sufficiently large" machine dictionary according to a certain strategy. If a certain string is found in the dictionary, the matching is successful (a word is recognized). According to the different scanning directions, the string matching word segmentation method can be divided into forward matching and reverse matching; according to the situation of preferentially matching different lengths, it can be divided into maximum (longest) matching and minimum (shortest) matching; according to whether it is combined with the词性标注 process, it can also be divided into a simple word segmentation method and an integrated method that combines word segmentation and annotation. (2) For the word segmentation algorithm based on understanding, it enables the computer to simulate the human understanding of sentences to achieve the effect of recognizing words. Its basic idea is to perform syntactic and semantic analysis while segmenting words, and use syntactic and semantic information to handle ambiguity phenomena. It usually includes three parts: a word segmentation subsystem, a syntactic and semantic subsystem, and a general control part. Under the coordination of the general control part, the word segmentation subsystem can obtain syntactic and semantic information about words, sentences, etc. to judge word segmentation ambiguity, that is, it simulates the human understanding process of sentences. (3) For the word segmentation algorithm based on statistics, formally, a word is a stable combination of characters. Therefore, in the context, the more frequently adjacent characters appear together, the more likely they are to form a word. Therefore, the frequency or probability of adjacent co-occurrence of characters can better reflect the credibility of word formation. The frequency of combinations of adjacent co-occurring characters in the corpus can be statistically calculated, and their mutual information can be calculated. The mutual information reflects the tightness of the combination relationship between Chinese characters. When the tightness is higher than a certain threshold, this character group can be considered likely to form a word. It can be understood that other word segmentation methods can also be adopted, and specific details are not limited here.

[0078] For example, in a chat customer service robot, the target user asks a question "What are the rights and interests of Fubaobao Insurance?", and after word segmentation, words such as "Fubaobao", "Insurance", "Rights and Interests", "Which", "Have" are obtained.

[0079] 203. Perform stop word filtering on multiple candidate words to obtain multiple target words.

[0080] The server performs stop word filtering on multiple candidate words to obtain multiple target words. Among them, stop words can include functional words, such as Chinese words like "that", "those", "and", or English words like "the", "a", "an", "that", and "those".

[0081] 204. Call the pre-set corpus to determine the term frequency TF of each target word among the multiple target words.

[0082] The server uses a pre-set corpus to determine the term frequency (TF) of each target term in the plurality of target terms. Specifically, the server obtains the pre-set corpus and identifies target documents in the pre-set corpus. The server determines the number of occurrences (T) of each target term in the target documents and generates the term frequency (TF) for each target term.

[0083] For example, the existing corpus w1,w2,w3,...,w 10 ,corpus w1,w2,w3,...,w 10 The number of words in the corpus is 10, among which "Fu Bao Bao" is concentrated in the corpus documents w1 and w2, and the number of occurrences of "Fu Bao Bao" in w1 and w2 is 1, so the word frequency TF of "Fu Bao Bao" is 1; "Qingyi" is distributed in the corpus documents w1, w2, and w4, and the number of occurrences of "Qingyi" in each corpus document is 10, so the word frequency TF of "Qingyi" is 10.

[0084] 205. Call a preset corpus to determine the inverse document frequency (IDF) of each target word in the plurality of target words.

[0085] The server calls a preset corpus to determine the inverse document frequency (IDF) of each target word among multiple target words. Specifically, the server obtains the preset corpus and determines the total number M of corpus documents in the preset corpus; the server determines the number Wi of documents containing the first target word in the M corpus documents, where i is a positive integer and the first target word is any one of the multiple target words; the server calls a first preset formula, the number Wi of documents, and the total number M to generate the inverse document frequency (IDF) of the first target word, where the first preset formula is IDF=log2(M / Wi+1); the server generates the inverse document frequency (IDF) for each target word.

[0086] For example, the existing corpus w1,w2,w3,...,w 10 ,corpus w1,w2,w3,...,w 10 The number of words in the corpus is 10, and "Fu Bao Bao" is concentrated in the corpus documents w1 and w2. The number of occurrences of "Fu Bao Bao" in w1 and w2 is 1. The inverse document frequency IDF value of "Fu Bao Bao" is “Equity” is distributed in the w1, w2, and w4 corpus documents. “Equity” appears 10 times in each corpus document. The inverse document frequency (IDF) of “Equity” is

[0087] 206. Calculate the binary mutual information of each target word in the multiple target words according to a preset formula.

[0088] The server calculates the binary mutual information of each target word among multiple target words according to a preset formula. Specifically, it includes:

[0089] (1) The server selects any one target word from multiple target words as a candidate target word.

[0090] (2) The server determines the count of occurrences of the candidate target word in two consecutive corpus documents Get the first count.

[0091] in, For the target word X in two consecutive documents w i ,w i+1 Occurrence count, for example: documents w1, w2, w3 all contain word A, then

[0092] (3) The server determines the count of occurrences of the candidate target word in two sequential corpus documents As the second count.

[0093] in, For the target word X in two sequential documents w i ,w i+k Occurrence count, for example: documents w1, w2, w3 all contain word A, then

[0094] (4) The server calculates the ratio of the first count to the second count and obtains the first ratio p(X|w i ,w i+k ,w i+1 );

[0095] in, It is the ratio of the count of word X appearing in two consecutive adjacent documents to the count of word A appearing in two sequential documents. For example, w1 and w2 are consecutive documents, and w1 and w3 are non-consecutive documents; w1, w2 and w1, w3 are sequential documents, and w2, w1 is non-sequential document.

[0096] (5) The server calculates the value of the first ratio p(X|w i ,w i+k ,w i+1 ) Determine the binary mutual information mi(x,w) of the candidate target words i ,w i+k ,w i+1 )=log2p(X|w i ,w i+k ,w i+1 );

[0097] For example, for keywords A and B, there are 10 corpus documents w1, w2, w3, ..., w 10, where the corpus document w1 contains x1 keywords A and y1 keywords B; the corpus document w2 contains x2 keywords A and y2 keywords B; the corpus document w 10 Contains y3 keywords B. The binary mutual information factors of keywords A and B are calculated as mi(A) and mi(B) respectively. The specific calculation process is as follows:

[0098]

[0099]

[0100] (6) The server generates binary mutual information of other target words in the multiple target words to obtain the binary mutual information of each target word.

[0101] 207. Obtain an adjustment factor for each target word among the multiple target words according to a preset algorithm.

[0102] The server obtains an adjustment factor for each of the multiple target terms based on a preset algorithm. Specifically, the server determines the current business scenario based on the target question text; the server classifies the multiple target terms into slang terms and key terms based on the current business scenario; the server sets the adjustment factor corresponding to the slang terms to a negative number and sets the adjustment factor corresponding to the key terms to a positive number; and the server generates an adjustment factor for each target term.

[0103] For example, if you want to lower the weight of the catchphrase "hello," you can initialize it with a μ value less than 0 (the default value is -1). If you want to increase the weight of the keyword "Fu Bao Bao," you can initialize it with a μ value greater than 0 (the default value is 1). This means you can initialize a weighted bag of words: ["hello": -1, "Fu Bao Bao": 1].

[0104] It should be noted that μ can also be adjusted. For example, the keyword A extracted by the word weight calculated by the BOOST algorithm is compared with the preset keyword B. If A is related to B, then μ is adjusted. a Fine-tune to get μ a =μ a +w, where w can be set to μ a 1 / 100 of the initial value. If word A is not related to word B, then μ a Fine-tune to get μ a =μ a -w.

[0105] 208. Calculate the weight value of each target word according to the adjustment factor of each target word and the binary mutual information of each target word.

[0106] The server calculates the weight value of each target word based on the adjustment factor of each target word and the binary mutual information of each target word. Specifically, the server selects a target word as the second target word from multiple target words, and determines the adjustment factor μ x corresponding to the second target word and the binary mutual information mi(x); the server calculates the weight value f(x) of the second target word according to a preset formula, and the preset calculation formula is: f(x) = mi(x) * TF * IDF + μ x ; the server calculates the weight values of other target words among the multiple target words to obtain the weight value of each target word.

[0107] For example, set "Fubaobao" as word A and "Rights and interests" as word B. If calculated according to the existing TF-IDF algorithm, it can be seen that in corpus w1, TF * IDF A <TF * IDF B , then the weight of "Rights and interests" is greater than the weight of "Fubaobao", but the word "Fubaobao" is more representative. However, due to its concentrated distribution in a certain corpus and the strong dependence of the TF-IDF algorithm on word frequency, the weight of "Fubaobao" is weakened. Therefore, it is necessary to adjust the weight value of the word.

[0108] For example, for "Fubaobao", assuming "Fubaobao" is word A, to increase the weight value of word A, the adjustment factor needs to be set greater than 0. Here, set μ A = 1, then the calculation process of the weight value is as follows:

[0109] For example, for "Rights and interests", assuming "Rights and interests" is word B, to decrease the weight value of word B, the adjustment factor needs to be set less than 0. Here, set μ A = -1, then the calculation process of the weight value is as follows: Therefore, f(A) > f(B), that is, f(Fubaobao) > f(Rights and interests).

[0110] 209. Determine the keywords of the target problem text according to the weight value of each target word, and retrieve the corresponding answers according to the keywords.

[0111] The server determines the keywords of the target problem text according to the weight value of each target word, and retrieves the corresponding answers according to the keywords.

[0112] It can be understood that one keyword corresponds to one or more answers. For example, if the keyword is "Rate of return", the corresponding answers can be: "5%", "10%", or "20%", and can also be other values. If the keyword is "Scenario", the corresponding answers can be: "Loan", "Regular financial management", or "Mortgage", and can also be other scenario types.

[0113] In the technical solution provided by the embodiment of the present invention, a target question text sent by a target user is obtained, and the target question text is used to indicate the answer corresponding to the target question text; the target question text is segmented to obtain multiple candidate words, each candidate word is unique; a preset corpus is called to determine the term frequency TF and inverse document frequency IDF of multiple target words in the multiple candidate words; the binary mutual information of each target word in the multiple target words is calculated according to a preset formula; an adjustment factor of each target word in the multiple target words is obtained according to a preset algorithm; a weight value of each target word is calculated based on the adjustment factor of each target word and the binary mutual information of each target word; the keywords of the target question text are determined based on the weight value of each target word, and the corresponding answer is retrieved based on the keyword. In the embodiment of the present invention, an adjustment factor is added to adjust the weight value of the term frequency, and the mutual information factor is used to eliminate the influence of different document distribution conditions on the weight value, reduce the dependence of the TF-IDF algorithm on the term frequency, improve the accuracy of the weight value, increase the accuracy of keyword extraction, and thus retrieve effective message responses.

[0114] The above describes the word search method based on binary mutual information in the embodiment of the present invention. The following describes the word search device based on binary mutual information in the embodiment of the present invention. Figure 3 An embodiment of a word search device based on binary mutual information in an embodiment of the present invention includes:

[0115] The first acquisition unit 301 is configured to acquire a target question text sent by a target user, wherein the target question text is used to indicate an answer to the target question text;

[0116] A word segmentation unit 302 is used to segment the target question text to obtain multiple candidate words, each candidate word is unique;

[0117] A calling determination unit 303 is configured to call a preset corpus to determine the term frequency (TF) and the inverse document frequency (IDF) of a plurality of target words in the plurality of candidate words;

[0118] A first calculation unit 304 is configured to calculate the binary mutual information of each target word in the plurality of target words according to a preset formula;

[0119] A second acquiring unit 305 is configured to acquire an adjustment factor for each target word in the plurality of target words according to a preset algorithm;

[0120] A second calculation unit 306 is configured to calculate a weight value of each target word based on the adjustment factor of each target word and the binary mutual information of each target word;

[0121] The determination and retrieval unit 307 is configured to determine keywords of the target question text according to the weight value of each target word, and retrieve corresponding answers based on the keywords.

[0122] In an embodiment of the present invention, an adjustment factor is added to adjust the weight value of the word frequency, and a mutual information factor is used to eliminate the influence of different document distributions on the weight value, thereby reducing the dependence on the word frequency, improving the accuracy of the weight value, and increasing the accuracy of keyword extraction, thereby retrieving effective message responses.

[0123] See also Figure 4 Another embodiment of the word search device based on binary mutual information in the embodiment of the present invention includes:

[0124] The first acquisition unit 301 is configured to acquire a target question text sent by a target user, wherein the target question text is used to indicate an answer to the target question text;

[0125] A word segmentation unit 302 is used to segment the target question text to obtain multiple candidate words, each candidate word is unique;

[0126] A calling determination unit 303 is configured to call a preset corpus to determine the term frequency (TF) and the inverse document frequency (IDF) of a plurality of target words in the plurality of candidate words;

[0127] A first calculation unit 304 is configured to calculate the binary mutual information of each target word in the plurality of target words according to a preset formula;

[0128] A second acquiring unit 305 is configured to acquire an adjustment factor for each target word in the plurality of target words according to a preset algorithm;

[0129] A second calculation unit 306 is configured to calculate a weight value of each target word based on the adjustment factor of each target word and the binary mutual information of each target word;

[0130] The determination and retrieval unit 307 is configured to determine keywords of the target question text according to the weight value of each target word, and retrieve corresponding answers based on the keywords.

[0131] Optionally, the calling determination unit 303 includes:

[0132] A filtering module 3031 is configured to perform stop word filtering on the plurality of candidate words to obtain a plurality of target words;

[0133] A first determining module 3032 is configured to call a preset corpus to determine a word frequency (TF) of each target word in a plurality of target words;

[0134] The second determining module 3033 is configured to call a preset corpus to determine the inverse document frequency (IDF) of each target word in the plurality of target words.

[0135] Optionally, the first determining module 3032 is specifically configured to:

[0136] A preset corpus is obtained, and target corpus documents in the preset corpus are determined; the number of occurrences T of each target word in the target corpus documents is determined, and a word frequency TF of each target word is generated.

[0137] Optionally, the second determining module 3033 is specifically configured to:

[0138] Obtain a preset corpus and determine the total number M of corpus documents in the preset corpus; determine the number Wi of documents containing a first target word in the M corpus documents, where i is a positive integer and the first target word is any one of the multiple target words; call a first preset formula, the number of documents Wi and the total number M to generate an inverse document frequency IDF for the first target word, where the first preset formula is IDF=log2(M / Wi+1); generate an inverse document frequency IDF for each target word.

[0139] Optionally, the first calculation unit 304 is specifically configured to:

[0140] Select any one of the multiple target words as a candidate target word; determine the count of occurrences of the candidate target word in two consecutive corpus documents Get a first count; determine the count of the candidate target word appearing in two sequential corpus documents As the second count; calculate the ratio of the first count and the second count to obtain a first ratio p(X|w i ,w i+k ,w i+1 ); According to the first ratio p(X|w i ,w i+k ,w i+1 ) Determine the binary mutual information mi(x,w i ,w i+k ,w i+1 )=log2p(X|w i ,w i+k ,w i+1 ); Generate binary mutual information of other target words in the multiple target words to obtain binary mutual information of each target word.

[0141] Optionally, the second acquiring unit 305 is specifically configured to:

[0142] Determine the current business scenario based on the target question text; divide the multiple target words into common sense words and key words based on the current business scenario; set the adjustment factor corresponding to the common sense words to a negative number, and set the adjustment factor corresponding to the key words to a positive number; generate an adjustment factor for each target word.

[0143] Optionally, the second calculation unit 306 is specifically configured to:

[0144] Select a target word from multiple target words as the second target word, and determine the adjustment factor μ corresponding to the second target word. x and binary mutual information mi(x); the weight value f(x) of the second target word is calculated according to the preset formula, and the preset calculation formula is: f(x)=mi(x)*TF*IDF+μ x ; Calculate the weight values ​​of other target words in the multiple target words to obtain the weight value of each target word.

[0145] In an embodiment of the present invention, a target question text sent by a target user is obtained, the target question text being used to indicate the answer corresponding to the target question text; the target question text is segmented to obtain multiple candidate words, each candidate word being unique; a preset corpus is called to determine the term frequency (TF) and inverse document frequency (IDF) of multiple target words in the multiple candidate words; the binary mutual information of each target word in the multiple target words is calculated according to a preset formula; an adjustment factor of each target word in the multiple target words is obtained according to a preset algorithm; a weight value of each target word is calculated based on the adjustment factor of each target word and the binary mutual information of each target word; keywords of the target question text are determined based on the weight value of each target word, and the corresponding answer is retrieved based on the keyword. In an embodiment of the present invention, an adjustment factor is added to adjust the weight value of the term frequency, and a mutual information factor is used to eliminate the influence of different document distribution conditions on the weight value, thereby reducing the dependence on the term frequency, improving the accuracy of the weight value, and increasing the accuracy of keyword extraction, thereby retrieving effective message responses.

[0146] above Figures 3 and 4 The word search device based on binary mutual information in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The word search device based on binary mutual information in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0147] Figure 5This is a structural diagram of a word retrieval device based on binary mutual information provided by an embodiment of the present invention. The word retrieval device 500 based on binary mutual information may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 501 (for example, one or more processors) and a memory 509, and one or more storage media 508 (for example, one or more massive storage devices) for storing application programs 507 or data 506. Among them, the memory 509 and the storage medium 508 can be short-term storage or permanent storage. The program stored in the storage medium 508 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the word retrieval device based on binary mutual information. Furthermore, the processor 501 can be configured to communicate with the storage medium 508 to execute a series of instruction operations in the storage medium 508 on the word retrieval device 500 based on binary mutual information.

[0148] The word search device 500 based on binary mutual information may further include one or more power supplies 502, one or more wired or wireless network interfaces 503, one or more input and output interfaces 504, and / or one or more operating systems 505, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 5 The structure of the word search device based on binary mutual information shown in the figure does not constitute a limitation on the word search device based on binary mutual information. It can include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. The processor 501 can perform the functions of the first acquisition unit 301, word segmentation unit 302, call determination unit 303, first calculation unit 304, second acquisition unit 305, second calculation unit 306, and determination and retrieval unit 307 in the above-mentioned embodiment.

[0149] The following combination Figure 5 A detailed introduction to the various components of the word retrieval device based on binary mutual information:

[0150] Processor 501 is the control center of the binary mutual information-based word search device and can process according to the set binary mutual information-based word search method. Processor 501 uses various interfaces and lines to connect the various parts of the entire binary mutual information-based word search device. By running or executing software programs and / or modules stored in memory 509 and calling data stored in memory 509, it performs various functions of the binary mutual information-based word search device and processes data, thereby reducing the TF-IDF algorithm's dependence on word frequency, improving the accuracy of weight values, and increasing the accuracy of keyword extraction, thereby retrieving valid message responses. Storage medium 508 and memory 509 are both carriers for storing data. In the embodiment of the present invention, storage medium 508 can refer to an internal memory with a small storage capacity but a fast speed, while memory 509 can be an external memory with a large storage capacity but a slow storage speed.

[0151] The memory 509 can be used to store software programs and modules. The processor 501 executes various functional applications and data processing of the word search device 500 based on binary mutual information by running the software programs and modules stored in the memory 509. The memory 509 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function (such as segmenting the target question text to obtain multiple candidate words, each candidate word is unique), etc.; the data storage area can store data created based on the use of the word search device based on binary mutual information (such as the weight value of each target word, etc.). In addition, the memory 509 can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The word search method program based on binary mutual information provided in the embodiment of the present invention and the received data stream are stored in the memory. When needed, the processor 501 calls it from the memory 509.

[0152] When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via a wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that includes one or more available media integrations. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., an optical disk), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0153] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0156] In addition, the functional units in the embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0157] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc., various media that can store program code.

[0158] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A word retrieval method based on binary mutual information, characterized in that: include: Obtaining a target question text sent by a target user, wherein the target question text is used to indicate obtaining an answer corresponding to the target question text; Segmenting the target question text to obtain multiple candidate words, each candidate word is unique; Calling a preset corpus to determine the term frequency TF and the inverse document frequency IDF of multiple target words among the multiple candidate words; Calculate the binary mutual information of each target word in the plurality of target words according to a preset formula; Obtaining an adjustment factor for each target word among the plurality of target words according to a preset algorithm; The weight value of each target word is calculated according to the adjustment factor of each target word and the binary mutual information of each target word; Determine the keywords of the target question text according to the weight value of each target word, and retrieve the corresponding answer according to the keyword; The calculating the binary mutual information of each target word in the plurality of target words according to a preset formula includes: Selecting any one of the plurality of target words as a candidate target word; Determine the count of occurrences of the candidate target word in two consecutive corpus documents , get the first count; Determine the count of occurrences of the candidate target word in two sequential corpus documents , as the second count; Calculate the ratio of the first count to the second count to obtain a first ratio ; According to the first ratio Determine the binary mutual information of the candidate target words ; Generate binary mutual information of other target words in the plurality of target words to obtain binary mutual information of each target word; The step of obtaining the adjustment factor of each target word among the plurality of target words according to a preset algorithm includes: Determine the current business scenario based on the target question text; Dividing the plurality of target words into common words and key words based on the current business scenario; The adjustment factor corresponding to the saliva words is set to a negative number, and the adjustment factor corresponding to the key words is set to a positive number; Generate adjustment factors for each target word; The weight value of each target word is calculated based on the adjustment factor of each target word and the binary mutual information of each target word, including: Selecting a target word from among the multiple target words as a second target word, and determining an adjustment factor corresponding to the second target word and binary mutual information ; The weight value of the second target word is obtained according to the preset calculation formula , the preset calculation formula is: ; The weight values ​​of other target words in the multiple target words are calculated to obtain the weight value of each target word.

2. The word search method based on binary mutual information according to claim 1, characterized in that: The calling of a preset corpus to determine the term frequency TF and the inverse document frequency IDF of the plurality of target words in the plurality of candidate words includes: Performing stop word filtering processing on the multiple candidate words to obtain multiple target words; Calling a preset corpus to determine the word frequency TF of each target word among a plurality of target words; A preset corpus is called to determine an inverse document frequency (IDF) of each target word in a plurality of target words.

3. The word search method based on binary mutual information according to claim 2, characterized in that: The calling of a preset corpus to determine the word frequency TF of each target word in a plurality of target words includes: Obtaining a preset corpus and determining a target corpus document in the preset corpus; The number of occurrences T of each target word in the target corpus document is determined, and a word frequency TF of each target word is generated.

4. The word search method based on binary mutual information according to claim 2, characterized in that: The calling of a preset corpus to determine the inverse document frequency (IDF) of each target word in a plurality of target words includes: Obtain a preset corpus, and determine the total number M of corpus documents in the preset corpus; Determine the number of documents that contain the first target word in the M corpus documents , is a positive integer, and the first target word is any one of the multiple target words; Call the first preset formula, the number of documents and the total number M, to generate the inverse document frequency IDF of the first target word, the first preset formula is: ; Generate the inverse document frequency (IDF) for each target word.

5. A word search device based on binary mutual information, characterized in that: include: A first acquisition unit is used to acquire a target question text sent by a target user, wherein the target question text is used to indicate an answer corresponding to the target question text; A word segmentation unit, used for segmenting the target question text to obtain multiple candidate words, each candidate word is unique; A calling determination unit is used to call a preset corpus to determine the term frequency TF and the inverse document frequency IDF of multiple target words in the multiple candidate words; A first calculation unit, configured to calculate the binary mutual information of each target word in the plurality of target words according to a preset formula; A second acquisition unit, configured to acquire an adjustment factor of each target word among the plurality of target words according to a preset algorithm; A second calculation unit, used for calculating a weight value of each target word according to an adjustment factor of each target word and binary mutual information of each target word; A determination retrieval unit is used to determine the keywords of the target question text according to the weight value of each target word, and retrieve the corresponding answer according to the keyword; The calculating the binary mutual information of each target word in the plurality of target words according to a preset formula includes: Selecting any one of the plurality of target words as a candidate target word; Determine the count of occurrences of the candidate target word in two consecutive corpus documents , get the first count; Determine the count of occurrences of the candidate target word in two sequential corpus documents , as the second count; Calculate the ratio of the first count to the second count to obtain a first ratio ; According to the first ratio Determine the binary mutual information of the candidate target words ; Generate binary mutual information of other target words in the plurality of target words to obtain binary mutual information of each target word; The step of obtaining the adjustment factor of each target word among the plurality of target words according to a preset algorithm includes: Determine the current business scenario based on the target question text; Dividing the plurality of target words into common words and key words based on the current business scenario; The adjustment factor corresponding to the saliva words is set to a negative number, and the adjustment factor corresponding to the key words is set to a positive number; Generate adjustment factors for each target word; The weight value of each target word is calculated based on the adjustment factor of each target word and the binary mutual information of each target word, including: Selecting a target word from among the multiple target words as a second target word, and determining an adjustment factor corresponding to the second target word and binary mutual information ; The weight value of the second target word is obtained according to the preset calculation formula , the preset calculation formula is: ; The weight values ​​of other target words in the multiple target words are calculated to obtain the weight value of each target word.

6. A word search device based on binary mutual information, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the word retrieval method based on binary mutual information as claimed in any one of claims 1 to 4 is implemented.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the word retrieval method based on binary mutual information as claimed in any one of claims 1 to 4 is implemented.

Images