Computer implementation methods for improving search engine queries

Abstract

To provide a computer implemented method for improving search engine queries.SOLUTION: A computer implemented method for improving search engine queries comprises the following steps of: (a) receiving a text corpus; (b) determining a list of n-gram candidates, each being a series of consecutive words of the text corpus, the number of the consecutive words within the series being an integer n superior or equal to two; (c) at least partially modifying the text corpus based on the list of n-gram candidates; (d) performing a machine learning embedding on the resulting text corpus; (e) for each element in the list of n-gram candidates, computing a score based on the embedding of the element and the embeddings of the words making up the element; and (f) adding one or more of the n-gram candidates to a search engine queries items list based on their respective scores.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a computer-implemented method for improving search engine queries.

Background Art

[0002] The field of search engines is not limited to the crawling of the Internet. Search engines are becoming an increasingly important part of a company's information system, especially for purposes of design and acquisition.

[0003] One problem with such search engines is that the need for search accuracy is far more important than for general applications. Therefore, it is important to establish vocabulary related to documents in order to improve this accuracy. One problem in establishing these vocabularies is that, often, the most important meaning-bearing elements are not single words but groups of words, also known as phrases.

[0004] In fact, in natural language, words are not the exact units of meaning. This is because some words have multiple meanings (homonyms), and some meanings are expressed by multiple words.

[0005] Therefore, it is very important to be able to detect phrases within a document. Of course, this can all be done "manually". However, there is a large bias regarding human indexing. Phrase detection is also necessary for the (semi)-automation of knowledge graphs, thesauruses, taxonomies, and ontologies, which includes proposing or automatically adding words and phrases to vocabularies, lists of entities, concepts, and morphological forms.

[0006] State-of-the-art phrase detection techniques use a frequentist approach to find n-grams that occur more frequently than by chance, where an n-gram is a sequence of n consecutive words and n is an integer that cannot exceed the maximum number of words in a sentence of a given text.

[0007] This is typically done using hypothesis tests such as t-tests or chi-squared tests, or by using mutual information (PMI). PMI is approximately proportional to the frequency of n-gram occurrences and inversely proportional to the product of the frequencies of each token occurrence. Note that the expression tokenization means dividing a portion of text into smaller units. The method of tokenization is language-specific, but the most common is dividing text based on spaces and punctuation.

[0008] The problem with using such a frequentist approach to detect new units of meaning within a phrase is that it works for expressions composed of the most frequent expressions and frequently occurring words. This means that less frequent expressions or expressions composed of less frequent words tend to have falsely high or low scores. Also, these methods are affected by frequency bias. In other words, they assume a statistical distribution of word occurrences that does not hold true for many real-world corpora. For example, the use of PMI to detect when groups of words occur more frequently than by chance relies on the assumption that word occurrences are independent and similarly distributed, which is generally not true. In real-world corpora, certain speech is artificially made frequent. For example, in Shakespeare's Hamlet, "Enter Horacio" is a frequently occurring expression not because it is a unit of meaning, but because Hamlet is a play. In the field of industrial components, in contrast to "passive component" or "linear integrated circuit," expressions such as "download part," "request price," and "download model" are commonly found in parts catalogs, but they lack a defined unit of measurement.

[0009] This also applies to most modern natural language processing techniques, which rely on the assumption that a word is a unit of meaning. This is the case with word embeddings and document vectorization (for example, not just vectorizing bags of words, but also using recurrent networks).

[0010] Therefore, there are problems with automatic phrase detection.

[0011] This invention aims to improve this situation. [Overview of the project]

[0012] The present invention aims to improve this situation. To this end, the applicant has provided a computer implementation method for improving search engine queries, a. Steps to receive a text corpus, b. A step of determining a list of n-gram candidates, each being a sequence of consecutive words from a text corpus, where the number of consecutive words in a sequence is an integer n greater than or equal to 2. The steps include modifying the text corpus at least partially based on a list of CN-gram candidates, d. The step of performing machine learning embedding on the resulting text corpus, For each element in the list of en-gram candidates, the step of calculating a score based on the embedding of the element and the embedding of the words that make up the element, f. The step of adding one or more n-gram candidates to the search engine query item list based on their respective scores. We propose a computer implementation method that includes this.

[0013] This method is advantageous because it allows for automated phrase detection that takes into account two or more words that form a new semantic unit, i.e., words that do not individually contribute to the meaning of the text when juxtaposed, but instead, their juxtaposition generates a single new semantic unit by using embeddings. This is the case, for example, with "ice cream" or "bottom line".

[0014] In various embodiments, this method may offer one or more of the following features:

[0015] -Step c includes the steps of parsing the text corpus with the list of n-gram candidates from step b, and replacing the sequence of words with the token associated with the corresponding given element whenever a sequence of words in the text corpus constitutes a given element in the list of n-gram candidates. Step c. further includes, when identifying a set of consecutive words in a text corpus that constitutes a given element of the list of n-gram candidates, determining whether one or more words following a set of consecutive words in the text corpus constitute a different element of the list of n-gram candidates having one or more of the last words of the set of consecutive words in the text corpus; if so, duplicating the set of consecutive words that constitute the different element; and replacing the set of consecutive words with tokens associated with the different element.

[0016] -Step c includes the steps of: copying the text corpus such that each word of the text corpus appears an equal number of times to a sequence of consecutive words n which is an integer greater than or equal to 2; and parsing each copy of the text corpus by parsing each sentence by a sequence of consecutive words n which is an integer greater than or equal to 2, wherein each sentence in the copy is parsed by an offset, the offset being different for each copy; - Step c includes the steps of: parsing the text corpus with the list of n-gram candidates from step b; determining whether, whenever a sequence of consecutive words in the text corpus constitutes a given element in the list of n-gram candidates, one or more words in the sequence of consecutive words in the text corpus constitute a different element in the list of n-gram candidates having one or more of the last words in the sequence of consecutive words in the text corpus; if so, calculating a value to determine whether the sequence of consecutive words constituting the given element should be tokenized as a single word token using a pseudo-random function parameterized by the number of times the sequence of words constituting the given element has already been replaced by tokens associated with the corresponding given element in the already parsed portion of the text corpus; and applying the same process to the sequence of consecutive words constituting the different element. -Step e includes the step of calculating the cosine distance or Euclidean distance between the embedding of an element and the embedding of words that make up the element, - The number of consecutive words in a sequence is equal to 2, and step e. includes the step of calculating the maximum distance between the embedding of an element and the sum of the embeddings of the words that make up a given element, and the distance between the embedding of an element and each separate embedding of the words that make up the element, - The number of consecutive words in a sequence is equal to 3, and step e. includes the step of calculating the maximum distance between the embedding of an element and the sum of the embeddings of two consecutive words that make up a given element and the embeddings of the remaining words that make up the element, - Step b. includes a step that uses the mutual information between words.

[0017] The present invention also relates to a computer system comprising a computer program including instructions for performing the method of the present invention, a data storage medium recording the computer program, and a processor coupled to a memory recording the computer program of claim 10.

[0018] Other features and advantages of the present invention will become apparent from the following description of the exemplary embodiments of the invention and from the accompanying drawings.

Brief Description of the Drawings

[0019] [Figure 1] FIG. 1 is a general schematic diagram of a system according to the present invention. [Figure 2] FIG. 2 shows a comparison of cases of the expression "fast food" and paraphrases of sets of those words.

Modes for Carrying Out the Invention

[0020] The drawings and the following description constitute most of the positively and clearly defined functions. As a result, they are not only useful for understanding the present invention, but can also be used to contribute to its definition if the need arises.

[0021] The description may refer to or use elements that are protected by copyright or are protectable. The applicant does not object to the reproduction of these elements as long as it is limited to the necessary legal publications, but this should not be construed as a waiver of rights or any form of license.

[0022] FIG. 1 shows a general schematic diagram of a system according to the present invention. The system 2 includes a memory 4, a candidate detection unit 6, a corpus correction unit 8, an embedding unit 10, and a scoring unit 12 that returns a list 14 of detected phrases.

[0023] The memory 4 stores text corpus data for which phrase detection is required, as well as any temporary data that may be generated during the execution of the present invention. The memory 4 may also store the list 14 of detected phrases after it has been determined.

[0024] In the examples described herein, memory 4 can be implemented by any suitable means, namely by means such as a hard disk drive, a solid-state drive, flash memory, memory embedded in a processor, or remote storage accessible in the cloud.

[0025] In the examples described herein, the candidate detection unit 6, the corpus correction unit 8, the embedding unit 10, and the scoring unit 12 are computer programs that run on one or more processors. Such processors include any means known to perform automated computation, such as CPUs, GPUs, CPU and / or GPU grids, remote computing grids, specially configured FPGAs, specially configured ASICs, special chips such as SOCs or NOCs, and AI-dedicated chips.

[0026] The candidate detection unit 6 receives the text corpus from memory 4 for phrase detection, and the resulting n-gram candidates are fed to the corpus modification unit 8. The corpus modification unit 8 modifies the text corpus to prepare embeddings of the words contained in the text corpus and to take into account possible overlaps between the n-gram candidates. The embedding unit 10 then performs machine learning embeddings on the modified text corpus and returns embeddings for each unique word in the modified text corpus. Finally, the scoring unit 12 compares the embeddings among them to determine whether the n-gram candidates actually constitute new meanings or whether they are simple juxtapositions of the words that constitute them.

[0027] To illustrate the concept of the present invention, Figure 2 shows the case of the expression “fast food” which is not a paraphrase of the set of words. “Fast food” is semantically different from fast food. The arrows in Figure 2 relate to the embeddings calculated for the respective expressions “fast,” “food,” and “fast food.”

[0028] As shown below, the present invention relies on the use of machine learning word embedding techniques such as word2vec, where the juxtaposition of the meanings of two words is the sum of their respective meanings, and the embedding of this juxtaposition is approximately equal to the sum of the embeddings of the words within the juxtaposition. In the case of "fast food," since it does not specify fast food, the sum of the embeddings of "fast" and "food" is different from the embedding of "fast food."

[0029] In the following steps, the embedding unit 10 proceeds to single-word representations. Therefore, the candidate detection unit 6 and the corpus correction unit 8 are needed to provide a corrected text corpus in which n-gram candidates are in single-word form.

[0030] To determine n-gram candidates, the candidate detection unit 6 may apply one or more of the following operations to the text corpus. - Split the text corpus into sentences to avoid multiple sentences and overlapping n-grams, - The text corpus is divided into noun chunks and verb chunks. -Optionally, the text can be split by stop words. This is useful for text with approximate or inaccurate grammar, but may result in false negatives. Also, phrases containing stop words such as "proof of concept" or "massive" will not be detected as a result. To clarify, "stop words" are a set of very common words such as articles (the, a) and prepositions (of, in, at) and usually do not contribute much to the meaning of the text corpus. Further optional, the text corpus can be filtered by parts of speech, i.e., grammatical groups such as nouns, verbs, and adjectives, into which words are split according to their usage.

[0031] Subsequently, the candidate detection unit 6 may use a tolerance threshold to employ state-of-the-art collocation detection methods, such as PMI, for each remaining text chunk. For example, it may select the n-gram with the highest amount of self-mutual information between words from all possible n-grams in the text corpus. Not all possible n-grams are retained, but the candidate detection unit 6 may be configured to select more n-grams than expected to be obtained in the list of detected phrases 14, e.g., 10, 100, or 1000 times. This temporarily allows for false positives and avoids missed detections. In the context of the present invention, a false positive is an n-gram that has no meaning other than the addition of meanings of the words that make up the n-gram, while a missed detection is an n-gram that is not detected as a set of potential words with a particular meaning. False positives are discarded by the combined work of the embedding unit 10 and the scoring unit 12. For clarity, the expression “words that make up the n-gram” should be understood as meaning the words that are put together make up that n-gram. For example, if the n-gram is “New York City”, the words that make up this n-gram are “New”, “York”, and “City”. In addition to the PMI, other hypothesis tests such as the t-test, chi-squared test, and frequency test can also be used.

[0032] After the candidate detection unit 6 identifies n-gram candidates, the text corpus needs to be modified so that the embedding unit 10 can see the n-gram candidates as single words. This is done by the corpus modification unit 8, which essentially performs two actions. - Convert n-gram candidates in the text corpus into single word tokens. This can be done, for example, by simply splitting tokens by whitespace and replacing whitespace between tokens in n-gram candidates with special characters such as underscores. -Optionally, duplicate part or all of the text corpus to consider overlaps in n-gram candidates.

[0033] To illustrate the concept of overlapping n-gram candidates, let's take the expression "New York City" as an example. This expression contains the following plausible 2-grams "New York" and "York City," as well as the 3-gram "New York City." If Corpus Modification Unit 8 works straightforwardly, this expression will always be tokenized into "New York" and "city," but this is not always necessary.

[0034] In other words, if there is duplication of n-grams, some n-grams will be lost if the representations are not duplicated, either disappearing completely or not appearing frequently enough to generate accurate embeddings. At the same time, systematically duplicating these representations will bias the embeddings by artificially repeating parts of the corpus.

[0035] The applicant identified several methods that can be used to replicate representations in order to account for n-gram overlaps while limiting induced bias.

[0036] According to the first method, the full-text corpus is duplicated, and each sentence is parsed by groups of words equal to n (an integer of n grams, which will be determined later). However, in each separate version of the corpus, the sentence is parsed with an offset of n, and the parsed text is tokenized by retaining the n grams that were detected as single word tokens.

[0037] For example, if n is equal to 2 and the text corpus is "I will use anti lock brake systems", the first version of the text corpus is parsed as "I will", "use anti", "lock break", and "systems", while the second version of the text corpus is parsed as "will use", "anti lock", and "brake system". Given the n-gram candidates "anti lock", "lock brake", and "brake systems", the first version of the text corpus is tokenized as "I, will, use, anti, lock_brake, systems". The second version is tokenized as "will, use, anti_lock, brake_systems".

[0038] Of course, the same thing can be done for other values ​​of n by using an offset in the range of 0 to n-1.

[0039] This method is very easy to implement and greatly limits bias, making it highly advantageous. However, it can be argued that duplication is disadvantageous because the rest of the text corpus is duplicated, but the duplicate n-grams are not.

[0040] According to the second method, the corpus modification unit 8 can flatten the Pareto curve of n-gram frequencies. This second method redistributes unnecessarily frequent words into the n-grams they constitute. It also duplicates content containing rare, overlapping n-grams to boost the training samples (chunks of text) with rare information (rare n-grams) while limiting training bias.

[0041] This method can be implemented as follows: - Select the minimum number of occurrences (MIN) of the target to properly embed the token, - Analyze the text corpus and count all possible occurrences of each n-gram candidate. For example, for the expression "New York City", count the occurrences of "New York", "York City", and "New York City". After this count is performed, discard any n-gram candidates whose count COUNT is less than MIN. - Re-analyze the text corpus, *Whenever a given n-gram candidate is encountered without repetition, it is tokenized as a single word, and the counter SELECT, which indicates the number of times this given n-gram candidate has been tokenized as a single word during analysis, is incremented, while the counter COUNT is decremented. *Whenever a sentence contains a candidate for an n-gram that is duplicated, the sentence is duplicated in the following way: - Set an empty list KEEP, - For each n-gram that is part of the duplicates in the sentence, a pseudo-random number function is used to generate a numerical value following a uniform distribution between 0 and 1. If the score ((MIN-SELECT) / COUNT) exceeds the resulting random number, the n-gram is added to the list KEEP. -Tokenize the sentence by saving at least one n-gram in list KEEP as a single word token (i.e., not splitting the n-gram into multiple tokens) while there are n-grams in list KEEP, add the sentence to the text corpus, and remove all n-grams that exist as single tokens within the tokenized sentence from list. - For each occurrence of an n-gram candidate added as a single token within the resulting tokenized sentence, -Increase the counter SELECT by one, -Decrement the COUNT by 1.

[0042] This method creates only limited bias through replication. Bias becomes a problem if a small portion of the text corpus is artificially replicated multiple times. Conversely, this method replicates chunks containing rare n-grams. Since n-gram frequencies generally follow a Pareto distribution, replication of many chunks is minimal.

[0043] After the text corpus has been modified to tokenize all n-gram candidates according to one of the methods described above, the embedding unit 10 is invoked to perform embedding of all tokens in the modified text corpus, regardless of whether they are single words present in the original text corpus or tokens created by the corpus modification unit 8.

[0044] According to one embodiment, the embedding unit 10 may be a neural network-based embedding such as word2vec and fastText. As described in the paper "Skip-Gram-Zipf+Uniform=Vector Additivity" by Gittens et al., in the minutes of the 55th Annual Meeting of the Computational Linguistics Society (Volume 1: Long Papers) 2017, the additive constructivity property is known in the skip-gram model, that is, the embedding of paraphrases of a set of words is similar to the sum of the embeddings of each word.

[0045] Finally, once the modified text corpus is fully tokenized, the scoring unit 12 can call up a list of embeddings and n-gram candidates.

[0046] The scoring unit 12 applies a scoring function to each n-gram to determine whether the embedding of the n-gram is close to the sum of the embeddings of the words that make up this n-gram. The applicant tested several scoring functions that partially depend on the number n-grams.

[0047] In the case where n is equal to 2, the simplest approach is to compare the distances between them. Thus, taking the example in Figure 1, a distance such as cosine distance or Euclidean distance can be calculated between the embedding of "Fast food" and the sum of the embeddings of "Fast" and "Food," respectively.

[0048] The applicant has discovered another formula that may be particularly useful when detecting n-grams containing stop words. This formula can be summarized as follows for n-gram AB consisting of words A and B: Score(AB)=Max(Distance(Embedding(AB), Embedding(A)+Embedding(B)), Distance(Embedding(AB), Embedding(A)), Distance(Embedding(AB), Embedding(B)) In the case where n is equal to 3, the "naive" formula can be used. That is, in the case of n-gram ABC consisting of words A, B, and C, Score(ABC)=Dsitance(Embedding(A_B_C), Embedding(A)+Embedding(B)+Embedding(C)) This formula can capture formal names such as red_roof_inn (hotel), hilton_grand_vacations (hotel), and gmbh_co_kg (company), because the meaning of the trigram is entirely different from the individual words that make up the trigram.

[0049] The thinking behind the second formula in the case where n is equal to 2 can also be extended. Score(ABC)=Max(Distance(Embedding(A_B_C), Embedding(A_B)+Embedding(C));Distance(Embedding(A_B_C), Embedding(A)+Embedding(B_C))) This formula treats the above trigrams as less important phrases because the individual sets of words (unigrams and bigrams) are used in similar contexts. For example, since Hilton and grand_vacations also refer to hotels, Hilton_grand_vacations is considered a paraphrase of that individual set of words.

[0050] On the other hand, this expression can capture, for example, full_length_mirror and safety_deposit_box. Trigrams have specific meanings, but since trigrams are assumed to simply paraphrase the objects of their individual words (mirror, box), naive expressions are not adapted to capture them.

[0051] For the case where n is greater than 3, the formula for n equal to 3 is easily extended.

[0052] After a score has been calculated for all n-gram candidates, the scoring unit 12 may return a list of detected phrases by holding the n-gram with the highest score.

[0053] Subsequently, this list can be added to the corpus of search engine phrases to improve the quality of queries. In fact, if a user later enters one of the n-grams returned by scoring unit 12 as a search term, the search engine can provide more meaningful returns.

Claims

1. A computer implementation method for improving search engine queries, a. The step of receiving a text corpus, b. A step of determining a list of n-gram candidates, each of which is a sequence of consecutive words from the text corpus, and the number of such sequences of consecutive words is an integer n of 2 or more. c. The step of modifying the text corpus at least partially based on the list of n-gram candidates, d. Step of performing machine learning embedding on the at least partially modified text corpus of step c, e. For each element in the list of n-gram candidates, a step of calculating a score based on the embedding of the element and the embedding of the words that constitute the element; f. The step of adding one or more of the aforementioned n-gram candidates to the search engine query item list based on their respective scores. Computer implementation methods including

2. The computer implementation method according to claim 1, wherein step c. is the step of parsing the text corpus with the list of n-gram candidates of step b. and whenever a series of consecutive words in the text corpus constitutes a given element of the list of n-gram candidates, the step of replacing the series of consecutive words with a token associated with the corresponding given element.

3. Step c. The computer implementation method according to claim 2, further comprising: identifying a series of consecutive words in the text corpus that constitute a given element of the list of n-gram candidates, determining whether one or more words consecutive to the series of consecutive words in the text corpus constitute a different element of the list of n-gram candidates having one or more of the last words of the series of consecutive words in the text corpus; if so, duplicating the series of consecutive words that constitute the different element; and replacing the series of consecutive words with tokens associated with the different element.

4. Step c. is the computer implementation method according to claim 1, which includes the steps of: copying the text corpus such that each word of the text corpus appears a number of times equal to the number of consecutive words in a sequence of words that is an integer n of 2 or more; and analyzing each copy of the text corpus by analyzing each sentence by a sequence of consecutive words that is an integer n of 2 or more, wherein each sentence in the copy is analyzed by an offset, the offset being different for each copy.

5. Step c. comprises the steps of parsing the text corpus with the list of n-gram candidates of step b.; determining whether, whenever a series of consecutive words in the text corpus constitutes a given element of the list of n-gram candidates, one or more words consecutive to the series of consecutive words in the text corpus constitute a different element of the list of n-gram candidates having one or more of the last words of the series of consecutive words in the text corpus; if so, calculating a value to determine whether the series of consecutive words constituting the given element should be tokenized as a single word token using a pseudo-random function parameterized by the number of times the series of consecutive words constituting the given element have already been replaced by tokens associated with the corresponding given element in the already parsed portion of the text corpus; and applying the same process to the series of consecutive words constituting the different element.

6. Step e. is a step of calculating the cosine distance or Euclidean distance between the embedding of the element and the embedding of the words constituting the element, according to any one of claims 1 to 5.

7. The computer implementation method according to any one of claims 1 to 5, wherein the number of the series of consecutive words is equal to 2, and step e. includes the step of calculating the maximum distance between the embedding of the element and the sum of the embeddings of the words that constitute the element, and the distance between the embedding of the element and each separate embedding of the words that constitute the element.

8. The computer implementation method according to any one of claims 1 to 5, wherein the number of the series of consecutive words is equal to 3, and step e. includes the step of calculating the maximum distance between the embedding of the element and the sum of the embeddings of two consecutive words of the words constituting the element and the embeddings of the remaining words constituting the element.

9. Step b. is a step of using the mutual information between words, the computer implementation method according to any one of claims 1 to 8.

10. A computer program comprising instructions for performing the method described in any one of claims 1 to 9.

11. A data storage medium on which the computer program described in claim 10 is recorded.

12. A computer system including a processor coupled to a memory (4), wherein the memory (4) stores the computer program described in claim 10.

Images