RECOGNITION OF NAMED ENTITIES IN SEARCH QUERIES

Abstract

Named entity recognition (NER) in a user search query on a real-time search engine can be achieved by training a machine learning algorithm to create a trained model. The trained model can be configured to receive a user search query as input and to recognize and output zero or more named entities in the search query. The training may include an iterative training process in which more training data is added in each iteration, in some modalities. The training may be based on three sets of training data, in some modalities. A first set of training data may be based on the user's search and engagement activity. A second set of training data may be artificially generated based on a catalog of named entity values.A third training dataset can be based on optimized and supplemented data pairs sampled from the first training dataset.

Description

RECOGNITION OF NAMED ENTITIES IN SEARCH QUERIES Field of Invention This description refers to the recognition of named entities in search queries and other Internet or electronic interface queries. Background of the Invention One of the challenges in understanding user queries on electronic interfaces, such as search queries on a search engine or website, is recognizing the specific named entities the user intends to identify. Named Entity Recognition (NER) is a common information retrieval task for locating, segmenting, and categorizing a predefined set of entities from unstructured text, such as people's names, place names, company names, dates, times, and measurements. Summary of the Invention The first part of this description outlines a method for responding to a search query. This method includes defining a machine learning algorithm, configured to receive a user query as input and to output zero or more entities. Q! Αηη / Ζζηζ / Ε / ΥΙΛΙ Ref. 336890, named from one or more entity types in the user query, each of the one or more entity types comprising a respective plurality of values. The method further includes defining a first dataset according to the user's behavioral data, the first dataset comprising a plurality of first data pairs, each data pair comprising a user search query and one or more named entity values ​​defined in the user search query. The method further includes defining a second dataset which creates a plurality of second artificial data pairs, each second data pair comprising an artificial search query and one or more named entity values ​​defined in the artificial search query.The method further includes training the machine learning algorithm to create a trained model by: (i) defining an initial training dataset comprising a portion of the first dataset; (ii) training the algorithm on the training dataset; (iii) adding more data from the first and second datasets to the training dataset to create an additional training dataset; (iv) training the algorithm on the additional training dataset; and (v) iteratively repeating the process. QI Aᑷη / Zζηζ / E / YΙΛΙ of (iii) and (iv). The method further includes receiving a search query from the user, recognizing one or more named entity values ​​in the search query with the trained model, and outputting a response to the user in accordance with one or more recognized named entity values. In one modality of the first aspect, the definition of the initial training dataset includes determining a plurality of named entity values ​​in the user's search queries of the first dataset that are not in the associated defined named entity values ​​of the first dataset and the portion of the second dataset, and adding the determined plurality of named entity values ​​to the defined named entity values ​​of the first dataset. In one embodiment of the first aspect, outputting a response to the user according to one or more recognized named entity values ​​includes directing the user to a web page dedicated to one or more of the recognized named entity values, filtering a set of search results according to one or more recognized named entity values ​​and outputting the filtered search result set to the user, or displaying the one or more named entity values ​​to the user. QI Αηη / Ζζηζ / Ε / ΥΙΛΙ recognized . In one modality of the first aspect, iteratively repeating (iii) and (iv) comprises iteratively repeating (iii) and (iv) until an accuracy of the trained model exceeds a predetermined threshold or until a predetermined number of iterations have been performed. In one modality of the first aspect, the addition of additional data from the first dataset and the second dataset to the training dataset includes the addition of data pairs for each of a plurality of entity type sequences In one version of the first aspect, the method also includes +evaluating the trained model based on a portion of the first dataset that is not in the initial training data. In one version of the first aspect, the machine learning algorithm includes a character-to-word layer and a word-to-label layer. In one modality of the first aspect, the character-word layer comprises a bidirectional network of short-term memory, and the word-to-label layer comprises a bidirectional network of recurrent units. In a second aspect of this description, a system for responding to a search query is provided. The system includes non-transient memory. A computer-readable instruction set (IAS) that stores instructions and a processor configured to execute the instructions to carry out a method. The method includes the definition of a machine learning algorithm, the algorithm configured to receive a user query as input and output zero or more named entities of one or more entity types in the user query, each of the one or more entity types comprising a respective plurality of values. The method further includes defining a first dataset according to the user's behavioral data, the first dataset comprising a plurality of first data pairs, each data pair comprising a user search query and one or more named entity values ​​defined in the user search query.The method further includes defining a second dataset by creating a plurality of second artificial data pairs, each second data pair comprising an artificial search query and one or more named entity values ​​defined in the artificial search query. The method further includes training the machine learning algorithm to create a trained model by: (i) defining an initial training dataset comprising a portion of the first dataset; (ii) training the algorithm on the training dataset; (iii) adding to it. The method includes: (iv) adding more data from the first dataset and the second dataset to the training dataset to create an additional training dataset; (v) training the algorithm according to the additional training dataset; and (iii) iteratively repeating (iv) and (iii). The method further includes receiving a search query from the user, recognizing one or more named entity values ​​in the search query with the trained model, and outputting a response to the user according to one or more recognized named entity values. In one modality of the second aspect, the definition of the initial training dataset includes determining a plurality of named entity values ​​in the user's search queries of the first dataset that are not in the associated defined named entity values ​​of the first dataset and the portion of the second dataset, and adding the determined plurality of named entity values ​​to the defined named entity values ​​of the first dataset. In one modality of the second aspect, the output of a response to the user in accordance with one or more recognized named entity values ​​includes directing the user to a web page dedicated to one of the one or more values ​​of QI Aηη / Zζηζ / E / YΙΛΙ recognized named entity, which filters a set of search results according to one or more recognized named entity values ​​and outputs the filtered search result set to the user, or displays the one or more recognized named entity values ​​to the user. In one modality of the second aspect, iteratively repeating (iii) and (iv) comprises iteratively repeating (iii) and (iv) until an accuracy of the trained model exceeds a predetermined threshold or until a predetermined number of iterations have been performed. In one modality of the second aspect, adding more data from the first dataset and the second dataset to the training dataset comprises adding data pairs for each of a plurality of entity type sequences. In one modality of the second aspect, the method involves evaluating the trained model based on a portion of the first dataset that is not in the initial training data. In one version of the second aspect, the machine learning algorithm includes a character-to-word layer and a word-to-label layer. In one modality of the second aspect, the character-to-word layer includes a bidirectional memory-to-word network QI Αηη / Zζηζ / E / YΙΛΙ short term, and the word-to-label one includes a bidirectional network of recurrent units. In a third aspect of this description, a method is provided for training a machine learning algorithm to create a trained model. This trained model is designed to recognize one or more named entities in a user search query, each of which has one or more entity types. The method includes defining a first dataset based on user behavior data. This first dataset comprises a plurality of first data pairs, each consisting of a user search query and one or more named entity values ​​defined in the user search query.The method further includes determining a plurality of named entity values ​​in the user's search queries of the first dataset that are not in the associated defined named entity values ​​of the first dataset, and adding the determined plurality of named entity values ​​to the defined named entity values ​​of the first dataset to create a supplemented first dataset. The method further includes defining a second dataset by creating a plurality of artificial second data pairs, each second data pair comprising one. QI Aηη / Zζηζ / E / YΙΛΙ artificial search query and one or more named entity values ​​defined in the artificial search query.The method further includes training the machine learning algorithm to create a trained model by: (i) defining an initial training dataset comprising a portion of the first supplemented training dataset; (ii) training the algorithm on the first supplemented training dataset; (iii) adding more data from the first supplemented training dataset and the second dataset to the initial training dataset to create an additional training dataset; (iv) training the algorithm on the additional training dataset; and (v) iteratively repeating (iii) and (iv) until the accuracy of the trained model exceeds a predetermined threshold or until a predetermined number of iterations have been performed. In one modality of the third aspect, the addition of more data from the first dataset and the second dataset to the training dataset comprises the addition of data pairs for each of a plurality of entity type sequences. In one version of the third aspect, the method also includes evaluating the trained model according to a portion QI Αηη / Zζηζ / E / YΙΛΙ of the first dataset that is not in the initial training data. In one modality of the third aspect, the model is to recognize one or more named entities selected from a predetermined set of named entity values. Brief Description of the Figures Figure 1 is a diagrammatic view of an example system for recognizing named entities in a search query and returning a result to the query. Figure 2 is a flowchart illustrating an example method for recognizing named entities in a search query and returning a result to the query. Figure 3 is a flowchart illustrating an example method of training a machine learning model for named entity recognition in a search query and deploying the trained model. Figure 4a is a diagrammatic view of an example of a character-to-word layer that may find use in the machine learning model trained and deployed in Figure 3. Figure 4b is a diagrammatic view of a word-to-label example that may find use in the machine learning model trained and deployed in Figure 3. QI Αηη / Ζζηζ / Ε / ΥΙΛΙ Figure 5 is a graph illustrating the accuracy scores (Fl) of three different NER approaches. Figure 6 is a schematic view of an example mode of a user computing environment. Detailed Description of the Invention A search engine associated with an electronic user interface, such as an e-commerce website or other website, can process millions, billions, or more search queries per year. A fundamental challenge for a search engine is understanding a search query and extracting attributes from it to help identify the primary objective of the search and retrieve the most appropriate search results. This task can be framed within Named Entity Recognition (NER), which is an information retrieval task for locating, segmenting, and categorizing a predefined set of entities from unstructured text. Although NER research has progressed since the early 1990s, viable industrial applications have not yet been found.The focus of existing research systems is often on marginally improving Fl scores without considering the use at scale of a search engine that processes a large volume of searches. QI Αηη / Ζζηζ / Ε / ΥΙΛΙ There are several challenges in known NER approaches that this description addresses: (1) Industrial applications benefit from customized, domain-specific knowledge and training data that cover the full range of entities of different types, including representative examples of noisy queries (spelling mistakes, abbreviations, etc.) and noisy signals of user intent, such as conversion (purchase) events on an e-commerce website. (2) Deep learning models typically require a large amount of high-quality training data, which is often expensive, time-consuming, and sometimes impossible to acquire. (3) Integrating NER recognition with an existing system often requires considerable effort because NER is not usually the end goal of the application and may require careful design and implementation to have a real-world impact.(4) Deployment is challenging because deep learning models are computationally expensive to train and run in a real-time application such as an e-commerce search engine. This description will refer to entity types in the context of an e-commerce website. These entity types can be, for example, brands, product types, categories, quantities, and attributes. QI Aạ / Zạ / E / YạI of products and others. It should be understood, however, that the teachings of the present description are also applicable to the recognition of named entities in contexts other than electronic commerce, and therefore also to entity types other than those listed above. Named entity recognition (NER) based solely on an ambitious comparison of known entity values ​​with a search query does not offer an optimal solution. For example, queries containing multiple product types or multiple brands but only one target, ambiguity between product types and brands, or new product types not included in a predefined taxonomy can all lead to incorrectly identified named entities. Table 1 lists three examples that illustrate these challenges. A named entity recognition system based on the description presented here is better equipped to address these issues. Q! Αηη / Ζζηζ / Ε / ΥΙΛΙ Table 1 NER Brand Inquiry Product Type NER Actual Brand Actual Product Type Lightweight Brush Cutter Brush Cutter Lightweight Ice Dispenser Brush Cutter Refrigerator without Ice Maker Refrigerator Cosco Table and Chair Set Cosco Table Cosco Table and Chair Set Existing methods for recognizing named entities in user search queries are inaccurate or not fast enough for deployment in a real-time search context, such as an e-commerce website or other website. This description improves upon existing methods to provide robust and computationally efficient named entity recognition (NER) for a website search engine. This description can improve upon known NER approaches by training a machine learning algorithm to create a trained model. The trained model can be configured to receive a user search query as input and to recognize and output zero or more named entities in the search query. The training can include an iterative training process in which more training data is added at each iteration, in some modalities. The training can be based on three training datasets, in some modalities. One or more different machine learning algorithms may find use with this description. For example, in some modalities, a bidirectional recurrent neural network with conditional random fields (RNN-CRF) may be employed. Furthermore, in some modalities, the RNN-CRF may be, or may include, a bidirectional short-term memory (BiLSTM-CRF). QI (Ani / Zzi / E / Yili) with pre-trained word embedding, character embedding, and dropout. Additionally, a closed recurrent unit (GRU) network, a simplified variant of the LSTM, can be employed. With reference now to the figures, where similar numbers refer to the same or similar features in the different views, Figure 1 is a diagrammatic view of an example of System 100 for processing search queries, including the recognition of named entities. System 100 can receive and respond to search queries from users of an electronic interface (such as an e-commerce website or other website). System 100 may include a database 102 of user behavior data, a database 104 of product data, and a search query processing system 106 that may include one or more functional modules 108, 110, 112, 114 embedded in hardware and / or software. In one embodiment, the functional modules 108, 110, 112, 114 of the search query processing system 106 may be embedded in a processor and a memory that stores instructions which, when executed by the processor, cause the processor to perform the functionality of one or more of the functional modules and / or QI Αηη / Ζζηζ / Ε / ΥΙΛΙ other functionality of this description. Functional modules 108, 110, 112, and 114 of the search query processing system 106 may include a training module 108 configured to train one or more machine learning models using data obtained from databases 102, 104, or another training data store. Training module 108 may define multiple training datasets. It may be configured to train one or more machine learning models using the training data. The trained machine learning model(s) may be configured to, given a user search query input, produce one or more named entities recognized in the search query.In some configurations where System 100 is deployed in support of a website or e-commerce platform, Training Module 108 can train one or more machine learning models to recognize one or more brand names and / or one or more product types. Training Module 108 can be configured to train a machine learning algorithm to create a trained model that recognizes one or more named entities, where the one or more named entities are selected from a predetermined list. Q! Αηη / Ζζηζ / Ε / ΥΙΛΙ A named entity recognition module 110 may include the machine learning model(s) trained by training module 108, or a portion thereof, in some configurations. Named entity recognition module 110 may be configured to accept a user search query as input and output one or more named entities recognized in the search query. In some configurations where system 100 is deployed in support of a website or e-commerce platform, NER module 108 may be configured to recognize one or more brand names and / or one or more product types. The named entities recognized by named entity recognition module 110 may be selected from a predefined list. The search query processing system 106 may further include a search module 112 and a search results filter module 114. The search module 112 may be configured to search a set of documents for documents relevant to a user's search query and output a set of documents relevant to the query, and the search results filter module 114 may be configured to apply one or more automatic or user-selected filters to the output of the search module 112. In some QI Aηη / Zζηζ / E / YΙΛΙ modalities, the function of the search module 112 and / or the search results filter module 114 can be in accordance with the output of the NER module 110. System 100 may further include a server 116 in electronic communication with the search query processing system 106 and with a plurality of user computing devices 118i, 1182, . . . 118N. The server 116 may provide a website, data for a mobile application, or another interface through which users of the user computing devices 118 can view products that have data in the product database 104 and through which users can perform search queries. For example, the server 118 may provide an e-commerce website of a retailer that includes listings of one or more products.In some modes, server 118 can receive a search query from a user, provide the search query to the search query processing system 106, receive a set of search results from the search query processing system 106, and provide the set of search results to the user. Figure 2 is a flowchart illustrating an example method for recognizing named entities in a search query and returning a result to the query. QI Αηη / Ζζηζ / Ε / ΥΙΛΙ Referring to Figures 1 and 2, method 200, or one or more portions thereof, can be performed by the search query processing system 106. Method 200 may include, in block 202, defining the training data. In some modalities, the definition of the training data may include the definition of one or more training datasets. For example, the definition of the training data may include the definition of three training datasets in some modalities. A first training dataset may include the user behavior-based training data 204, which may include the user's search queries, the search results they respond to, and the users' brand and product type selections from those search results. Referring to Figures 1 and 2, the user behavior-based training data may be derived from the user behavior data 102 and the product data 104. With continued reference to Figure 2, a second training dataset may include synthetic training data 206. The synthetic training data 206 may include one or more data points that are absent from the user behavior-based training data 204 in some modalities. For example, synthetic training data can include a complete set of data points covering the data space. In a particular example, where method 200 is implemented in support of an e-commerce website, synthetic training data 206 might include brands and / or product types that are not represented in the user behavior-based training data 204. In some modalities, synthetic training data 206 might include all or most of the brands and / or product types that are represented or not represented in the user behavior-based training data 208. A third set of training data may include idealized training data 208. The idealized training data 208 may be a subset of the user behavior-based training data 204 and / or the synthetic training data 206. In one modality, the idealized training data may be curated by the user. Method 200 may also include, in block 210, training a machine learning model with the training data. The training in block 210 may include, in sub-block 212, training the machine learning model on a portion of the idealized training data 208. He QI Aᑷη / Zζηζ / E / YΙΛΙ training initially on idealized training data 208 can provide a first generation of quality training. The training in block 210 may also include, in sub-block 214, applying the model (as initially trained on the idealized training data 208) to the user behavior-based training data 204 to determine high-quality user behavior-based training data 204. Sub-block 214 may operate as a filter on the user behavior-based training data 204, and the resulting high-quality user behavior-based training data may be used for further training. The training in block 210 may also include, in sub-block 216, continued training of the model on the high-quality training data based on user behavior and on the synthetic data 206. Sub-blocks 214 and 216 may be repeated until the generation-to-generation improvement in model accuracy falls below a predetermined threshold, or until a predetermined number of training generations have been completed. In some modalities, after each training generation, the model's accuracy may be evaluated by applying the model to a subset of the data. QI Αηη / Zζηζ / E / YΙΛΙ of idealized training (for example, a subset that was not used to train the model in the first case). Training in block 210 may also include, in some modalities, continuing to train the model on a combination of portions of the user behavior-based training data 204, the synthetic data 206, and the idealized training data 208. In addition, training in block 210 may include selecting portions of the user behavior-based training data 204, the synthetic data 206, and the idealized training data 208 for a particular training generation based on the model's performance after a previous training generation and based on the training data used in previous generations.For example, a current generation model can be used as a filter to select particular data from the user behavior-based training data 204 (e.g., the quality user behavior-based training data mentioned above). Portions of the synthetic data 206 and the idealized training data 208 can be selected for training randomly or according to one or more statistical sampling methods. Once trained, the model can be applied to QI Aηη / Zζηζ / E / YΙΛΙ supports a search engine in recognizing named entities in user search queries. Method 200 may therefore also include, in block 218, the receipt of a search query from a user's computing device. The search query may be made through an e-commerce website, in some cases. The search query may include one or more named entities, such as one or more brands or product types, to be recognized. Method 200 may also include, in block 220, applying the trained model to the search query to recognize one or more named entities in the search query. Method 200 can also include, in block 222, the search method, the filtering of search results, and / or user navigation based on the named entities recognized in the search query. For example, when one or more brands are recognized in the search query, block 222 might include performing a search according to the complete search query and filtering the results to limit them to the recognized brands, or to place the recognized brands at the top of the results. In another example, when one or more product types are recognized in the search query, block 222 might include... QI An / Zzez / E / YILI includes performing the search for one or more product types. In another example, when the complete search query is for a single brand or product type, block 222 can include navigating the user to a brand or product page associated with the named entity. An answer to the search query can be delivered to the user, such as filtered search results, or a particular web page or document associated with a brand or product type, as noted above, and / or through a display of the named entities recognized to the user as a search suggestion or alongside the search results, as a result of block 222. Figure 3 is a flowchart illustrating an example of Method 300 for training a machine learning model for named entity recognition in a search query and deploying the trained model. Method 300 can be considered triple-learning, a novel approach to machine learning training for named entity recognition. Method 300, or one or more parts of it, can be performed by the search query processing system 106. The flowchart is generally illustrated in three parts: a training data preparation phase 302, a phase of QI Αηη / Zζηζ / E / YΙΛΙ iterative training of model 304, and a deployment phase of model 306. Phase I: Preparation of training data 302. A desirable training dataset for a deep learning model might have three characteristics: (1) large size; (2) high-quality labels; and (3) high coverage of all label values ​​(e.g., all brands and product types in this problem). However, in many applications, preparing a single training dataset that meets all three characteristics is too time-consuming and expensive. Consequently, three separate datasets can be defined, each addressing one of the three characteristics above, and the three datasets can be used collectively for training. The first two datasets can be based partially (or entirely) on the product catalog data 308 and the customer behavior data 310.Product catalog data 308 can include truth information about all values ​​of all named entity types that the machine learning algorithm will be trained to recognize. Consequently, in a mode where named entity types refer to products (such as brand and product type), product catalog data 308 can. QI An / Zzez / E / YILI includes basic truth information for all available products, encompassing all brands and product types. In other embodiments, a data source may be used that includes basic truth information for another type of named entity (i.e., other than brand and product type, or for information other than products). Customer behavior data 310 may include data concerning customer user interactions with a website where the products reflected in catalog data 308 are available, including searches, impressions, clicks, purchases, and / or other information. In other embodiments, other data concerning user activity may be included in a user behavior database. The method may include, in block 312, applying a rule-based algorithm to the product catalog data 308 and the customer behavior data 310, which matches the tokens in the product catalog data 308 with the tokens in the customer behavior data 310 to automatically generate a first training dataset 314. As a result, the first training dataset 314 may include a plurality of data pairs, each pair including an actual user search query or other user interaction and one or more named entities QI Αηη / Zζηζ / E / YΙΛΙ included in that search query that match an entity with a name from the product catalog data 308. The method may also include, in block 316, sampling a portion of the first training dataset 314 and, in block 318, supplementing the sampled data to prepare a supplemented dataset 320, which, in some modalities, may be considered high-quality gold data. The sampled data may be supplemented by determining one or more named entity values ​​present in the search queries of the sampled data pairs that are not included in the named entity values ​​of those data pairs, and adding the determined named entity values ​​to the relevant data pairs.In other words, when a sampled data pair includes a search query that has named entity values ​​A, B, and C, but the corresponding named entity set for that data pair includes only entity values ​​A and C, block 318 can include adding entity value B to the data pair. In some modes, the supplementation can be manual, or it can include a combination of greedy named entity recognition and manual checking or intervention. Supplemented data set 320 can be used in this process. QI Aηη / Zζηζ / E / YΙΛΙ training of the algorithm and to measure the actual performance of the model in each training iteration, in some modalities. To avoid bias or overfitting of the model trained on a small number of named entity sequence patterns, sampling in block 316 can include stratified sampling of the data by label pattern, so that each label pattern is shown in a relatively equal number. In this case, a pattern is defined as a sequence of entity types, regardless of the individual entity values. For example, Table 2 illustrates four (of many) possible label patterns in a mode where the possible named entity types are brand (BRD) and product (PRD), where 0 represents "outside," i.e., text in a query that is not a named entity. Table 2 BRD O PRO milwaukee economical drill BRD0,PRDO ge 0.21 m3 gas dryer BRD PRD O behr paint on sale OR PRD OR bronze folding tap The method may also include, in block 322, QI Aηη / Zζηζ / E / YΙΛΙ creates a second synthetic dataset 324 that applies a rule-based algorithm to the named entity types and values ​​in the product catalog data 308 to create a plurality of artificial search queries that have different combinations of those entity types and values. The second dataset 324 may include a plurality of data pairs, each data pair including an artificial query and one or more named entity values ​​present in that artificial query. In some embodiments, each entity value of each entity type may be represented at least once in the synthetic dataset 324. Phase II: Iterative Model Training. In the model training phase 304, the machine learning algorithm can be iteratively trained to create a trained model. In each iteration, more training data can be added. The goal is to progressively improve the model's performance (Fl score). Phase 304 of model training may include defining a machine learning algorithm to be trained. The machine learning algorithm could be, for example, a BiGRU-CRF with BiLSTM character-based word representations, in one modality. The machine learning algorithm can Q ! Aᑷη / Zζηζ / E / YᥙΛΙ may include one or more layers for character-based word representation. For character-based word embeddings, the algorithm may include a BiLSTM 400 layer, as shown in Figure 4a. The BiLSTM layer can extract features from each word using a character-level model. The BiLSTM layer may be a character-to-word layer. The character-to-word layer may accept an unstructured query as input and may output one or more character-to-word representations of the words present in that query. As shown in Figure 4a, the character-to-word layer may, as an intermediate step, compute the embeddings of the query characters. Alternatively, the character-to-word layer may receive character embeddings as input. The machine learning algorithm may also include one or more layers for word-to-label representation. For example, as shown in Figure 4b, the algorithm may include a BiGRU word-to-label 450 layer. The word-to-label layer may receive one or more words (for example, one or more words forming a query, as a result of a character-to-word layer) and may produce one or more named entity identifiers. Such identifiers may include, for example, the beginning of a named entity type, the interior of a named entity type, and a word that QI An / Zzez / E / YILI is outside of any named entity. For example, as shown in Figure 4b, the words-to-tags layer can output an indication of a named entity type for each input word. In the example in Figure 4b, the word milwaukee is recognized as a start of the brand entity type (where B-BRD stands for Start Brand), economic is recognized as outside (O) of any named entity, and drill is recognized as a start of the product entity type (B-PRD). In some modalities, as shown in Figure 4b, the input to the word-to-label layer 450 may include the character-to-word representations produced by the character-to-word layer 400, concatenated with the respective word embeds of the query words. Consequently, the algorithm definition and the execution of the deployed model may include calculating the respective word embeds of a query. For example, in some modalities, Word2Vec or GloVe may be used to calculate the word embeds. The word-to-label layer may include a CRF sub-layer, which can improve the prediction of the most likely label sequence and can prevent invalid sequence transitions, such as B-BRD^I-PRD (Inner Product) and O-^I-BRD. Q! Αηη / Ζζηζ / Ε / ΥΙΛΙ In block 326, method 300 may include the definition of an initial training dataset from the first supplemented dataset 320. For example, a certain percentage of the data pairs from the first supplemented training dataset 320 may be selected. In one modality, fifteen percent of the first supplemented training dataset 320 is randomly selected as test data, and the remainder of the first supplemented training dataset 320 is randomly divided into training data (90% of the remaining data) and validation data. In block 328, method 300 can include preprocessing the initial training dataset. Preprocessing in block 328 can include, for example, balancing named entity values ​​in the initial training dataset, such as by leveraging domain-specific knowledge. For instance, named entity values ​​that could be one named entity type or another (for example, a brand or a product), such as cutter, Instant Pot, and anchor, can be identified. In block 328, these queries can be balanced by oversampling entity types with fewer queries to avoid introducing a trend. QI Αηη / Zζηζ / E / YΙΛΙ due to training data, in some modalities. Oversampling can involve creating new queries on the minority feature type to balance the distribution of that type in the training data for an unbalanced dataset. For example, for a feature value A that could represent either feature type Y or feature type Z, new queries can be created and added to the training dataset to ensure that feature type Y and feature type Z have an equal, or approximately equal, number of data points for feature A in the training data. Method 300 may also include, in block 330, training the machine learning algorithm according to the initial training dataset (e.g., the preprocessed initial training dataset). The algorithm may be trained in block 330 until the score of El on the validation dataset stops improving or until a predetermined number of epochs is reached. The method may also include, in block 332, the evaluation of the trained model to determine whether it has met any predetermined criteria, such as whether its F1 score has reached a predetermined threshold. As noted earlier, a portion of the initial supplemented training data may be used to evaluate the trained model. QI Αηη / Ζζηζ / Ε / ΥΙΛΙ If the trained model has not yet met the predetermined criteria, in block 334, method 300 can include additional stratified sampling (for example, by named feature type pattern as described above) of the first test dataset 314. In block 336, the trained model can be tested on the data pairs sampled in block 334. If, in block 336, the model's prediction on a query matches the named features in the data pair, the data pair is added to the training dataset 326 for the next training iteration. In block 338, method 300 can include additional stratified sampling of the second test dataset 324. The data pairs sampled in block 338 can also be added to the training dataset 326 for the next training iteration. Phase III: Model Deployment. As noted earlier, the model training phase 304 may include the training and evaluation of multiple model types, in some modalities. Consequently, in block 342, the highest-performing model can be selected for deployment, and in block 344, the final trained model can be packaged for deployment. For example, in a In the QI Aηη / Zζηζ / E / YΙΛI modality, the trained model can be packaged as a TensorFlow protocol buffer file with the required data (model weights, vocabulary, word embedding, etc.). In some modalities, variables from a TensorFlow checkpoint can be converted into constants stored directly in the model graph, eliminating unreachable parts of the graph, folding constants, folding batch normalizations, removing training and debugging operations, etc. In block 346, method 300 can include deploying the final package of the trained model to a server, such as a cloud virtual machine.In block 348, additional components associated with the trained model can be deployed to enable its use in connection with a real-time search engine, such as components for parsing a raw user search query and for converting a model prediction into a machine-readable format for use in the search engine's output. Blocks 350 and 352 illustrate an example of an input search query and an example of output from a real-time service that uses the trained model, respectively. Figure 4 is a 400 chart illustrating example results (in the form of Fl scores) of three types of named entity recognition approaches. QI Αηη / Ζζηζ / Ε / ΥΙΛΙ The first approach, shown by the dotted line 402, is a greedy recognition approach. The second approach, shown by the dashed line 404, is an approach consistent with the present description, but using all the training data in a single training session, instead of an iterative training process in which more training data is added at each iteration, as described above with respect to Figure 3. Finally, the third approach, shown by the diamond dashed line 406, is an approach consistent with the present description with iterative training. As shown in Figure 400, the advantage of an iterative training process is evident in three aspects. First, the F1 score improves from 87.1 in iteration 1 to 93.3 in iteration 7, demonstrating that this process can iteratively improve the model. Second, compared to simply training the model once it has used all the data, this iterative process achieves better results with each iteration. Finally, compared to the greedy approach, an iteratively trained model increases the F1 score from 69.5 to 93.3. The recognition of named entities according to the present description offers many advantages over known approaches. First, instead of Instead of polishing a single dataset to meet all the requirements for training the machine learning algorithm, separate datasets can be prepared to meet different requirements. This separation of datasets leads to simpler implementation and maintenance because each individual dataset can be prepared and improved independently and modularly. For example, if the set of possible named feature values ​​increases, more synthetic data can be added, and the model can be retrained from scratch to recognize the new feature values. Secondly, instead of training the model in a single pass, iterative model training, with more training data added in each iteration, allows the model to learn more patterns of named feature types and cover more feature values ​​per iteration. Figure 5 is a diagrammatic view of an example user computing environment that includes a general-purpose computing system environment, such as a desktop computer, laptop, smartphone, tablet, or any other such device capable of executing instructions, such as those stored on non-transient, computer-readable media. Furthermore, although it is described and illustrated in the context of a single system QI Aηη / Zζηζ / E / YΙΛΙ Computing 600, experts in the field will also appreciate that the various tasks described below can be practiced in a distributed environment having multiple Computing 600 systems linked through a local or wide area network in which executable instructions can be associated with and / or executed by one or more of the multiple Computing 600 systems. In its most basic configuration, the 600 computer system environment typically includes at least one 602 processing unit and at least one 604 memory, which may be connected via a 606 collector. Depending on the exact configuration and type of computer system environment, the 604 memory may be volatile (such as 610 RAM), non-volatile (such as 608 ROM, flash memory, etc.), or some combination of both. The 600 computer system environment may have additional features and / or functionalities. For example, the 600 computer system environment may also include additional storage (removable and / or non-removable) including, but not limited to, magnetic or optical disks, tape drives, and / or flash drives.These additional memory devices can be made accessible to the computer system environment 600 by means of, for example, a hard disk drive interface 612, a magnetic disk drive interface 614, and / or an optical disk drive interface 616. As will be understood,. These devices, which would be linked to the system collector 606, respectively, allow reading from and writing to a hard disk 618, reading from or writing to a removable magnetic disk 620, and / or reading from or writing to a removable optical disk 622, such as a CD / DVD ROM or other optical media. The drive interfaces and their associated computer-readable media allow the non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computer system environment 600. Those skilled in the art will further appreciate that other types of computer-readable media capable of storing data can be used for this same purpose.Examples of such media devices include, but are not limited to, magnetic cassettes, flash memory cards, digital videodiscs, Bernoulli cartridges, random access memories, nano drives, memory sticks, other read / write and / or read-only memories, and / or any other method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Any of these computing storage media may form part of the 600 computing system environment. A number of program modules can be stored on one or more of the devices QI Αηη / Ζζηζ / Ε / ΥΙΛΙ memory / media. For example, a basic input / output system (BIOS) 624, which contains the basic routines that help transfer information between elements within the computing system environment 600, such as during startup, can be stored in ROM 608. Similarly, RAM 610, the hard disk 618, and / or peripheral memory devices can be used to store computer-executable instructions comprising an operating system 626, one or more application programs 628 (which may include the search query processing system functionality 106 of Figure 1, for example), other program modules 630, and / or program data 622. In addition, computer-executable instructions can be downloaded to the computing environment 600 as needed, for example, through a network connection. An end user can input commands and information into the computer system environment 600 through input devices such as a keyboard 634 and / or a pointing device 636. Although not illustrated, other input devices may include a microphone, joystick, gamepad, scanner, etc. These and other input devices would typically be connected to the processing unit 602 via a peripheral interface 638, which, in turn, would be coupled to the collector. QI Αηη / Ζζηζ / Ε / ΥΙΛΙ 606. Input devices can be connected directly or indirectly to the 602 processor via interfaces such as a parallel port, a game port, FireWire, or a Universal Serial Collector (USB). To view information about the 600 computer system environment, a 640 monitor or other display device can also be connected to the 606 collector via an interface, such as the 632 video adapter. In addition to the 640 monitor, the 600 computer system environment can also include other output peripherals, not shown, such as speakers and printers. The computing system environment 600 can also use logical connections with one or more other computing system environments. Communication between the computing system environment 600 and the remote computing system environment can be exchanged through an additional processing device, such as a network router 652, which is responsible for network routing. Communication with the network router 652 can be carried out through a network interface component 654. Therefore, within such a network environment—for example, the Internet, the World Wide Web, a LAN, or another similar type of wired or wireless network—the program modules represented in relation to the system environment will be appreciated. Computing system 600 QI Aế / Zế / E / Yếế, or portions thereof, may be stored in the memory storage device(s) of the computing system 600 environment The computing system environment 600 may also include location hardware 686 to determine the location of the computing system environment 600. In some modes, the location hardware 656 may include, for example, only a GPS antenna, a REID chip or reader, a WiFi antenna, or other computing hardware that can be used to capture or transmit signals that can be used to determine the location of the computing system environment 600. The computing environment 600, or parts thereof, may comprise one or more components of system 100 of Figure 1, in the following modalities. Although this description has described certain embodiments, it is understood that the claims are not intended to be limited to these embodiments, except as explicitly stated in the claims. On the contrary, the present description is intended to encompass alternatives, modifications, and equivalents, which may be included within the spirit and scope of the description. Furthermore, in the detailed description of this description, numerous specific details are set forth in order to provide a complete understanding of the embodiments described. However, it will be obvious to a person skilled in the art that systems and methods consistent with this description can be practiced without these specific details. In other cases, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure various aspects of the present description. Some portions of the detailed descriptions in this text have been presented in terms of procedures, logic blocks, processing, and other symbolic representations of operations on data bits within a computer or digital system memory. These descriptions and representations are the means used by experts in the arts of data processing to more effectively convey the substance of their work to other experts in the field. A procedure, logic block, process, etc., is conceived here, and generally, as a self-consistent sequence of steps or instructions leading to a desired result. The steps are those that require physical manipulations of physical quantities.Typically, though not necessarily, these physical manipulations take the form of electrical or magnetic data that can be stored, transferred, combined, compared, and otherwise manipulated in a computer system or similar electronic device. For reasons. QI Αηη / Zζηζ / E / YΙΛΙ for convenience, and with reference to common usage, such data are called bits, values, elements, symbols, characters, terms, numbers, or the like, with reference to various modalities currently described. It should be noted, however, that these terms should be interpreted as referring to physical manipulations and quantities and are merely convenient labels that should be interpreted in light of the terms commonly used in the art. Unless specifically stated otherwise, as is clear from the discussion herein, it is understood throughout the discussions in this modality that any discussion using terms such as determine, issue, transmit, record, locate, store, display, receive, acknowledge, use, generate, provide, access, verify, notify, deliver, or similar, refers to the action and processes of a computing system, or similar electronic computing device, that manipulates and transforms data.Data is represented as physical (electronic) quantities within the registers and memories of the computer system and is transformed into other data represented similarly as physical quantities within the memories or registers of the computer system, or in other storage, transmission or display devices of information, as described in the. Q ! Αηη / Zζηζ / E / YΙΛΙ present document or is otherwise understood to an expert in the field. It is hereby stated that, as of this date, the best method known to the applicant for putting the aforementioned invention into practice is the one that is clear from the present description of the invention.

Claims

Having described the invention as above, the following claims are claimed as property:

1. A method for responding to a search query, characterized in that it comprises: defining a machine learning algorithm, the algorithm configured to receive a user query as input and to output zero or more named entities of one or more entity types in the user query, each of the one or more entity types comprising a plurality of respective values; defining a first dataset according to the user's behavior data, the first dataset comprising a plurality of first data pairs, each data pair comprising a user search query and one or more named entity values ​​defined in the user search query;define a second dataset by creating a plurality of second artificial data pairs, each second data pair comprising an artificial search query and one or more named entity values ​​defined in the artificial search query; train the machine learning algorithm to create a trained model by: (i) defining an initial training dataset comprising a portion of the first dataset; (ii) training the algorithm on the training dataset; (iii) adding more data from the first dataset and the second dataset to the training dataset to create another training dataset; (iv) training the algorithm on the new training dataset; and (v) iteratively repeating (iii) and (iv); receive a search query from the user;recognize one or more named entity values ​​in the search query with the trained model; and issue a response to the user according to one or more recognized named entity values.

2. The method according to claim 1, characterized in that the definition of the initial training dataset comprises: determining a plurality of named entity values ​​in the user's search queries of the first dataset that are not in the associated defined named entity values ​​of the first dataset and the portion of the second dataset; and adding the determined plurality of named entity values ​​to the defined named entity values ​​of the first dataset.

3. - The method according to claim 1, characterized in that the output of a response to the user in accordance with one or more recognized named entity values ​​comprises: directing the user to a web page dedicated to one or more of the recognized named entity values; filtering a set of search results in accordance with one or more recognized named entity values ​​and outputting the filtered set of search results to the user; or displaying to the user one or more recognized named entity values.

4. The method according to claim 1, characterized in that the iterative repetition of (i) and (ii) comprises iteratively repeating (iii) and (iv) until the accuracy of the trained model exceeds a predetermined threshold or until a predetermined number of iterations have been performed.

5. The method according to claim 1, characterized in that the addition of further data from the first dataset and the second dataset to the training dataset comprises adding data pairs for each of a plurality of entity type sequences.

6. The method according to claim 1, characterized in that it further comprises evaluating the trained model on a portion of the first dataset that is not in the initial training data.

7. The method according to claim 1, characterized in that the machine learning algorithm comprises: a character-to-word layer; and a word-to-label layer.

8. The method according to claim 7, characterized in that: the character-to-word layer comprises a bidirectional short-term memory network; and the word-to-label layer comprises a bidirectional recurrent unit network.

9. A system for responding to a search query, characterized in that it comprises: a non-transient, computer-readable memory that stores instructions; and a processor configured to execute the instructions to carry out a method comprising: defining a machine learning algorithm, the 1 QIAn algorithm configured to receive a user query as input and output zero or more named entities of one or more entity types in the user query, each of the one or more entity types comprising a plurality of respective values; defining a first dataset based on user behavior data, the first dataset comprising a plurality of first data pairs, each data pair comprising a user search query and one or more named entity values ​​defined in the user search query; defining a second dataset by creating a plurality of second artificial data pairs, each second data pair comprising an artificial search query and one or more named entity values ​​defined in the artificial search query; training the machine learning algorithm to create a trained model by: (i) defining a datasetinitial training comprising a portion of the first dataset; (ii) training the algorithm on the training dataset; (iii) adding more data from the first dataset and the second dataset to the training dataset to create another training dataset; (iv) training the algorithm on the new training dataset; and (v) iteratively repeating (iii) and (iv); receiving a search query from the user; recognizing one or more named entity values ​​in the search query with the trained model; and issuing a response to the user according to one or more recognized named entity values.

10. The system according to claim 9, characterized in that the definition of the initial training dataset comprises: determining a plurality of named entity values ​​in the search queries of theuser of the first dataset that are not in the associated defined named entity values ​​of the first dataset and the portion of the second dataset; and adding the determined plurality of named entity values ​​to the defined named entity values ​​of the first dataset.

11. The system according to claim 9, characterized in that the output of a response to the user according to one or more recognized named entity values ​​comprises: directing the user to a web page dedicated to one or more of the recognized named entity values; filtering a set of search results according to one or more recognized named entity values ​​and outputting the filtered search results set to the user; or displaying to the user one or more recognized named entity values.

12. The system according to claim 9, characterized in that the repetitionThe iterative process of (i) and (ii) comprises iteratively repeating (iii) and (iv) until the accuracy of the trained model exceeds a predetermined threshold or until a predetermined number of iterations have been performed.

13. The system according to claim 9, characterized in that the addition of more data from the first dataset and the second dataset to the training dataset comprises adding data pairs for each of a plurality of entity type sequences.

14. The system according to claim 9, characterized in that it further comprises evaluating the trained model on a portion of the first dataset that is not in the initial training data.

15. The system according to claim 9, characterized in that the machine learning algorithm comprises: a character-to-word layer; and a word-to-label layer.

16. The system ofin accordance with claim 15, characterized in that: the character-to-word layer comprises a bidirectional short-term memory network; and the word-to-label layer comprises a bidirectional network of recurrent units.

17. A method for training a machine learning algorithm to create a trained model, the trained model recognizing one or more named entities in a user search query, the one or more named entities having one or more entity types, characterized in that it comprises: defining a first dataset according to user behavior data, the first dataset comprising a plurality of first data pairs, each first data pair comprising a user search query and one or more named entity values ​​defined in the user search query; determining a plurality of named entity values ​​in the user search queries of the firsta set of data that are not in the associated defined named entity values ​​of the first data set; adding the specified plurality of named entity values ​​to the defined named entity values ​​of the first data set to create a first supplemented data set; defining a second data set by creating a plurality of second artificial data pairs, each second data pair comprising an artificial search query and one or more named entity values ​​defined in the artificial search query; training the machine learning algorithm to create a trained model by: (i) defining an initial training data set comprising a portion of the first supplemented training data set; (ii) training the algorithm against the first supplemented training data set; (iii) adding more data from the first data set ofsupplemented training and from the second dataset to the initial training dataset to create a new training dataset; (iv) training the algorithm on the new training dataset; and (v) iteratively repeating (iii) and (iv) until the accuracy of the trained model exceeds a predetermined threshold or until a predetermined number of iterations have been performed.

18. The method according to claim 17, characterized in that adding more data from the first dataset and the second dataset to the training dataset comprises adding data pairs for each of a plurality of entity type sequences.

19. The method according to claim 17, characterized in that it further comprises evaluating the trained model on a portion of the first dataset that is not in the initial training data.

20. Themethod according to claim 17, characterized in that the model is for recognizing one or more named entities selected from a predetermined set of named entity values.

Images