Commodity retrieval method and device based on AI and ES

Abstract

The invention provides a commodity retrieval method and device based on AI and ES, and the method comprises the steps: carrying out the classification of keywords through employing naive Bayes, and obtaining a target result; constructing an ES end query condition query object, and setting an item id as a query condition; transmitting the keywords into a structured perceptron, returning the identified fields and labels, and setting the identified fields and labels into a query object; the query object is sent to an ES end, the ES end uses a corresponding word segmentation device to perform word segmentation according to the field, and a corresponding word is obtained from a field word bank; the corresponding document is found from the inverted index according to the word, retrieval is completed, the accuracy of commodity search is greatly improved while the search performance is guaranteed, and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of data query technology, and in particular to a commodity retrieval method and device based on AI and ES. Background Art

[0002] Currently, the most popular way to search for products in the industry is to import product data into ES (Elasticsearch, a search engine), create a corresponding inverted index, and use wildcard, term, match, match_phrase and other methods (the most commonly used methods in ES) to search on different fields.

[0003] 1. Fuzzy matching: use wildcards (*, asterisk) for keywords and then use wildcard search to match the product name, item, brand, model, supplier and other fields. This will help you search for products that contain the complete keywords.

[0004] 2. Word segmentation search, using the match or match_phrase method to search, relies on the underlying inverted index of ES; when searching, first segment the keywords, and then search the inverted index for the data document corresponding to the matched words.

[0005] 3. Exact match: Use term to completely match the field content. This is suitable for situations where the search field is known and completely matched.

[0006] 4. Weighted search: In keyword searches, some fields, such as supplier name and place of origin, have a very low actual search share, while items and brands have a very high actual search share. By statistically analyzing search logs, we can increase the weights of fields with a higher actual search share, and reduce the weights of fields with a lower actual share.

[0007] The above method has the following defects:

[0008] 1. The defects of fuzzy matching are: it can only search for products with complete keywords, and it will not work if one character is missing; for example, "intel i5" cannot search for "inteli5", and the fault tolerance is low; the performance is poor, and it is not suitable for searching massive data.

[0009] 2. The defects of word segmentation search are: the quality of word segmentation search results is highly dependent on the quality of the word library and the quality of data segmentation. By default, Chinese word segmentation is based on characters. "Portable computer" will be divided into five characters: "convenient", "portable", "style", "calculation", "calculation", and "machine". This cannot match the complete word, and even the matching results of the words after entering them in a shuffled order are consistent, which is completely inaccurate. The industry's practice is to introduce Chinese word segmenters, such as the IK word segmenter. The above words will be divided into two words: "portable" and "computer", which can improve a certain degree of accuracy. You can also use a custom word library to add a custom word "portable computer", which will be divided into complete words, further improving the accuracy of word segmentation; but the general solution cannot cope with all situations. For example, in the word library, there is a brand "apple" and a fruit item "apple". At this time, it is impossible to distinguish whether the user wants to search for the brand or the item.

[0010] In the commodity business, the omission of commodities with average relevance has a greater negative impact on user experience than the search for commodities with low relevance. Therefore, the industry prefers to sacrifice some precision to improve recall. Generally speaking, in order to improve recall, it is necessary to increase the number of segmentation items as much as possible. For example, "portable computer" is not only divided into "portable computer", but also "portable" and "computer" to ensure that users can search for it by only entering "computer". However, adding segmentation items will also have a negative impact. ES will score the search results, and the score is the basis for sorting. If a word appears more frequently in a piece of data, then its score will be higher and it will be ranked higher. Then suppose the merchant discovers this mechanism and deliberately repeats important words in the product name, such as "portable computer portable computer portable computer", so that his product will naturally be ranked in front, so a reasonable scoring mechanism is also needed.

[0011] Sometimes users want to enter multiple pieces of information to search accurately, for example, to search for "portable computer core", "portable computer" is the item, and "core" is the CPU brand. The default match can search for products containing "portable computer" or "core". Although the operator can be modified to AND to search for products containing "portable computer" and "core", there is no basis for modification. Therefore, match_phrase is generally used, which is a phrase match. In addition to matching the participle items themselves with the product data, the participle items must also maintain the same order, and the maximum interval between participles can be set through slop. If the participle results are "portable computer" and "core", when slop is 0, it can match the product data containing "portable computer core"; when slop is 3, it can improve compatibility to a certain extent, and can also match additional data such as "portable computer i5 core". However, in the above example, "portable computer" is the item, "Core" is the CPU brand, and "i5" is the CPU series. They are distributed in different fields. Of course, they can also be combined into a large field and match_phrase can be used on the large field, but the value of slop cannot be completely determined, so this method has certain limitations.

[0012] 3. The defect of exact match is that the exact match term can only search for data that is exactly the same as the value, which is suitable for scenarios with clear fields and content.

[0013] 4. The defect of weighted search is: no matter using fuzzy matching, word segmentation search or exact matching, when we don’t know which field the user wants to search on, we search by weight after considering its possibility. For example, on the homepage of the mall, according to industry experience, the keywords entered by the user are most likely to be items, brands and models, with a moderate possibility of being a certain parameter of the product, and a lower possibility of being the supplier name, place of origin, etc. Therefore, the weight value can be increased in the item, brand, and model fields, and the weight value can be reduced in the supplier name and place of origin. This is a weight that is hard-coded according to experience and cannot cover all situations. Summary of the invention

[0014] The technical problem to be solved by the present invention is to provide a product retrieval method and device based on AI and ES, which greatly improves the accuracy of product search and improves user experience while ensuring search performance.

[0015] In a first aspect, the present invention provides a commodity retrieval method based on AI and ES, comprising:

[0016] Step 1: Use Naive Bayes to classify keywords and obtain item results;

[0017] Step 2: Build the query object on the ES side and set the item ID as the query condition;

[0018] Step 3: Pass the keywords into the structured perceptron, return the identified fields and labels, and set them in the query object;

[0019] Step 4: Send the query object to the ES end. The ES end uses the corresponding word segmenter to segment words according to the field and obtains the corresponding words from the domain vocabulary.

[0020] Step 5: Find the corresponding document from the inverted index according to the term to complete the retrieval.

[0021] In a second aspect, the present invention provides a commodity search device based on AI and ES, comprising:

[0022] The item result module classifies keywords using naive Bayes to obtain item results;

[0023] Set the object module, build the ES query condition query object, and set the item id as the query condition;

[0024] Set up the object module, pass the keywords into the structured perceptron, return the identified fields and labels, and set them in the query object;

[0025] The word segmentation module sends the query object to the ES end. The ES end uses the corresponding word segmenter to segment words according to the field and obtains the corresponding words from the domain vocabulary.

[0026] The retrieval module finds the corresponding document from the inverted index according to the term to complete the retrieval.

[0027] One or more technical solutions provided in the embodiments of the present invention have at least the following technical effects or advantages:

[0028] The present invention provides a complete and feasible technical solution for improving the accuracy of product search. It operates from five aspects: domain vocabulary construction, word segmentation method expansion, scoring model modification, product data expansion and intent recognition. While ensuring search performance, it greatly improves the accuracy of product search and improves user experience.

[0029] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0031] Figure 1 It is a timing diagram showing the process of the present invention;

[0032] Figure 2 It is a schematic diagram of the process of the present invention;

[0033] Figure 3 is a schematic diagram of a structured perceptron of the present invention;

[0034] Figure 4 is a flow chart of the method in Embodiment 1 of the present invention;

[0035] Figure 5 It is a schematic diagram of the structure of the device in the second embodiment of the present invention. DETAILED DESCRIPTION

[0036] The embodiment of the present application solves the technical problem of inaccurate product search in the prior art by providing a product retrieval method and device based on AI and ES. While ensuring the search performance, the present application greatly improves the accuracy of product search and improves the user experience.

[0037] The technical solution in the embodiment of the present application has the following general idea:

[0038] like Figure 1 As shown in the figure, the user enters the keyword search, and after the request reaches the App Server (application service layer), the naive Bayes is used to classify the keywords and obtain the item results (i.e., goodsClassGuid); the ES query condition query object is constructed, and the goodsClassGuid (item result) is set as the query condition; the keyword is then passed into the structured perceptron, and the fields and labels it recognizes are returned and also set in the query object; for example, for "Lenovo desktop computer Core", the label of "Lenovo" is recognized as "brand", the label of "desktop computer" is "item", and the label of "Core" is "parameter value"; these labels and values ​​are then set as query conditions in the query object.

[0039] The query object is sent to the ES side. The ES side uses the corresponding word segmenter on the field to segment the query field and obtains the corresponding words from the domain vocabulary. After the segmentation is obtained, the corresponding document is found from the inverted index and then returned to the App Server. The App Server parses the data, packages it according to the business scenario, and returns it to the user, completing the entire search process.

[0040] like Figure 2As shown, the specific implementation steps of the present invention are to take optimization measures from multiple aspects such as domain word library construction, word segmentation method expansion, scoring model modification, product data expansion, and intention recognition. For example, the accuracy of word segmentation is improved by building a domain word library; the product category is predicted using machine learning classification technology; and the information such as category, brand, and parameter in the keyword is accurately extracted using NER (Named Entity Recognition). In this way, the accuracy of product search is improved.

[0041] 1. Construction of vocabulary database in the field of government procurement.

[0042] (1) Collecting domain data. In the field of government procurement, we collect the categories, brands, parameter names, and parameter values ​​of commodities; at the same time, we divide them into different business categories, such as manufacturer commodities, e-commerce commodities, centralized procurement, decentralized procurement, etc.

[0043] (2) Data cleaning and review: The collected data is cleaned of extra spaces, special symbols, error correction, and deduplication, and finally manually reviewed and included in the domain vocabulary.

[0044] (3) Different lexicons are formed according to different dimensions, such as centralized procurement items and decentralized items. Lexicons are formed by item, brand, parameter name and parameter value. When conditions permit, synonyms are expanded for words in the same field.

[0045] 2. Expansion of word segmentation methods.

[0046] Create multiple field types and use different tokenizers for each field. For example, add the following to the goodsName (product name field):

[0047] (1) goodsName.ik uses the ik tokenizer;

[0048] When using the ik word segmenter, you need to add the domain word library in 1 to the general word library. In this way, when users search, they can correctly separate the domain words and match them with the product data.

[0049] (2) goodsName.pinyin uses the pinyin word segmenter;

[0050] The pinyin word segmenter is mainly used when users input pinyin. After using the pinyin word segmenter, the product data will generate pinyin.

[0051] (3) goodsName.2_10_gram uses the n-gram tokenizer, with a minimum word length of 2 and a maximum word length of 10. The smaller the word length, the worse the precise matching effect, and the longer the word length, the better the precise matching effect. However, the word spacing will affect the number of tokens, which in turn affects the storage space and query performance. Therefore, the length of the domain data and business fields should be counted before setting. The use of n-gram is mainly used to solve the tokenization of mixed Chinese, English and digital words. For example, the product model "R361K" will be used as a token item to solve the problem that other tokenizers cannot correctly segment the word.

[0052] The codes for the three fields extended for the product name are as follows:

[0053] PUT goods_index

[0054] {"settings":{"analysis":{"analyzer":{"2_10_gram":{"tokenizer":"2_10_gra m"}},"tokenizer":{"2_10_gram":{"type":"ngram","min_gram":2,"max_gram":10,"token_chars":["letter","digit"]}}}},"ma ppings":{"properties":{"goodsName":{"type":"keyword","fields":{"ik":{"type":"text","analyzer":"ik_max_word","sear ch_analyzer":"ik_smart"},"pinyin":{"type":"text","analyzer":"pinyin","search_a nalyzer":"pinyin"},"2_10_gram":{"type":"text","analyzer":"2_10_gram","search_analyzer":"2_10_gram"}}}}}}

[0055] 3. Modification of scoring model.

[0056] (1) The BM25 scoring model is used to represent the score of ES documents. The model includes two free parameters k1 and b. The scoring model can be modified by adjusting the values ​​of k1 and b.

[0057] In the word segmentation search, removing the extra points brought by the word frequency does not affect the search accuracy, so k1 is set to 0, b is adjusted to 0.75, and the code for adjusting the product name similarity is as follows:

[0058] PUT goods_index / _settings

[0059] {"settings":{"similarity":{"BM25_0TF":{"type":"BM25","b":"0.75","k1":"0"}}},"mappings":{"properties":{"goodsName":{"type": "keyword","fields":{"ik":{"type":"text","analyzer":"ik_max_word","search_analyzer":"ik_smart","similarity":"BM25_0TF"}}}}}}

[0060] (2) Use function_score or script_score special rule scoring formula

[0061] For different business scenarios, you can adjust the document score through function_score or script_score to give priority to more matching products. For example, you can build a scoring function based on the product listing date, evaluation score, and purchase status of similar users:

[0062]

[0063]

[0064] 4. Product data expansion.

[0065] (1) Obtain the same or similar data from websites that disclose commodity item data (e.g. the centralized procurement catalog published annually on the China Government Procurement Network), such as colloquial terms for “portable computer”, aliases such as “notebook” and “laptop”, and expand them into the item field to reduce the number of cases where products are not recalled.

[0066] 5. Intent recognition.

[0067] (1) Item Forecast

[0068] In the process of implementing item prediction, we cleverly combined TF-IDF (term frequency-inverse document frequency) technology and naive Bayes classifier. First, let's analyze the essence of these two technologies in depth:

[0069] TF-IDF Technology

[0070] TF (Term Frequency) Word Frequency: What is calculated in this step is the frequency of a certain word appearing in a document. It reflects the importance of the word to the content of the document, but relying solely on word frequency may not be sufficient to accurately evaluate the weight of a word in a document collection. Because some common words (such as "的" (de), "是" (shi), etc.) will appear frequently in all documents, but their role in distinguishing document categories is very small.

[0071] IDF (Inverse Document Frequency) Inverse Document Frequency: IDF is used to make up for this deficiency of TF. It evaluates the general importance of a word. Specifically, IDF is the logarithm of the total number of documents divided by the logarithm of the number of documents containing the word, and then the reciprocal of the obtained quotient is taken. If a word appears in a large number of documents, then its IDF value will be very low; on the contrary, if it only appears in a small number of documents, then its IDF value will be very high.

[0072] TF-IDF Value: The TF-IDF value is the product of TF and IDF. It aims to evaluate the importance of a word for a document collection or a single document in a corpus. Through TF-IDF, we can convert text data into numerical vectors, which take into account both the frequency of the word appearing in the document and the uniqueness of the word in the entire document collection.

[0073] TF-IDF:

[0074] Formula: TF-IDF = TF * IDF

[0075] A high word frequency within a specific document, along with a low document frequency of that word in the entire document collection, can produce a high-weight TF-IDF. Therefore, TF-IDF tends to filter out common words and retain important words.

[0076] Naive Bayes Classifier

[0077] The naive Bayes classifier is a simple probability classifier based on Bayes' theorem, which assumes that features are independent of each other. In the context of text classification, the naive Bayes classifier uses TF-IDF vectors as input features and predicts the category of new documents by learning the TF-IDF feature distributions of different category documents in the training dataset.

[0078] Specifically, the naive Bayes classifier will calculate the probability that a given document belongs to each category and select the category with the highest probability as the prediction result. This calculation process involves the estimation of conditional probability and prior probability, as well as the use of Bayes' theorem for probability update.

[0079] The mathematical representation of Bayesian theory can be written as follows:

[0080]

[0081] Here we assume that A and B are both product descriptions and P(B) is not 0.

[0082] Basically, as long as we have the item obtained given product description B, we can calculate the probability of product description A occurring. Product description B is also called evidence.

[0083] P(A) is the prior (prior probability, i.e., probability of occurrence before evidence) of product description A. Evidence is an attribute value of an unknown product description (in this case, product description B).

[0084] P(A|B) is the posterior probability of item description B, i.e. the probability of it occurring after the evidence.

[0085] Bayesian theory can be used like this:

[0086]

[0087] Here y is the class variable and X is the dependent feature vector (of size n): X = (x1, x2, x3, …, x n )

[0088] To make it clearer, the feature vector and related class variables based on the product search scenario are: X = (notebook, thin and light, computer, memory, ...) y = portable computer;

[0089] So in this scenario, if the user's search term is: "I want to buy a laptop computer, it needs 32G of memory and it needs to be thin and light", then P(X|y) means the probability of getting the item "portable computer" given the known search terms: "notebook", "computer", "thin" and "memory".

[0090] In summary, we combine TF-IDF and Naive Bayes to build an efficient and accurate item prediction system

[0091] First, the TF-IDF value of the vocabulary of each item in the document is carefully calculated based on the product data. This process not only considers the frequency of occurrence of the vocabulary in the item document (TF), but also incorporates the general importance of the vocabulary in the entire item document collection (IDF), thereby constructing the TF-IDF vector representation of each item document. Subsequently, these carefully calculated TF-IDF vectors are used as key features to drive the training process of the naive Bayes classifier. During the training phase, the classifier deeply analyzes these features, learns and establishes the mapping relationship between TF-IDF vectors and different item categories. Finally, when faced with new product description text, we only need to convert it into the corresponding TF-IDF vector and input it into the trained naive Bayes classifier to efficiently and accurately predict the item category to which the text belongs.

[0092] (2) NER accurate recognition

[0093] In addition to product category prediction, it is particularly important to accurately identify the composition of keywords in order to further identify user search intent. Therefore, we use NER technology to identify the label of each word in the keyword. For example, in "Apple Notebook A5", the word "Apple" is identified as a brand, "Notebook" is identified as a product category, and "A5" is identified as a CPU model.

[0094] First, collect users’ search logs to obtain keyword data; establish four tags: item / nc, brand / nb, parameter name / np, parameter value / nv. After data cleaning, manually annotate and include it in the raw corpus for model training. For example, “Apple Notebook A5” is annotated as “Apple / nb Notebook / nc A5 / nv”.

[0095] In addition, product search is a scenario that requires particularly high response speed; it also needs to be continuously adjusted based on user keywords. Therefore, among the many machine learning algorithms and deep learning algorithms, we selected the structured perceptron with high accuracy and low computational complexity.

[0096] like Figure 3 As shown in Figure 1, the structured perceptron is a bionic neuron structure that can represent input using a function:

[0097] f(x)=w·x

[0098] Where x is the feature vector, w is the weight vector, and f(x) is the score. When the score reaches a certain threshold, t is output, indicating whether f(x) reaches the threshold. t is 0 or 1. If the word belongs to the set label, it is 1, and if it does not, it is 0.

[0099] The task of word recognition is mainly to combine the context and predict the label of the word segmentation, so we also need to consider the structure of the sequence and define the structural feature vector y. Then the actual scoring function is:

[0100] score1(x)=w·φ(x,y)

[0101] For a set of text sequences, different structural feature vectors y are considered, and the structure with the maximum score each time is taken as the prediction result to obtain the prediction function:

[0102]

[0103] Now that we have the prediction function, we can calculate:

[0104] Perform feature engineering on the labeled words to obtain the feature vector x (i) , and the word structure vector y is also obtained (i) , calculate the prediction

[0105] Compared with the annotation results, if Then update the parameters Where α is the learning rate and can be adjusted based on the results.

[0106] However, at this time, we can only learn the structure of words, which is equivalent to being able to segment words, but cannot determine word labels. Therefore, we need to extract features from labels and extract transfer features from continuous labels in the sequence:

[0107]

[0108] Among them, y t is the tth label of the sequence, s i is the i-th label of the annotation set, i=0,1…N; j=1,…,N is the size of the annotation set, and k=i*N+j is the transfer feature number. Define s0 as BOS, which represents the virtual label before the first element of the sequence. Define the state features of each word at different positions:

[0109]

[0110] A word may be a product in one context, but a brand in another context. So this feature is only related to the current state, not the previous state. You can calculate the score of a word being a certain label:

[0111]

[0112] By iteratively calculating the above maximum score value, the label of the word can be obtained.

[0113] Then, every time a user performs a keyword search, the model is used to separate the key words in the keyword and determine the labels of each word. The query statement is then rewritten to directly query the word in the corresponding field, thereby greatly improving the accuracy of the word search. Compared with other complex models, this method takes little calculation time and can fully meet the real-time call scenario.

[0114] Embodiment 1

[0115] like Figure 4 As shown, this embodiment provides a commodity retrieval method based on AI and ES, including:

[0116] Step 1: Use Naive Bayes to classify keywords and obtain item results;

[0117] Step 2: Build the query object on the ES side and set the item ID as the query condition;

[0118] Step 3: Pass the keywords into the structured perceptron, return the identified fields and labels, and set them in the query object;

[0119] Step 4: Send the query object to the ES end. The ES end uses the corresponding word segmenter to segment words according to the field and obtains the corresponding words from the domain vocabulary.

[0120] Step 5: Find the corresponding document from the inverted index according to the term to complete the retrieval.

[0121] In this embodiment, preferably, the step 1 is specifically as follows: the TF-IDF value of the vocabulary of each item of the commodity in the document is calculated according to the commodity data, the TF-IDF vector of the document corresponding to each item is constructed, and the TF-IDF vector is used as the key feature to drive the training of the naive Bayes classifier; in the training stage, the naive Bayes classifier learns and establishes the mapping relationship between the TF-IDF vector and different item categories according to the key feature, and completes the establishment of the naive Bayes classifier; the keywords are converted into corresponding TF-IDF vectors, and are input into the trained naive Bayes classifier to obtain the item results.

[0122] In this embodiment, preferably, the domain word library construction in step 4 includes: in the field of government procurement, collecting the items, brands, parameter names and parameter values ​​of commodities, and dividing them according to different businesses; removing redundant spaces, special symbols, error correction, and deduplication from the collected data, and finally manually reviewing and incorporating them into the domain word library; forming corresponding word libraries by item, brand, parameter name and parameter value respectively;

[0123] The word segmenters include: ik word segmenter, pinyin word segmenter and n-gram word segmenter; the corresponding word segmenter is used according to the field.

[0124] In this embodiment, preferably, the step 5 is specifically as follows: according to the term, a corresponding document is found from the inverted index, and a BM25 scoring model is used, wherein the calculation formula of the BM25 scoring model includes adjustable parameters k1 and b, k1 is set to 0, b is adjusted to 0.75, and the function_score or script_score on the ES side is used to adjust the BM25 document score, and more matching products are displayed preferentially to complete the retrieval.

[0125] In this embodiment, preferably, the step 3 specifically includes: using NER technology to identify the label of each word in the keyword,

[0126] First, search logs are collected to obtain keyword data; four tags are established: category, brand, parameter name, and parameter value. After data cleaning, they are annotated and included in the raw corpus for model training. The structured perceptron uses a function to represent the input: f(x) = w x, where x is the feature vector and w is the weight vector. Then f(x) is the score. When the score reaches the threshold, t is output.

[0127] Define the structural feature vector y, and the scoring function is:

[0128] score1(x)=w·φ(x,y)

[0129] For a set of text sequences, different structural feature vectors y are considered, and the structure with the maximum score is taken as the prediction result each time to obtain the prediction function:

[0130]

[0131] Perform feature engineering on the labeled words to obtain the feature vector x (i) , and we get the word structure vector y (i) , calculate the prediction Compared with the annotation results, if Then update the parameters Where α is the learning rate, which is adjusted according to the results;

[0132] Extract features from labels and extract transfer features from continuous labels in the sequence:

[0133]

[0134] Among them, y t is the tth label of the sequence, s iis the i-th label of the annotation set, i=0,1…N; j=1,…,N; N is the size of the annotation set, k=i*N+j is the transfer feature number; define s0 as BOS, which means the virtual label before the first element of the sequence; define the state features of each word at different positions:

[0135]

[0136] Calculate the score of a word being a certain tag:

[0137]

[0138] By iteratively calculating the above maximum score value, the label of the word can be obtained.

[0139] Based on the same inventive concept, the present application also provides a device corresponding to the method in Example 1, see Example 2 for details.

[0140] Embodiment 2

[0141] like Figure 5 As shown, in this embodiment, a commodity search device based on AI and ES is provided, including:

[0142] The item result module classifies keywords using naive Bayes to obtain item results;

[0143] Set the object module, build the ES query condition query object, and set the item id as the query condition;

[0144] Set up the object module, pass the keywords into the structured perceptron, return the identified fields and labels, and set them in the query object;

[0145] The word segmentation module sends the query object to the ES end. The ES end uses the corresponding word segmenter to segment words according to the field and obtains the corresponding words from the domain vocabulary.

[0146] The retrieval module finds the corresponding document from the inverted index according to the term to complete the retrieval.

[0147] In this embodiment, preferably, the item result module is specifically as follows: the TF-IDF value of the vocabulary of each item of the commodity in the document is calculated according to the commodity data, the TF-IDF vector of the document corresponding to each item is constructed, and the TF-IDF vector is used as the key feature to drive the training of the naive Bayes classifier; in the training stage, the naive Bayes classifier learns and establishes the mapping relationship between the TF-IDF vector and different item categories according to the key feature, and completes the establishment of the naive Bayes classifier; the keywords are converted into corresponding TF-IDF vectors, and are input into the trained naive Bayes classifier to obtain the item results.

[0148] In this embodiment, preferably, the domain word library construction in the word segmentation module includes: in the field of government procurement, collecting the items, brands, parameter names and parameter values ​​of commodities, and dividing them according to different businesses; removing redundant spaces, special symbols, error correction, and deduplication from the collected data, and finally incorporating them into the domain word library after manual review; forming corresponding word libraries respectively through items, brands, parameter names and parameter values;

[0149] The word segmenters include: ik word segmenter, pinyin word segmenter and n-gram word segmenter; the corresponding word segmenter is used according to the field.

[0150] In this embodiment, preferably, the retrieval module is specifically as follows: according to the term, the corresponding document is found from the inverted index, and the BM25 scoring model is used. The calculation formula of the BM25 scoring model includes adjustable parameters k1 and b, k1 is set to 0, b is adjusted to 0.75, and the function_score or script_score on the ES side is used to adjust the BM25 document score, and more matching products are displayed first to complete the retrieval.

[0151] In this embodiment, preferably, the object setting module specifically includes: using NER technology to identify the label of each word segment in the keyword,

[0152] First, search logs are collected to obtain keyword data; four tags are established: category, brand, parameter name, and parameter value. After data cleaning, they are annotated and included in the raw corpus for model training. The structured perceptron uses a function to represent the input: f(x) = w x, where x is the feature vector and w is the weight vector. Then f(x) is the score. When the score reaches the threshold, t is output.

[0153] Define the structural feature vector y, and the scoring function is:

[0154] score1(x)=w·φ(x,y)

[0155] For a set of text sequences, different structural feature vectors y are considered, and the structure with the maximum score is taken as the prediction result each time to obtain the prediction function:

[0156]

[0157] Perform feature engineering on the labeled words to obtain the feature vector x (i) , and the word structure vector y is also obtained (i) , calculate the prediction Compared with the annotation results, if Then update the parameters Where α is the learning rate, which is adjusted according to the results;

[0158] Extract features from labels and extract transfer features from continuous labels in the sequence:

[0159]

[0160] Among them, y t is the tth label of the sequence, s i is the i-th label of the annotation set, i=0,1…N; j=1,…,N; N is the size of the annotation set, k=i*N+j is the transfer feature number; define s0 as BOS, which means the virtual label before the first element of the sequence; define the state features of each word at different positions:

[0161]

[0162] Calculate the score of a word being a certain tag:

[0163]

[0164] By iteratively calculating the above maximum score value, the label of the word can be obtained.

[0165] Since the device introduced in the second embodiment of the present invention is a device used to implement the method of the first embodiment of the present invention, those skilled in the art can understand the specific structure and deformation of the device based on the method introduced in the first embodiment of the present invention, so it is not described here in detail. All devices used in the method of the first embodiment of the present invention belong to the scope of protection of the present invention.

[0166] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A commodity retrieval method based on AI and ES, characterized in that: include: Step 1: Use Naive Bayes to classify keywords and obtain item results; Step 2: Build the query object on the ES side and set the item ID as the query condition; Step 3: Pass the keywords into the structured perceptron, return the identified fields and labels, and set them in the query object; Step 4: Send the query object to the ES end. The ES end uses the corresponding word segmenter to segment words according to the field and obtains the corresponding words from the domain vocabulary. Step 5: Find the corresponding document from the inverted index according to the term to complete the retrieval.

2. The product retrieval method based on AI and ES according to claim 1, characterized in that: The step 1 is specifically as follows: the TF-IDF value of the vocabulary of each item of the product in the document is calculated according to the product data, the TF-IDF vector of the document corresponding to each item is constructed, and the TF-IDF vector is used as the key feature to drive the training of the naive Bayes classifier; in the training stage, the naive Bayes classifier learns and establishes the mapping relationship between the TF-IDF vector and different item categories according to the key feature, and completes the establishment of the naive Bayes classifier; the keywords are converted into corresponding TF-IDF vectors, and are input into the trained naive Bayes classifier to obtain the item results.

3. The product retrieval method based on AI and ES according to claim 1, characterized in that: The construction of the domain word library in step 4 includes: in the field of government procurement, collecting the items, brands, parameter names and parameter values ​​of commodities, and dividing them according to different businesses; removing redundant spaces, special symbols, error correction, and deduplication from the collected data, and finally manually reviewing and incorporating them into the domain word library; forming corresponding word libraries by item, brand, parameter name and parameter value respectively; The word segmenters include: ik word segmenter, pinyin word segmenter and n-gram word segmenter; the corresponding word segmenter is used according to the field.

4. The product retrieval method based on AI and ES according to claim 1, characterized in that: The step 5 is specifically as follows: according to the term, the corresponding document is found from the inverted index, and the BM25 scoring model is used. The calculation formula of the BM25 scoring model includes adjustable parameters k1 and b, k1 is set to 0, b is adjusted to 0.75, and the function_score or script_score on the ES side is used to adjust the BM25 document score, and more matching products are displayed first to complete the retrieval.

5. The product retrieval method based on AI and ES according to claim 1, characterized in that: The step 3 specifically includes: using NER technology to identify the label of each word in the keyword, First, search logs are collected to obtain keyword data; four tags are established: category, brand, parameter name, and parameter value. After data cleaning, they are annotated and included in the raw corpus for model training. The structured perceptron uses a function to represent the input: f(x) = w x, where x is the feature vector and w is the weight vector. Then f(x) is the score. When the score reaches the threshold, t is output. Define the structural feature vector y, and the scoring function is: score1(x)=w·φ(x,y) For a set of text sequences, different structural feature vectors y are considered, and the structure with the maximum score is taken as the prediction result each time to obtain the prediction function: Perform feature engineering on the labeled words to obtain the feature vector x (i) , and the word structure vector y is also obtained (i) , calculate the prediction Compared with the annotation results, if Then update the parameter w←w+ Where α is the learning rate, which is adjusted according to the results; Extract features from labels and extract transfer features from continuous labels in the sequence: Among them, y t is the tth label of the sequence, s i is the i-th label of the annotation set, i=0,1…N; j=1,…,N; N is the size of the annotation set, k=i*N+j is the transfer feature number; define s0 as BOS, which means the virtual label before the first element of the sequence; define the state features of each word at different positions: Calculate the score of a word being a certain tag: By iteratively calculating the above maximum score value, the label of the word can be obtained.

6. A commodity search device based on AI and ES, characterized in that: include: The item result module classifies keywords using naive Bayes to obtain item results; Set the object module, build the ES query condition query object, and set the item id as the query condition; Set up the object module, pass the keywords into the structured perceptron, return the identified fields and labels, and set them in the query object; The word segmentation module sends the query object to the ES end. The ES end uses the corresponding word segmenter to segment words according to the field and obtains the corresponding words from the domain vocabulary. The retrieval module finds the corresponding document from the inverted index according to the term to complete the retrieval.

7. The commodity search device based on AI and ES according to claim 6, characterized in that: The item result module is specifically as follows: the TF-IDF value of the vocabulary of each item of the commodity in the document is calculated according to the commodity data, the TF-IDF vector of the document corresponding to each item is constructed, and the TF-IDF vector is used as the key feature to drive the training of the naive Bayes classifier; in the training stage, the naive Bayes classifier learns and establishes the mapping relationship between the TF-IDF vector and different item categories according to the key feature to complete the establishment of the naive Bayes classifier; the keywords are converted into corresponding TF-IDF vectors and input into the trained naive Bayes classifier to obtain the item results.

8. The commodity search device based on AI and ES according to claim 6, characterized in that: The domain word library construction in the word segmentation module includes: in the field of government procurement, collecting the items, brands, parameter names and parameter values ​​of commodities, and dividing them according to different businesses; removing redundant spaces, special symbols, error correction, and deduplication from the collected data, and finally manually reviewing and incorporating them into the domain word library; forming corresponding word libraries through items, brands, parameter names and parameter values; The word segmenters include: ik word segmenter, pinyin word segmenter and n-gram word segmenter; the corresponding word segmenter is used according to the field.

9. The commodity search device based on AI and ES according to claim 6, characterized in that: The retrieval module specifically includes: finding the corresponding document from the inverted index according to the term, using the BM25 scoring model, wherein the calculation formula of the BM25 scoring model includes adjustable parameters k1 and b, k1 is set to 0, b is adjusted to 0.75, and the function_score or script_score on the ES side is used to adjust the BM25 document score, and more matching products are displayed first to complete the retrieval.

10. The commodity search device based on AI and ES according to claim 6, characterized in that: The object setting module specifically includes: using NER technology to identify the label of each word in the keyword, First, search logs are collected to obtain keyword data; four tags are established: category, brand, parameter name, and parameter value. After data cleaning, they are annotated and included in the raw corpus for model training. The structured perceptron uses a function to represent the input: f(x) = w x, where x is the feature vector and w is the weight vector. Then f(x) is the score. When the score reaches the threshold, t is output. Define the structural feature vector y, and the scoring function is: score1(x)=w·φ(x,y) For a set of text sequences, different structural feature vectors y are considered, and the structure with the maximum score is taken as the prediction result each time to obtain the prediction function: Perform feature engineering on the labeled words to obtain the feature vector x (i) , and the word structure vector y is also obtained (i) , calculate the prediction Compared with the annotation results, if Then update the parameters Where α is the learning rate, which is adjusted according to the results; Extract features from labels and extract transfer features from continuous labels in the sequence: Among them, y t is the tth label of the sequence, s i is the i-th label of the annotation set, i=0,1…N; j=1,…,N; N is the size of the annotation set, k=i*N+j is the transfer feature number; define s0 as BOS, which means the virtual label before the first element of the sequence; define the state features of each word at different positions: Calculate the score of a word being a certain tag: By iteratively calculating the above maximum score value, the label of the word can be obtained.

Images