Commodity search lexicon generation method and system

Abstract

The invention provides a commodity search lexicon generation method and system in the technical field of search engine word segmentation, and the method comprises the steps: S1, obtaining a large number of historical search statements, carrying out the preprocessing and marking of each historical search statement, and constructing a data set; s2, creating a word segmentation model, and setting a loss function of the word segmentation model; s3, training the word segmentation model through the data set; s4, presetting a plurality of initial search words, and creating a commodity search word library based on each initial search word; s5, acquiring an input real-time search statement, performing word segmentation operation on the real-time search statement through the word segmentation model to obtain a plurality of real-time search words, and storing each real-time search word into a commodity search word library; and S6, real-time search behaviors corresponding to the real-time search statements are collected in real time, and the commodity search word bank is dynamically updated based on the real-time search behaviors. The method has the advantage that the word segmentation accuracy of the search word is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of search engine word segmentation, and in particular to a method and system for generating a commodity search word library. Background Art

[0002] A search engine (search system) is a software that uses a specific computer program to collect information resources according to a certain strategy and returns the required information to the user based on the user's input. With the rapid development of Internet technology, search engines have become the core system of various Internet application fields. For example, e-commerce malls need to use search engines to return and display corresponding products based on the search statements entered by users.

[0003] In the e-commerce mall, in order to better match products based on search statements, it is necessary to perform word segmentation on the search statements to obtain the corresponding search terms, and then display the corresponding products based on the search terms. For example, if the search statement is "power system relay protection experimental device", it can be segmented into search terms including "power system", "relay protection", "experiment" and "device", and then match products based on each search term, calculate the priority order of each product display based on the weight, relevance and other indicators of each search term, and display the matching products in order based on the priority order.

[0004] In order to perform word segmentation operations quickly and accurately, a search word library that stores several search words is indispensable. However, traditional search word libraries only rely on manually presetting some common words as search words, and only mechanically traverse and match each search word with the search sentence to perform the word segmentation operation. It does not combine the search characteristics such as the user's search behavior, resulting in poor word segmentation results and frequent wrong search words. For example, wrong search words are generated by word segmentation of sentences that should not be segmented. The search word library is not dynamically updated, and needs to be manually updated regularly, which has a lag.

[0005] Therefore, how to provide a method and system for generating a commodity search word library to improve the accuracy of search word segmentation has become a technical problem that needs to be solved urgently. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method and system for generating a commodity search word library, so as to improve the accuracy of search word segmentation.

[0007] In a first aspect, the present invention provides a method for generating a commodity search word library, comprising the following steps:

[0008] Step S1, obtaining a large number of historical search statements, preprocessing and annotating each of the historical search statements to construct a data set;

[0009] Step S2: creating a word segmentation model and setting a loss function of the word segmentation model;

[0010] Step S3, training the word segmentation model using the data set;

[0011] Step S4: preset a number of initial search terms, and create a product search term library based on each of the initial search terms;

[0012] Step S5: obtaining an input real-time search statement, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a plurality of real-time search words, and storing each of the real-time search words in a commodity search word library;

[0013] Step S6: collecting the real-time search behavior corresponding to the real-time search statement in real time, and dynamically updating the commodity search word library based on the real-time search behavior.

[0014] Furthermore, the step S1 is specifically as follows:

[0015] Acquire a large number of historical search statements, and perform preprocessing on each of the historical search statements, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spelling errors, removing special characters, removing punctuation marks, removing blanks, removing stop words, and removing low-frequency words;

[0016] Then, each of the pre-processed historical search statements is annotated with at least the length of the word segmentation window, the word segmentation position, the number of word segments, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; the user characteristics at least include browsing behavior, clicking behavior, collecting behavior, following behavior, purchasing behavior, adding to cart behavior, and sharing behavior; the product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; the scene characteristics at least include industry scenes and exclusive scenes;

[0017] A data set is constructed based on the annotated historical search statements.

[0018] Furthermore, the step S2 is specifically as follows:

[0019] A word segmentation model is created based on an input module, a BERT module, a classifier and an output module, and the loss function of the word segmentation model is set to a cross entropy function; the input module is used to input historical search sentences, and performs layer normalization and random inactivation operations on the input historical search sentences; the BERT module is used to identify entities from historical search sentences; the classifier is used to classify the identified entities; the output module is used to predict the probability of search word segmentation based on the classified entities, and output the search word segmentation results.

[0020] Furthermore, the step S3 is specifically as follows:

[0021] The data set is divided into a training set, a validation set, and a test set based on a ratio of 8:1:1;

[0022] The word segmentation model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters of the word segmentation model including at least the number of iterations, the learning rate, and the training batch are continuously optimized during the training process;

[0023] The trained word segmentation model is verified by the verification set to determine whether the word segmentation accuracy of the word segmentation model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0024] The recall rate, precision rate and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate and F1-Score.

[0025] Furthermore, in step S4, the commodity search word library is used to store search words and search weights, search times and user portraits corresponding to the search words; the user portraits at least include gender, age, region corresponding to the search IP address, occupation, price preference and search duration; the search words are initial search words or real-time search words;

[0026] The step S5 is specifically as follows:

[0027] Obtaining an input real-time search statement through a visual interface, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a number of real-time search words, and determining whether the corresponding real-time search words already exist in the commodity search word library. If not, storing each of the real-time search words in the commodity search word library; if yes, ending the process;

[0028] The step S6 is specifically as follows:

[0029] During the execution of the real-time search statement, real-time search behaviors including at least user characteristics, product characteristics and scene characteristics are collected in real time, and the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behaviors.

[0030] In a second aspect, the present invention provides a system for generating a commodity search word library, comprising the following modules:

[0031] A data set construction module is used to obtain a large number of historical search statements, pre-process each of the historical search statements, and construct a data set after annotating;

[0032] A word segmentation model creation module, used to create a word segmentation model and set a loss function of the word segmentation model;

[0033] A word segmentation model training module, used to train the word segmentation model using the data set;

[0034] A commodity search word library creation module, used to preset a number of initial search words and create a commodity search word library based on each of the initial search words;

[0035] A commodity search word library expansion module is used to obtain an input real-time search sentence, perform a word segmentation operation on the real-time search sentence through the word segmentation model, obtain a plurality of real-time search words, and store each of the real-time search words in the commodity search word library;

[0036] The commodity search word library updating module is used to collect the real-time search behavior corresponding to the real-time search sentence in real time, and dynamically update the commodity search word library based on the real-time search behavior.

[0037] Furthermore, the data set construction module is specifically used for:

[0038] Acquire a large number of historical search statements, and perform preprocessing on each of the historical search statements, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spelling errors, removing special characters, removing punctuation marks, removing blanks, removing stop words, and removing low-frequency words;

[0039] Then, each of the pre-processed historical search statements is annotated with at least the length of the word segmentation window, the word segmentation position, the number of word segments, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; the user characteristics at least include browsing behavior, clicking behavior, collecting behavior, following behavior, purchasing behavior, adding to cart behavior, and sharing behavior; the product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; the scene characteristics at least include industry scenes and exclusive scenes;

[0040] A data set is constructed based on the annotated historical search statements.

[0041] Furthermore, the word segmentation model creation module is specifically used for:

[0042] A word segmentation model is created based on an input module, a BERT module, a classifier and an output module, and the loss function of the word segmentation model is set to a cross entropy function; the input module is used to input historical search sentences, and performs layer normalization and random inactivation operations on the input historical search sentences; the BERT module is used to identify entities from historical search sentences; the classifier is used to classify the identified entities; the output module is used to predict the probability of search word segmentation based on the classified entities, and output the search word segmentation results.

[0043] Furthermore, the word segmentation model training module is specifically used for:

[0044] The data set is divided into a training set, a validation set, and a test set based on a ratio of 8:1:1;

[0045] The word segmentation model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters of the word segmentation model including at least the number of iterations, the learning rate, and the training batch are continuously optimized during the training process;

[0046] The trained word segmentation model is verified by the verification set to determine whether the word segmentation accuracy of the word segmentation model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0047] The recall rate, precision rate and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate and F1-Score.

[0048] Furthermore, in the commodity search word library creation module, the commodity search word library is used to store search words and search weights, search times and user portraits corresponding to the search words; the user portraits at least include gender, age, region corresponding to the search IP address, occupation, price preference and search duration; the search words are initial search words or real-time search words;

[0049] The commodity search word library expansion module is specifically used for:

[0050] Obtaining an input real-time search statement through a visual interface, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a number of real-time search words, and determining whether the corresponding real-time search words already exist in the commodity search word library. If not, storing each of the real-time search words in the commodity search word library; if yes, ending the process;

[0051] The commodity search word library update module is specifically used for:

[0052] During the execution of the real-time search statement, real-time search behaviors including at least user characteristics, product characteristics and scene characteristics are collected in real time, and the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behaviors.

[0053] The advantages of the present invention are:

[0054] 1. Build a data set by obtaining a large number of historical search sentences, preprocessing and annotating them; then create a word segmentation model, set the loss function of the word segmentation model, and train the word segmentation model through the data set; then preset several initial search terms, and create a product search word library based on each initial search term; then obtain the input real-time search sentence, perform word segmentation operation on the real-time search sentence through the word segmentation model to obtain several real-time search terms, store each real-time search term in the product search word library, collect the real-time search behavior corresponding to the real-time search sentence in real time, and dynamically update the product search word library based on the real-time search behavior; since the search terms stored in the product search word library include not only the preset initial search terms, but also the real-time search terms of the word segmentation model created by the neural network for the real-time search sentence The real-time search terms obtained through word segmentation effectively expand the vocabulary of the product search word library, and the word segmentation model is trained on a data set constructed from a large number of historical search sentences, which can ensure the accuracy of word segmentation. In the search process, the real-time search behavior corresponding to the real-time search sentence is collected in real time, and the search weight, search times and user portrait of the corresponding search term in the product search word library are dynamically updated based on the real-time search behavior. That is, the product search word library is dynamically expanded based on the real-time search sentence, and labels are set for each search term in the product search word library based on the search weight, search times and user portrait. Subsequently, the most relevant search terms can be matched from the product search word library based on the user's real-time search behavior to perform word segmentation on the real-time search sentence, thereby greatly improving the accuracy of search word segmentation.

[0055] 2. By preprocessing each historical search statement, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spellings, removing special characters, removing punctuation marks, removing spaces, removing stop words, and removing low-frequency words, the data quality is effectively improved, thereby effectively improving the training effect of the word segmentation model and effectively improving the accuracy of word segmentation.

[0056] 3. Each historical search statement is annotated with at least the length of the word segmentation window, the word segmentation position, the number of words, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; user characteristics at least include browsing behavior, clicking behavior, collection behavior, attention behavior, purchase behavior, add-to-cart behavior, and sharing behavior; product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; scene characteristics at least include industry scenes and exclusive scenes; that is, historical search statements are comprehensively annotated, and subsequently word segmentation operations can be performed based on the search behavior to match the most relevant search terms, and related products can be recommended based on synonym matching, which greatly improves the personalization of product search recommendations and thus greatly improves the user experience.

[0057] 4. By setting the loss function of the word segmentation model to the cross entropy function, compared with other loss functions, because its gradient is proportional to the difference between the predicted value and the true value, it avoids the problem of gradient vanishing or exploding, and can converge to the optimal solution faster during the back propagation process, greatly improving the training speed of the word segmentation model.

[0058] 5. By continuously optimizing the hyperparameters including at least the number of iterations, learning rate, and training batches during the word segmentation model training process, the word segmentation accuracy is calculated through the validation set to verify the trained word segmentation model, the recall rate, precision rate, and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate, and F1-Score. That is, the word segmentation model is continuously optimized during the training process, and the word segmentation model is continuously verified and tested through a number of indicators, thereby greatly improving the accuracy of word segmentation of the word segmentation model.

[0059] 6. By collecting real-time search behaviors during the execution of real-time search statements, including at least user characteristics, product characteristics and scene characteristics, the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behavior, so that the product search vocabulary can capture the latest search characteristics (search trends) in real time, avoid lags, and further improve the accuracy of search word segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0061] Figure 1 The present invention is a flowchart of a method for generating a commodity search word library.

[0062] Figure 2 It is a structural schematic diagram of a commodity search word library generation system of the present invention. DETAILED DESCRIPTION

[0063] The technical solution in the embodiments of the present application has the following overall idea: pre-set initial search terms and real-time search terms obtained by segmenting real-time search statements through a word segmentation model are stored in a product search word library to expand the vocabulary of the product search word library, and the word segmentation model is trained with a data set constructed from a large number of historical search statements to ensure the accuracy of word segmentation; and real-time search behaviors corresponding to real-time search statements are collected in real time during the search process, and the search weights, search times and user portraits of the corresponding search terms in the product search word library are dynamically updated based on the real-time search behaviors; that is, the product search word library is dynamically expanded based on the real-time search statements, and labels are set for each search term in the product search word library based on the search weights, search times and user portraits, and subsequently the most relevant search terms can be matched from the product search word library based on the user's real-time search behavior to perform word segmentation operations on the real-time search statements to improve the accuracy of search word segmentation.

[0064] Please refer to Figure 1 to Figure 2 As shown, a preferred embodiment of a method for generating a commodity search word library of the present invention comprises the following steps:

[0065] Step S1, obtaining a large number of historical search statements, preprocessing and annotating each of the historical search statements to construct a data set;

[0066] Step S2: creating a word segmentation model and setting a loss function of the word segmentation model;

[0067] Step S3, training the word segmentation model using the data set;

[0068] Step S4: preset a number of initial search terms, and create a product search term library based on each of the initial search terms;

[0069] Step S5: obtaining an input real-time search statement, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a plurality of real-time search words, and storing each of the real-time search words in a commodity search word library;

[0070] Step S6: collecting the real-time search behavior corresponding to the real-time search statement in real time, and dynamically updating the commodity search word library based on the real-time search behavior.

[0071] The step S1 is specifically as follows:

[0072] Acquire a large number of historical search statements, and perform preprocessing on each of the historical search statements, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spelling errors, removing special characters, removing punctuation marks, removing blanks, removing stop words, and removing low-frequency words;

[0073] By preprocessing each historical search statement, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spellings, removing special characters, removing punctuation marks, removing whitespace, removing stop words, and removing low-frequency words, the data quality can be effectively improved, thereby effectively improving the training effect of the word segmentation model and effectively improving the accuracy of word segmentation.

[0074] Then, each of the pre-processed historical search statements is annotated with at least the length of the word segmentation window, the word segmentation position, the number of word segments, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; the user characteristics at least include browsing behavior, clicking behavior, collecting behavior, following behavior, purchasing behavior, adding to cart behavior, and sharing behavior; the product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; the scene characteristics at least include industry scenes and exclusive scenes;

[0075] Each historical search statement is annotated with at least the length of the word segmentation window, the word segmentation position, the number of words, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; user characteristics at least include browsing behavior, clicking behavior, collection behavior, attention behavior, purchase behavior, add-to-cart behavior, and sharing behavior; product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; scene characteristics at least include industry scenes and exclusive scenes; that is, historical search statements are comprehensively annotated, and subsequently word segmentation operations can be performed based on the search behavior to match the most relevant search terms, and recommendations can also be made based on synonym matching related products, which greatly improves the personalization of product search recommendations and thus greatly improves the user experience.

[0076] A data set is constructed based on the annotated historical search statements.

[0077] The step S2 is specifically as follows:

[0078] A word segmentation model is created based on an input module, a BERT module, a classifier and an output module, and the loss function of the word segmentation model is set to a cross entropy function; the input module is used to input historical search sentences, and performs layer normalization and random inactivation operations on the input historical search sentences; the BERT module is used to identify entities from historical search sentences; the classifier is used to classify the identified entities; the output module is used to predict the probability of search word segmentation based on the classified entities, and output the search word segmentation results.

[0079] By setting the loss function of the word segmentation model to the cross entropy function, compared with other loss functions, because its gradient is proportional to the difference between the predicted value and the true value, it avoids the problem of gradient vanishing or exploding, and can converge to the optimal solution faster during the back propagation process, greatly improving the training speed of the word segmentation model.

[0080] The step S3 is specifically as follows:

[0081] The data set is divided into a training set, a validation set, and a test set based on a ratio of 8:1:1;

[0082] The word segmentation model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters of the word segmentation model including at least the number of iterations, the learning rate, and the training batch are continuously optimized during the training process;

[0083] The trained word segmentation model is verified by the verification set to determine whether the word segmentation accuracy of the word segmentation model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0084] The recall rate, precision rate and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate and F1-Score.

[0085] By continuously optimizing at least the hyperparameters including the number of iterations, learning rate, and training batches during the word segmentation model training process, calculating the word segmentation accuracy through the validation set to verify the trained word segmentation model, calculating the recall rate, precision rate, and F1-Score of the word segmentation model through the test set, and testing the word segmentation model through the recall rate, precision rate, and F1-Score, that is, continuously optimizing the word segmentation model during the training process, and continuously verifying and testing the word segmentation model through a number of indicators, the accuracy of word segmentation of the word segmentation model is greatly improved.

[0086] In step S4, the commodity search word library is used to store search words and search weights, search times and user portraits corresponding to the search words; the user portraits at least include gender, age, region corresponding to the search IP address, occupation, price preference and search duration; the search words are initial search words or real-time search words;

[0087] The step S5 is specifically as follows:

[0088] Obtaining an input real-time search statement through a visual interface, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a number of real-time search words, and determining whether the corresponding real-time search words already exist in the commodity search word library. If not, storing each of the real-time search words in the commodity search word library; if yes, ending the process;

[0089] The step S6 is specifically as follows:

[0090] During the execution of the real-time search statement, real-time search behaviors including at least user characteristics, product characteristics and scene characteristics are collected in real time, and the search weight, search times and user portrait of the corresponding search terms in the product search vocabulary are dynamically updated based on the real-time search behaviors.

[0091] By collecting real-time search behaviors during the execution of real-time search statements, including at least user characteristics, product characteristics and scene characteristics, the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behavior, so that the product search vocabulary can capture the latest search characteristics (search trends) in real time, avoid lags, and further improve the accuracy of search term segmentation.

[0092] A preferred embodiment of a commodity search word library generation system of the present invention includes the following modules:

[0093] A data set construction module is used to obtain a large number of historical search statements, pre-process each of the historical search statements, and construct a data set after annotating;

[0094] A word segmentation model creation module, used to create a word segmentation model and set a loss function of the word segmentation model;

[0095] A word segmentation model training module, used to train the word segmentation model using the data set;

[0096] A commodity search word library creation module, used to preset a number of initial search words and create a commodity search word library based on each of the initial search words;

[0097] A commodity search word library expansion module is used to obtain an input real-time search sentence, perform a word segmentation operation on the real-time search sentence through the word segmentation model, obtain a plurality of real-time search words, and store each of the real-time search words in the commodity search word library;

[0098] The commodity search word library updating module is used to collect the real-time search behavior corresponding to the real-time search sentence in real time, and dynamically update the commodity search word library based on the real-time search behavior.

[0099] The data set construction module is specifically used for:

[0100] Acquire a large number of historical search statements, and perform preprocessing on each of the historical search statements, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spelling errors, removing special characters, removing punctuation marks, removing blanks, removing stop words, and removing low-frequency words;

[0101] By preprocessing each historical search statement, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spellings, removing special characters, removing punctuation marks, removing whitespace, removing stop words, and removing low-frequency words, the data quality can be effectively improved, thereby effectively improving the training effect of the word segmentation model and effectively improving the accuracy of word segmentation.

[0102] Then, each of the pre-processed historical search statements is annotated with at least the length of the word segmentation window, the word segmentation position, the number of word segments, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; the user characteristics at least include browsing behavior, clicking behavior, collecting behavior, following behavior, purchasing behavior, adding to cart behavior, and sharing behavior; the product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; the scene characteristics at least include industry scenes and exclusive scenes;

[0103] Each historical search statement is annotated with at least the length of the word segmentation window, the word segmentation position, the number of words, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; user characteristics at least include browsing behavior, clicking behavior, collection behavior, attention behavior, purchase behavior, add-to-cart behavior, and sharing behavior; product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; scene characteristics at least include industry scenes and exclusive scenes; that is, historical search statements are comprehensively annotated, and subsequently word segmentation operations can be performed based on the search behavior to match the most relevant search terms, and recommendations can also be made based on synonym matching related products, which greatly improves the personalization of product search recommendations and thus greatly improves the user experience.

[0104] A data set is constructed based on the annotated historical search statements.

[0105] The word segmentation model creation module is specifically used for:

[0106] A word segmentation model is created based on an input module, a BERT module, a classifier and an output module, and the loss function of the word segmentation model is set to a cross entropy function; the input module is used to input historical search sentences, and performs layer normalization and random inactivation operations on the input historical search sentences; the BERT module is used to identify entities from historical search sentences; the classifier is used to classify the identified entities; the output module is used to predict the probability of search word segmentation based on the classified entities, and output the search word segmentation results.

[0107] By setting the loss function of the word segmentation model to the cross entropy function, compared with other loss functions, because its gradient is proportional to the difference between the predicted value and the true value, it avoids the problem of gradient vanishing or exploding, and can converge to the optimal solution faster during the back propagation process, greatly improving the training speed of the word segmentation model.

[0108] The word segmentation model training module is specifically used for:

[0109] The data set is divided into a training set, a validation set, and a test set based on a ratio of 8:1:1;

[0110] The word segmentation model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters of the word segmentation model including at least the number of iterations, the learning rate, and the training batch are continuously optimized during the training process;

[0111] The trained word segmentation model is verified by the verification set to determine whether the word segmentation accuracy of the word segmentation model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and:

[0112] The recall rate, precision rate and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate and F1-Score.

[0113] By continuously optimizing at least the hyperparameters including the number of iterations, learning rate, and training batches during the word segmentation model training process, calculating the word segmentation accuracy through the validation set to verify the trained word segmentation model, calculating the recall rate, precision rate, and F1-Score of the word segmentation model through the test set, and testing the word segmentation model through the recall rate, precision rate, and F1-Score, that is, continuously optimizing the word segmentation model during the training process, and continuously verifying and testing the word segmentation model through a number of indicators, the accuracy of word segmentation of the word segmentation model is greatly improved.

[0114] In the commodity search word library creation module, the commodity search word library is used to store search words and search weights, search times and user portraits corresponding to the search words; the user portraits at least include gender, age, region corresponding to the search IP address, occupation, price preference and search duration; the search words are initial search words or real-time search words;

[0115] The commodity search word library expansion module is specifically used for:

[0116] Obtaining an input real-time search statement through a visual interface, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a number of real-time search words, and determining whether the corresponding real-time search words already exist in the commodity search word library. If not, storing each of the real-time search words in the commodity search word library; if yes, ending the process;

[0117] The commodity search word library update module is specifically used for:

[0118] During the execution of the real-time search statement, real-time search behaviors including at least user characteristics, product characteristics and scene characteristics are collected in real time, and the search weight, search times and user portrait of the corresponding search terms in the product search vocabulary are dynamically updated based on the real-time search behaviors.

[0119] By collecting real-time search behaviors during the execution of real-time search statements, including at least user characteristics, product characteristics and scene characteristics, the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behavior, so that the product search vocabulary can capture the latest search characteristics (search trends) in real time, avoid lags, and further improve the accuracy of search term segmentation.

[0120] In summary, the advantages of the present invention are:

[0121] 1. Build a data set by obtaining a large number of historical search sentences, preprocessing and annotating them; then create a word segmentation model, set the loss function of the word segmentation model, and train the word segmentation model through the data set; then preset several initial search terms, and create a product search word library based on each initial search term; then obtain the input real-time search sentence, perform word segmentation operation on the real-time search sentence through the word segmentation model to obtain several real-time search terms, store each real-time search term in the product search word library, collect the real-time search behavior corresponding to the real-time search sentence in real time, and dynamically update the product search word library based on the real-time search behavior; since the search terms stored in the product search word library include not only the preset initial search terms, but also the real-time search terms of the word segmentation model created by the neural network for the real-time search sentence The real-time search terms obtained through word segmentation effectively expand the vocabulary of the product search word library, and the word segmentation model is trained on a data set constructed from a large number of historical search sentences, which can ensure the accuracy of word segmentation. In the search process, the real-time search behavior corresponding to the real-time search sentence is collected in real time, and the search weight, search times and user portrait of the corresponding search term in the product search word library are dynamically updated based on the real-time search behavior. That is, the product search word library is dynamically expanded based on the real-time search sentence, and labels are set for each search term in the product search word library based on the search weight, search times and user portrait. Subsequently, the most relevant search terms can be matched from the product search word library based on the user's real-time search behavior to perform word segmentation on the real-time search sentence, thereby greatly improving the accuracy of search word segmentation.

[0122] 2. By preprocessing each historical search statement, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spellings, removing special characters, removing punctuation marks, removing spaces, removing stop words, and removing low-frequency words, the data quality is effectively improved, thereby effectively improving the training effect of the word segmentation model and effectively improving the accuracy of word segmentation.

[0123] 3. Each historical search statement is annotated with at least the length of the word segmentation window, the word segmentation position, the number of words, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; user characteristics at least include browsing behavior, clicking behavior, collection behavior, attention behavior, purchase behavior, add-to-cart behavior, and sharing behavior; product characteristics at least include product nouns, product categories, shelf time, price, brand, sales volume, and ratings; scene characteristics at least include industry scenes and exclusive scenes; that is, historical search statements are comprehensively annotated, and subsequently word segmentation operations can be performed based on the search behavior to match the most relevant search terms, and related products can be recommended based on synonym matching, which greatly improves the personalization of product search recommendations and thus greatly improves the user experience.

[0124] 4. By setting the loss function of the word segmentation model to the cross entropy function, compared with other loss functions, because its gradient is proportional to the difference between the predicted value and the true value, it avoids the problem of gradient vanishing or exploding, and can converge to the optimal solution faster during the back propagation process, greatly improving the training speed of the word segmentation model.

[0125] 5. By continuously optimizing the hyperparameters including at least the number of iterations, learning rate, and training batches during the word segmentation model training process, the word segmentation accuracy is calculated through the validation set to verify the trained word segmentation model, the recall rate, precision rate, and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate, and F1-Score. That is, the word segmentation model is continuously optimized during the training process, and the word segmentation model is continuously verified and tested through a number of indicators, thereby greatly improving the accuracy of word segmentation of the word segmentation model.

[0126] 6. By collecting real-time search behaviors during the execution of real-time search statements, including at least user characteristics, product characteristics and scene characteristics, the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behavior, so that the product search vocabulary can capture the latest search characteristics (search trends) in real time, avoid lags, and further improve the accuracy of search word segmentation.

[0127] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for generating a commodity search word library, characterized in that: The steps include: Step S1, obtaining a large number of historical search statements, preprocessing and annotating each of the historical search statements to construct a data set; Step S2: creating a word segmentation model and setting a loss function of the word segmentation model; Step S3, training the word segmentation model using the data set; Step S4: preset a number of initial search terms, and create a product search term library based on each of the initial search terms; Step S5: obtaining an input real-time search statement, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a plurality of real-time search words, and storing each of the real-time search words in a commodity search word library; Step S6: collecting the real-time search behavior corresponding to the real-time search statement in real time, and dynamically updating the commodity search word library based on the real-time search behavior.

2. A method for generating a commodity search word library according to claim 1, characterized in that: The step S1 is specifically as follows: Acquire a large number of historical search statements, and perform preprocessing on each of the historical search statements, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spelling errors, removing special characters, removing punctuation marks, removing blanks, removing stop words, and removing low-frequency words; Then, each of the pre-processed historical search statements is annotated with at least the length of the word segmentation window, the word segmentation position, the number of word segments, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; The user characteristics at least include browsing behavior, clicking behavior, collecting behavior, following behavior, purchasing behavior, adding to cart behavior and sharing behavior; the product characteristics at least include product name, product category, shelf time, price, brand, sales volume and rating; the scenario characteristics at least include industry scenarios and exclusive scenarios; A data set is constructed based on the annotated historical search statements.

3. A method for generating a commodity search word library according to claim 1, characterized in that: The step S2 is specifically as follows: A word segmentation model is created based on an input module, a BERT module, a classifier and an output module, and the loss function of the word segmentation model is set to a cross entropy function; the input module is used to input historical search sentences, and performs layer normalization and random inactivation operations on the input historical search sentences; the BERT module is used to identify entities from historical search sentences; the classifier is used to classify the identified entities; the output module is used to predict the probability of search word segmentation based on the classified entities, and output the search word segmentation results.

4. A method for generating a commodity search word library according to claim 1, characterized in that: The step S3 is specifically as follows: The data set is divided into a training set, a validation set, and a test set based on a ratio of 8:1:1; The word segmentation model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters of the word segmentation model including at least the number of iterations, the learning rate, and the training batch are continuously optimized during the training process; The trained word segmentation model is verified by the verification set to determine whether the word segmentation accuracy of the word segmentation model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and: The recall rate, precision rate and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate and F1-Score.

5. A method for generating a commodity search word library according to claim 1, characterized in that: In step S4, the commodity search word library is used to store search words and search weights, search times and user portraits corresponding to the search words; the user portraits at least include gender, age, region corresponding to the search IP address, occupation, price preference and search duration; the search words are initial search words or real-time search words; The step S5 is specifically as follows: Obtaining an input real-time search statement through a visual interface, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a number of real-time search words, and determining whether the corresponding real-time search words already exist in the commodity search word library. If not, storing each of the real-time search words in the commodity search word library; if yes, ending the process; The step S6 is specifically as follows: During the execution of the real-time search statement, real-time search behaviors including at least user characteristics, product characteristics and scene characteristics are collected in real time, and the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behaviors.

6. A system for generating a commodity search word library, characterized in that: Includes the following modules: A data set construction module is used to obtain a large number of historical search statements, pre-process each of the historical search statements, and construct a data set after annotating; A word segmentation model creation module, used to create a word segmentation model and set a loss function of the word segmentation model; A word segmentation model training module, used to train the word segmentation model using the data set; A commodity search word library creation module, used to preset a number of initial search words and create a commodity search word library based on each of the initial search words; A commodity search word library expansion module is used to obtain an input real-time search sentence, perform a word segmentation operation on the real-time search sentence through the word segmentation model, obtain a plurality of real-time search words, and store each of the real-time search words in the commodity search word library; The commodity search word library updating module is used to collect the real-time search behavior corresponding to the real-time search sentence in real time, and dynamically update the commodity search word library based on the real-time search behavior.

7. A commodity search word library generation system as claimed in claim 6, characterized in that: The data set construction module is specifically used for: Acquire a large number of historical search statements, and perform preprocessing on each of the historical search statements, including at least converting full-width to half-width, converting uppercase and lowercase, converting traditional Chinese to simplified Chinese, correcting spelling errors, removing special characters, removing punctuation marks, removing blanks, removing stop words, and removing low-frequency words; Then, each of the pre-processed historical search statements is annotated with at least the length of the word segmentation window, the word segmentation position, the number of word segments, the synonyms corresponding to the search terms, and the search behavior; the search behavior at least includes user characteristics, product characteristics, and scene characteristics; The user characteristics at least include browsing behavior, clicking behavior, collecting behavior, following behavior, purchasing behavior, adding to cart behavior and sharing behavior; the product characteristics at least include product name, product category, shelf time, price, brand, sales volume and rating; the scenario characteristics at least include industry scenarios and exclusive scenarios; A data set is constructed based on the annotated historical search statements.

8. A commodity search word library generation system as claimed in claim 6, characterized in that: The word segmentation model creation module is specifically used for: A word segmentation model is created based on an input module, a BERT module, a classifier and an output module, and the loss function of the word segmentation model is set to a cross entropy function; the input module is used to input historical search sentences, and performs layer normalization and random inactivation operations on the input historical search sentences; the BERT module is used to identify entities from historical search sentences; the classifier is used to classify the identified entities; the output module is used to predict the probability of search word segmentation based on the classified entities, and output the search word segmentation results.

9. A commodity search word library generation system as claimed in claim 6, characterized in that: The word segmentation model training module is specifically used for: The data set is divided into a training set, a validation set, and a test set based on a ratio of 8:1:1; The word segmentation model is trained by the training set until the loss value of the loss function is less than a preset loss threshold, and the hyperparameters of the word segmentation model including at least the number of iterations, the learning rate, and the training batch are continuously optimized during the training process; The trained word segmentation model is verified by the verification set to determine whether the word segmentation accuracy of the word segmentation model is greater than a preset accuracy threshold. If not, the verification fails and the training set is expanded to continue training; if yes, the verification passes, and: The recall rate, precision rate and F1-Score of the word segmentation model are calculated through the test set, and the word segmentation model is tested through the recall rate, precision rate and F1-Score.

10. A commodity search word library generation system as claimed in claim 6, characterized in that: In the commodity search word library creation module, the commodity search word library is used to store search words and search weights, search times and user portraits corresponding to the search words; the user portraits at least include gender, age, region corresponding to the search IP address, occupation, price preference and search duration; the search words are initial search words or real-time search words; The commodity search word library expansion module is specifically used for: Obtaining a real-time search statement input through a visual interface, performing a word segmentation operation on the real-time search statement through the word segmentation model to obtain a number of real-time search words, and determining whether a corresponding real-time search word already exists in the commodity search word library; if not, storing each of the real-time search words in the commodity search word library; If yes, then end the process; The commodity search word library update module is specifically used for: During the execution of the real-time search statement, real-time search behaviors including at least user characteristics, product characteristics and scene characteristics are collected in real time, and the search weight, search times and user portrait of the product search vocabulary are dynamically updated based on the real-time search behaviors.

Images