A query expression automatic generation device for a search system
By constructing an automated query word tree generation device, and utilizing machine learning algorithms and training datasets, the problem of low efficiency in traditional query expression construction relying on expert experience is solved. This achieves efficient and accurate query expression generation, improving the user experience of information retrieval.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAOHONGSHU TECH CO LTD
- Filing Date
- 2020-11-18
- Publication Date
- 2026-06-02
AI Technical Summary
Traditional retrieval methods rely on expert experience to construct query expressions, which is inefficient and ineffective, making it difficult to meet the requirements of high efficiency and accuracy in information retrieval, especially in large-scale indexing scenarios.
An automated query expression generation device for search systems is adopted. It constructs a query word tree through machine learning algorithms and uses training datasets and information source data to automatically generate query expressions, including word segmentation, classification, and determination of query domain priority. It constructs query conditions and groups them into a tree structure.
It improves the accuracy of autocomplete for query terms, simplifies the process of constructing query expressions, and enhances user experience and retrieval efficiency.
Smart Images
Figure HDA0002786670410000011
Abstract
Description
Technical Field
[0001] This invention relates to the field of information retrieval, and more particularly to a device for automatically generating retrieval expressions. Background Technology
[0002] With the advent of the information age, information and data have been growing explosively. The importance of information search is increasingly prominent, and the demands for efficiency and accuracy in retrieval are also rising. The workflow of a search system begins with the user entering a query term. The system parses the query term, constructs a query expression, provides the query expression to the inverted index, and finally returns the results to the user. In traditional retrieval models, constructing a query expression from the user's input relies heavily on the expert experience of the retrieval system engineer; novices often find it difficult to construct effective query expressions.
[0003] As information continues to increase, the size of the index also continues to expand. In order to ensure the efficiency and accuracy of the search, it is often necessary to refactor the query expression. If the traditional method of refactoring, which relies on expert experience, is used, the efficiency of the refactoring will be very low and the effect will be very poor. Summary of the Invention
[0004] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide an automated query expression generation device for search systems, which can improve the accuracy of automatic completion of search terms and enhance user experience in search scenarios targeting specific information sources.
[0005] To achieve the above and other related objectives, the present invention provides an automated query expression generation device for search systems, characterized by the following steps: selecting a training dataset; preprocessing query terms; determining a query domain; generating query conditions; and constructing a query term tree based on the query conditions, the query domain, and information from the information source.
[0006] Preferably, in the above-mentioned automated query expression generation device for search systems, the training dataset contains data clicked by the user as positive samples and randomly sampled data from the information source as negative samples.
[0007] Preferably, in the above-mentioned automated query expression generation device for search systems, the step of preprocessing the query terms includes word segmentation and / or classification of the query terms.
[0008] Preferably, in the above-mentioned automated query expression generation device for search systems, the classification step of the query terms may include at least one level.
[0009] Preferably, in the above-mentioned automated query expression generation device for search systems, the step of determining the query domain further includes assigning different priorities to the query domain.
[0010] Preferably, in the above-mentioned automated query expression generation device for search systems, the step of generating query conditions is that the query conditions are composed of the recognition results of the query terms and the query domain.
[0011] Preferably, in the above-mentioned automated query expression generation device for search systems, in the step of constructing a query term tree based on the query conditions, the query domain, and the information source information, the query conditions are used as nodes of the tree, the information source information is grouped according to the query conditions, and the left and right children of the nodes are determined.
[0012] This invention also provides an automated query expression generation method for search systems, characterized by including the steps of constructing a query term tree and generating query expressions; the step of constructing the query term tree includes: selecting a training dataset; preprocessing the query terms; determining the query domain; generating query conditions; and constructing the query term tree based on the query conditions, the query domain, and the information from the information source.
[0013] Preferably, in the above-described automated query expression generation method for search systems, the query expression generation step generates the query expression by traversing the query term tree.
[0014] Preferably, in the above-described automated query expression generation method for search systems, the data in the training dataset that is clicked by the user is used as positive samples, and the randomly sampled data in the information source is used as negative samples.
[0015] Preferably, in the above-described automated query expression generation method for search systems, the step of preprocessing the query terms includes word segmentation and / or classification of the query terms.
[0016] Preferably, in the above-described automated query expression generation method for search systems, the classification step of the query terms may include at least one level.
[0017] Preferably, in the above-described automated query expression generation method for search systems, the step of determining the query domain further includes assigning different priorities to the query domain.
[0018] Preferably, in the above-described automated query expression generation method for search systems, the step of generating query conditions involves combining the recognition results of the query terms with the query domain.
[0019] Preferably, in the above-mentioned automated query expression generation method for search systems, in the step of constructing a query term tree based on the query conditions, the query domain, and the information source information, the query conditions are used as nodes of the tree, the information source information is grouped according to the query conditions, and the left and right children of the nodes are determined. Attached Figure Description
[0020] Figure 1 A schematic diagram of a query tree constructed according to the query tree construction method of the present invention. Detailed Implementation
[0021] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention.
[0022] Please refer to the accompanying drawings. It should be noted that the illustrations provided in this embodiment are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.
[0023] The automated query expression generation device for search systems of the present invention divides the construction of query expressions into two parts. The first part is to build a query tree and generate a query expression. The second part is to use the query expression to find notes related to the query based on the user's specific input query term. The steps of building the query tree are illustrated below, using notes shared by netizens as an example:
[0024] Step 1: Select the training dataset. In practice, tree construction is mostly achieved by feeding the training dataset to a machine learning algorithm, which then automatically learns and constructs the dataset. In this example, the positive samples (relevant query and note pairs) in the training dataset are historical user click data, while the negative samples (irrelevant query and note pairs) are randomly sampled from the total note database. Through training, the machine learns the criteria for judging the relevance of queries and notes, determining which notes are relevant and which are not. It's understandable that the sample selection criteria and data feature construction for training the machine can be adjusted based on the distribution of information source data and the target user group for the final query expression.
[0025] Step 2: Preprocess the query. This preprocessing can include word segmentation, phrase recognition, and classification. Word segmentation technology is mature; the segmented words are called terms. Currently, different word segmenters often differ in their granularity based on different business needs. In this example, the information source is a user-shared shopping note, which tends to treat brand names and product names as phrases, thus requiring phrase recognition and classification. A phrase is one or more terms, and the object of classification is a phrase. Classification uses different standards based on different business needs and can have different classification levels. For example, in this case, the first level can divide terms into entity phrases and non-entity phrases, and the second level can further classify entity phrases into brand words, category words, personal names, place names, etc. For example, a selected query (selected by the trained machine based on user input) is "ysl lipstick square tube 50". In this example, the word segmentation result is four terms: "ysl", "lipstick", "square tube", and "50". Among them, "ysl" and "lipstick" are the brand name and product name, respectively, and are recognized as entity phrases. "Square tube" and "50" are descriptions of product attributes, identified as shape and model in non-entity phrases. Entity phrases are the core of a query. Given a defined entity phrase, concatenating non-entity phrases improves the accuracy of search results. For example, in the above example, if "square tube" and "50" were not concatenated, the search results might include "YSL square tube lipstick" and "50 lipsticks." However, concatenating "square tube" and "50" helps eliminate irrelevant search results (the notes must contain both "square tube" and "50" consecutively). In this example, such concatenation is based on the data characteristics of the information source, calculating the density score between non-entity terms. For terms with a density score greater than a preset threshold, the two terms are concatenated.
[0026] Step 3: Determine the query fields. Selecting query fields involves more than just identifying each field. The selection of query fields typically needs to be adjusted based on different business requirements. In this example, the query fields are tags, titles, and body text. The query priority tree can be learned from the training set.
[0027] Step 4: Generate query conditions. In this example, query conditions are composed of entity phrases, non-entity phrases, and fields. For example, a query condition could be something like "tags include ysl", "text includes ysl", or "tags include lipstick". You can choose and combine them according to different business needs, but in this step, you should ensure that each term is covered by at least one query condition.
[0028] Step 5: Construct the query tree. The method for building the tree is to group information based on nodes (query conditions). Notes that meet the node's criteria are placed to the left of the node, and notes that do not meet the criteria are placed to the right. When the proportion of relevant or irrelevant notes under a node reaches a certain threshold (95% in this example), that group becomes a leaf node, and grouping stops. The category with the highest proportion in the leaf node (both relevant and irrelevant categories) becomes the category of that leaf node. Let's assume that the query conditions generated in Step 4 are three: "Does the tag include lipstick?", "Does the title include YSL?", and "Does the body text include YSL?". There are a total of 9 relevant notes in the information source, numbered 1 to 9 (assuming 1, 2, 5, and 8 should be search results; 3, 4, 6, 7, and 9 should not be search results; specific rules will be explained later). First, group the messages 1-9 using the condition X1 = {"Tags" include "lipstick"}. Messages (1-4, 7, 8) satisfy the condition and are placed to the left of X1, while messages (5, 6, 9) do not and are placed to the right. Then, further group the left and right children of X1 using the condition X2 = {"Titles" include "ysl"}. Finally, group each group using X3 = {"Body" includes "ysl"}. The resulting query tree is as follows. Figure 1 As shown, relevant (as search results) and irrelevant (not as search results) leaf nodes are ultimately generated based on the conditions.
[0029] based on Figure 1 With a query tree and pre-defined retrieval rules, query expressions can be easily constructed. In this example, the retrieval rule is: at least two of the above query conditions must be relevant to be considered a search result; otherwise, the information will not be considered a search result. Of course, this rule is only for illustrative purposes; in practice, retrieval rules are far more complex and are usually obtained through machine training. Under this retrieval rule, for... Figure 1 After traversing the query tree, the following query expression can be obtained:
[0030] (“tag.lipstick=1”and“title.ysl=1”and“body.ysl=1”)or(“tag.lipstick=1”and“title.ysl=0”and“body.ysl=1”)or(“tag.lipstick=0”and“title.ysl=1”and“body.ysl=1”)
[0031] Equivalent to:
[0032] (“tag.lipstick=1”and“title.ysl=1”and“body.ysl=1”)or(“tag.lipstick=1”and“body.ysl=1”)or(“title.ysl=1”and“body.ysl=1”).
[0033] In summary, the automated query expression generation device for search systems of the present invention can construct query expressions based on a query tree. The query tree is trained using a training set based on a machine learning algorithm, which can transform the problem of constructing query expressions into a tree traversal problem. Since tree traversal algorithms are already mature, the operation is greatly simplified.
[0034] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. All equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.
Claims
1. An automated query expression generation device for search systems, characterized in that, This includes a query term tree, which is constructed through the following steps: The step of selecting a training dataset is as follows: in the training dataset, data that users click on are used as positive samples and randomly sampled data from the information source are used as negative samples. The positive samples include relevant query terms and note pairs, and the negative samples include irrelevant query terms and note pairs. The steps for preprocessing query terms include: segmenting the query terms to obtain at least one term; classifying the term into entity phrases and non-entity phrases, wherein the entity phrases include brand terms and category terms, and the non-entity phrases include descriptions of product attributes; calculating the closeness score between terms classified as non-entity phrases based on the data characteristics of the information source; and concatenating terms of non-entity phrases with a closeness score greater than a preset threshold. Steps to determine the query domain; The step of generating query conditions, wherein the query conditions are composed of entity phrases, non-entity phrases and query domains; The steps of constructing a query term tree based on the query conditions, the query domain, and the information source include: using the query conditions as nodes of the tree; placing notes that meet the node conditions to the left of the node and notes that do not meet the node conditions to the right of the node based on the GINI coefficient defined by the classification purity on the node; when the proportion of related or unrelated notes under any node reaches a threshold, the node is designated as a leaf node and grouping stops; and the category with the higher proportion of related and unrelated notes in the leaf node is designated as the category of the leaf node. The device is also used to traverse the query term tree and generate a query expression according to preset retrieval rules.
2. The automated query expression generation device for a search system according to claim 1, characterized in that, The preprocessing steps for query terms include word segmentation and / or classification of query terms.
3. The automated query expression generation device for a search system according to claim 2, characterized in that, In the step of classifying query terms, the classification may include at least one level.
4. The automated query expression generation device for a search system according to claim 1, characterized in that, The step of determining the query domain also includes assigning different priorities to the query domain.
5. The automated query expression generation device for a search system according to claim 1, characterized in that, In the step of generating query conditions, the query conditions are formed by combining the recognition result of the query term with the query domain.
6. The automated query expression generation device for a search system according to claim 1, characterized in that, In the step of constructing a query term tree based on the query conditions, the query domain, and the information source information, the query conditions are used as nodes of the tree, the information source information is grouped according to the query conditions, and the left and right children of the nodes are determined.
7. The automated query expression generation device for a search system according to claim 1, characterized in that, It also includes the step of generating a query expression based on the query term tree.
8. A method for automatically generating query expressions for search systems, characterized in that, This includes the steps of constructing the query term tree and generating the query expression; The steps for constructing the query term tree include: The step of selecting a training dataset is as follows: in the training dataset, data that users click on are used as positive samples and randomly sampled data from the information source are used as negative samples. The positive samples include relevant query terms and note pairs, and the negative samples include irrelevant query terms and note pairs. The steps for preprocessing query terms include: segmenting the query terms to obtain at least one term; classifying the term into entity phrases and non-entity phrases, wherein the entity phrases include brand terms and category terms, and the non-entity phrases include descriptions of product attributes; calculating the closeness score between terms classified as non-entity phrases based on the data characteristics of the information source; and concatenating terms of non-entity phrases with a closeness score greater than a preset threshold. Steps to determine the query domain; The step of generating query conditions, wherein the query conditions are composed of entity phrases, non-entity phrases and query domains; The steps of constructing a query term tree based on the query conditions, the query domain, and the information source include: using the query conditions as nodes of the tree; placing notes that meet the node conditions to the left of the node and notes that do not meet the node conditions to the right of the node based on the GINI coefficient defined by the classification purity on the node; when the proportion of related or unrelated notes under any node reaches a threshold, the node is designated as a leaf node and grouping stops; and the category with the higher proportion of related and unrelated notes in the leaf node is designated as the category of the leaf node. The step of generating a query expression includes traversing the query term tree and generating a query expression according to preset retrieval rules.
9. The method for automatically generating query expressions for a search system according to claim 8, characterized in that, The preprocessing steps for query terms include word segmentation and / or classification of query terms.
10. The method for automatically generating query expressions for a search system according to claim 9, characterized in that, In the step of classifying query terms, the classification may include at least one level.
11. The method for automatically generating query expressions for a search system according to claim 10, characterized in that, The step of determining the query domain also includes assigning different priorities to the query domain.
12. The automated query expression generation method for search systems according to claim 8, characterized in that, In the step of generating query conditions, the query conditions are formed by combining the recognition result of the query term with the query domain.
13. The method for automatically generating query expressions for a search system according to claim 8, characterized in that, In the step of constructing a query term tree based on the query conditions, the query domain, and the information source, the query conditions are... For each node in the tree, the information from the information source is grouped according to the query conditions to determine the left and right children of the node.
14. The method for automatically generating query expressions for a search system according to claim 8, characterized in that, It also includes the step of generating a query expression based on the query term tree.