Retrieval method and device based on tree structure

Abstract

The invention discloses a retrieval method and device based on a tree structure, relates to the technical field of information retrieval, and mainly aims to ensure that a retrieval result is comprehensive and accurate so as to improve the retrieval recall rate. According to the main technical scheme, the method comprises the steps that a retrieval request of a user is received and analyzed, and retrieval content corresponding to the retrieval request is obtained; detecting whether the retrieval content has a hit branch index in a tree index structure or not, the tree index structure being constructed by a minimum category in a hierarchical classification structure of original data, and the minimum category being in one-to-one correspondence with the branch index; if yes, determining a retrieval result matched with the retrieval content in the target branch index according to a first retrieval logic; and if not, determining a retrieval result matched with the retrieval content in all branch indexes of the tree index structure according to a second retrieval logic. The method and the device are used for realizing information retrieval based on the tree index structure.

Description

Technical Field

[0001] The present application relates to the field of information retrieval technology, and in particular to a retrieval method and device based on a tree structure. Background Art

[0002] With the development of information technology, especially the continuous progress in natural language processing (NLP) and information retrieval, people are increasingly relying on intelligent retrieval systems to quickly retrieve the information they need. The RAG (Retrieval-Augmented Generation) technology combines the advantages of information retrieval and generative models, aiming to enhance the performance of generative models by retrieving relevant information. It is especially suitable for application scenarios that require real-time update and expansion of knowledge bases, such as voice assistants, intelligent customer service, etc.

[0003] At present, when searching based on RAG technology, the index structure of its database is one-dimensional, so the search results usually use a single-dimensional recall. This method is prone to omissions when searching for highly homogeneous content, resulting in incomplete and inaccurate search results, thereby reducing the recall rate of the search. Summary of the invention

[0004] In view of the above problems, the present application provides a tree-structured retrieval method and device, the main purpose of which is to ensure that the retrieval results are comprehensive and accurate so as to improve the retrieval recall rate.

[0005] In order to solve the above technical problems, this application proposes the following solutions:

[0006] In a first aspect, the present application provides a tree-structure based retrieval method, the method comprising:

[0007] Receive and analyze the user's search request to obtain the search content corresponding to the search request;

[0008] Detecting whether there is a hit branch index in the search content in a tree index structure, wherein the tree index structure is constructed based on the smallest category in the hierarchical classification structure of the original data, and the smallest category corresponds to the branch index one by one;

[0009] If so, determining the search result matching the search content in the target branch index according to the first search logic, the target branch index being the hit branch index, and the first search logic being used to characterize matching and recalling the search result in the classification path corresponding to the target branch index;

[0010] If not, then the search results matching the search content are determined in all branch indexes of the tree index structure according to a second search logic, where the second search logic is used to represent matching and recalling the search results in the classification paths corresponding to all branch indexes respectively.

[0011] In a second aspect, the present application provides a tree-structure-based retrieval device, the device comprising:

[0012] An acquisition unit, used to receive and analyze a user's search request to obtain search content corresponding to the search request;

[0013] a detection unit, configured to detect whether the search content obtained by the acquisition unit has a hit branch index in a tree index structure, wherein the tree index structure is constructed based on the smallest category in the hierarchical classification structure of the original data, and the smallest category corresponds to the branch index one by one;

[0014] A first processing unit is configured to determine, if the detection unit detects that the search content has a hit branch index in the tree index structure, a search result matching the search content in the target branch index according to a first search logic, wherein the target branch index is the hit branch index, and the first search logic is configured to characterize matching and recalling the search result in a classification path corresponding to the target branch index;

[0015] A second processing unit is configured to determine, if the detection unit detects that there is no hit branch index for the search content in the tree index structure, a search result matching the search content in all branch indexes of the tree index structure according to a second search logic, wherein the second search logic is configured to represent and recall the search result in the classification paths corresponding to all the branch indexes respectively.

[0016] In order to achieve the above-mentioned purpose, according to the third aspect of the present application, a storage medium is provided, wherein the storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute the tree-structure-based retrieval method of the above-mentioned first aspect.

[0017] In order to achieve the above-mentioned purpose, according to a fourth aspect of the present application, a processor is provided, wherein the processor is used to run a program, wherein the tree-structure-based retrieval method of the above-mentioned first aspect is executed when the program is run.

[0018] By means of the above technical scheme, the present application provides a tree-structure-based retrieval method and device, which receives and parses the user's retrieval request when searching for the information required by the user, and after obtaining the retrieval content corresponding to the retrieval request, detects whether there is a hit branch index in the retrieval content in the tree index structure. The tree index structure is constructed based on the minimum category in the hierarchical classification structure of the original data, and the minimum category corresponds to the branch index one by one. If there is a hit branch index, it is used as the target branch index, and the retrieval result matching the retrieval content is determined in the target branch index according to the first retrieval logic, that is, the retrieval result is matched and recalled in the classification path corresponding to the target branch index. If there is no hit branch index, the retrieval result matching the retrieval content is determined in all branch indexes of the tree index structure according to the second retrieval logic, that is, the retrieval result is matched and recalled in the classification paths corresponding to all branch indexes respectively. The technical solution provided by this application can detect whether the search content hits the matching branch index, that is, determine whether there is a category corresponding to the search content in the tree index structure. When it hits, it can quickly locate the branch that is most likely to contain the search results, that is, determine and recall the matching search results in the matching category, narrowing the search scope and significantly improving the search efficiency and accuracy. When it misses, it can search in all branch indexes in parallel, that is, determine and recall the matching search results in all categories separately, which not only speeds up the search speed, but also ensures that no information is missed, especially when searching for highly homogenized content, it can provide more accurate and comprehensive result coverage and improve the recall rate of the search.

[0019] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0021] Figure 1 A flowchart of a tree-structure-based retrieval method provided in an embodiment of the present application is shown;

[0022] Figure 2 Another flowchart of a tree-structure-based retrieval method provided in an embodiment of the present application is shown;

[0023] Figure 3A block diagram showing a tree-structure-based retrieval device provided in an embodiment of the present application is shown;

[0024] Figure 4 A block diagram showing another tree-structure based retrieval device provided in an embodiment of the present application is shown;

[0025] Figure 5 A schematic diagram comparing the prior art search and the tree structure-based search provided in the embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0027] At present, when searching based on RAG technology, the index structure of its database is one-dimensional, so the search results usually use a single-dimensional recall. This method is prone to omissions when searching for highly homogeneous content, resulting in incomplete and inaccurate search results, thereby reducing the recall rate of the search.

[0028] For example, suppose that the database has a total of 5 million scientific articles, of which about 500,000 are related to "artificial intelligence", and of these 500,000 articles, about 200,000 are highly homogeneous, that is, the topics, experimental methods, data sets, etc. discussed are very similar. Suppose the search request entered by the user is: "The latest research results in artificial intelligence". If a single-dimensional recall is used, it is only searched under the large category of "artificial intelligence". Because the subtle differences between highly homogeneous scientific articles are difficult to distinguish, the search results may miss some key but different scientific articles.

[0029] After research, the inventor found that the index can be reconstructed according to the tree structure, that is, the data is graded and layered, and the tree index structure is constructed according to the minimum category after the grading and layering, so that each branch index of the tree index structure corresponds to a minimum category, so that when the user's search request hits a branch index, the corresponding search results can be matched and recalled in the minimum category corresponding to the branch index, and an accurate single-way recall effect can be achieved; and when the user's search request does not hit any branch index, the corresponding search results can be matched and recalled in each minimum category corresponding to all branch indexes, so as to return as many matching results as possible and achieve multi-way recall. In this way, not only can the search efficiency and accuracy be improved, but also it can ensure that no information is missed, especially when searching for highly homogeneous content, it can provide more accurate and comprehensive result coverage and improve the search recall rate.

[0030] Combining the above example, we can search for "artificial intelligence" based on the tree index structure:

[0031] In the case of a single-way recall, a deep search will be performed under the general category of "artificial intelligence". If the search request also contains more specific sub-category information such as "natural language processing", a deep search will be performed directly under the category of "natural language processing" to find the most relevant latest research results, which narrows the search scope and improves the search efficiency and accuracy.

[0032] During multi-way recall: deep searches will be performed in parallel in all subcategories related to "artificial intelligence", such as "natural language processing", "machine learning", "computer vision", etc., to ensure that even in highly homogeneous content, no potentially relevant latest research results will be missed, which provides more accurate and comprehensive result coverage and improves the recall rate of searches.

[0033] like Figure 1 As shown, an embodiment of the present application provides a retrieval method based on a tree structure, through which the comprehensiveness and accuracy of the retrieval results can be guaranteed to improve the retrieval recall rate. The executor of this application can be any system or device with information retrieval function, including but not limited to search engine servers, intelligent customer service robots, voice assistants or big data analysis platforms. In some scenarios, the embodiments of the present application are suitable for processing data sets with hierarchical relationships, such as product classifications, document subject classifications, news categories, etc. The tree-structured retrieval method provided in the embodiments of the present application includes at least 101-104.

[0034] 101. Receive and parse a user's search request to obtain search content corresponding to the search request.

[0035] In this step, the search request can be submitted to the system by the user through text input, voice command input or graphical user interface (GUI). The search request may contain keywords, phrases or more complex query statements, such as keywords such as "artificial intelligence", phrases such as "machine learning algorithm", and more complex query statements such as "artificial intelligence AND ethics" or "big data OR data mining" containing logical operators (such as AND, OR and NOT). The system receives and parses the search request and obtains the specific content that the user wants to find, that is, the search content. During the parsing process, the search request can be parsed by natural language processing technologies such as lexical analysis, grammatical analysis and semantic analysis, so that the system can correctly identify the user's search intention.

[0036] 102. Check whether there is a hit branch index in the tree index structure for the search content.

[0037] The tree index structure is constructed based on the smallest category in the hierarchical classification structure of the original data, and the smallest category corresponds to the branch index one by one.

[0038] In the embodiments of the present application, the original data may be data stored according to preset standards or acquired source data. It should be noted that the information, data and signals involved in the embodiments of the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0039] The hierarchical classification structure of raw data is used to represent the structure obtained by organizing and constructing raw data according to certain levels and categories. It can be determined by extracting the category identifier in the raw data meta-information, such as from the file name, label, description and other meta-information. The category identifier includes but is not limited to keywords, labels or other information describing data attributes. After obtaining the category identifier, it can be classified first, and after classification, the hierarchical classification structure can be constructed according to the hierarchical affiliation of each category identifier. By establishing this hierarchical classification structure, more efficient management and retrieval can be achieved, and a certain information category can be quickly located without traversing all the data.

[0040] After the hierarchical classification structure is determined, the category represented by the lowest classification layer of the hierarchical classification structure is the smallest category. The index can be reconstructed by taking the lowest classification layer as a leaf node. Specifically, a smallest category is used as a leaf node, and a leaf node represents a branch index, that is, a branch index corresponds to a smallest category, thereby obtaining a tree index structure.

[0041] Exemplarily, assume that the extracted category identifiers are "artificial intelligence", "machine learning algorithm" and "deep neural network", respectively, where "machine learning algorithm" and "deep neural network" are both one of the types of "artificial intelligence", that is, the top-level classification layer of the hierarchical classification structure is "artificial intelligence", and the bottom-level classification layer is "machine learning algorithm" and "deep neural network", and "machine learning algorithm" and "deep neural network" are respectively used as leaf nodes to reconstruct the index, that is, "machine learning algorithm" and "deep neural network" are respectively used as a branch index, so that when the subsequent search content points to "machine learning algorithm" and "deep neural network" for retrieval, the information the user wants can be directly matched in the branch index corresponding to "machine learning algorithm" or "deep neural network".

[0042] In this step, after obtaining the search content, the system will detect whether the keywords or phrases contained in the search content match a branch index in the tree index structure, that is, determine whether the search content contains keywords or phrases that can represent the category, and then determine the corresponding branch index based on the category information.

[0043] If the search content contains keywords or phrases that can represent the category and hits a leaf node, it means that there is a branch index constructed with the minimum category as the leaf node in the tree index structure. Then, the target branch index can be determined through the leaf node. At this time, the required information can be retrieved from the target branch index as the search result, that is, step 103 is executed.

[0044] If the search content does not contain keywords or phrases that can represent the category, or the search content contains keywords or phrases that can represent the category but does not hit any leaf node, it means that there is no branch index constructed with the minimum category as a leaf node in the tree index structure. In order to return the most comprehensive search results possible, all leaf nodes can be traversed at this time, that is, the required information can be retrieved separately in all branch indexes and used as the search results together, that is, step 104 is executed.

[0045] 103. Determine, in the target branch index, a search result matching the search content according to the first search logic.

[0046] The target branch index is the hit branch index, and the first search logic is used to represent the matching and recalling of search results in the classification path corresponding to the target branch index.

[0047] In this step, after determining the target branch index, the system will perform a more detailed search under the target branch index. Since the tree index structure in this embodiment is constructed based on the minimum category, each leaf node corresponds to a specific classification path as a branch index, and each classification path usually contains one or more feature information, which is used to characterize the data instance corresponding to the minimum category. For example, if the data instance is an academic document, the corresponding feature information includes but is not limited to the title, author, abstract, keywords, etc. Therefore, the first search logic is used to determine the search result, that is, to select the feature information that matches the search content in the classification path corresponding to the target branch index as the search result. In this process, the relevance of each feature information to the search content can be determined by calculating semantic similarity, cosine similarity or Euclidean distance, and quantitative filtering can be performed on this basis to obtain the required feature information as the search result. Exemplarily, the requirement for the quantitative filtering can be to retain the feature information ranked in the top N in terms of relevance. By matching and recalling the search results in the classification path corresponding to the target branch index, when searching for highly homogeneous content, it is possible to directly locate the minimum category corresponding to the search content and determine the search results in the minimum category, which can effectively narrow the search scope and improve the accuracy and efficiency of the search. Wherein, N is a positive integer. In some examples, N is 2; in other examples, N is 3, 4 or 5, and this embodiment does not limit N.

[0048] 104. Determine, in all branch indexes of the tree index structure, a search result matching the search content according to the second search logic.

[0049] The second search logic is used to represent the matching and recall of search results in the classification paths corresponding to all branch indexes respectively.

[0050] In this step, since there is no corresponding branch index for the search content, the system can perform a more extensive and comprehensive search in the entire tree index structure. Combined with the description in step 103, the second search logic is used to determine the search results, that is, to select the feature information matching the search content in the classification paths corresponding to all branch indexes, and the feature information obtained in each branch path is used as the search result. Similarly, the relevance of each feature information corresponding to each branch index to the search content can be determined by calculating semantic similarity, cosine similarity or Euclidean distance, and quantitative filtering is performed on this basis, so as to obtain the required feature information in each branch index, and use it as the search result. By matching and recalling the search results in the classification paths corresponding to all branch indexes, when searching for highly homogenized content, it is possible to avoid information omissions caused by the limitation of a single path when there is no corresponding branch index, that is, to determine the search results in all minimum categories, thereby ensuring that as comprehensive as possible relevant information is obtained from different dimensions and levels, and improving the recall rate of the search.

[0051] Based on the above Figure 1 It can be seen from the implementation method that a retrieval method based on a tree structure provided in an embodiment of the present application is to receive and parse the user's retrieval request when retrieving the information required by the user, and after obtaining the retrieval content corresponding to the retrieval request, detect whether there is a hit branch index in the retrieval content in the tree index structure. The tree index structure is constructed based on the minimum category in the hierarchical classification structure of the original data, and the minimum category corresponds to the branch index one by one. If there is a hit branch index, it is used as the target branch index, and the retrieval result matching the retrieval content is determined in the target branch index according to the first retrieval logic, that is, the retrieval result is matched and recalled in the classification path corresponding to the target branch index. If there is no hit branch index, the retrieval result matching the retrieval content is determined in all branch indexes of the tree index structure according to the second retrieval logic, that is, the retrieval result is matched and recalled in the classification paths corresponding to all branch indexes respectively. The technical solution provided by this application can detect whether the search content hits the matching branch index, that is, determine whether there is a category corresponding to the search content in the tree index structure. When it hits, it can quickly locate the branch that is most likely to contain the search results, that is, determine and recall the matching search results in the matching category, narrowing the search scope and significantly improving the search efficiency and accuracy. When it misses, it can search in all branch indexes in parallel, that is, determine and recall the matching search results in all categories separately, which not only speeds up the search speed, but also ensures that no information is missed, especially when searching for highly homogenized content, it can provide more accurate and comprehensive result coverage and improve the recall rate of the search.

[0052] like Figure 2As shown, the preferred embodiment of the present application is in the above Figure 1 Based on the above, a detailed description is given of the process of tree-structure-based retrieval, wherein the specific steps include 201-208.

[0053] 201. Extract the category identifier from the metadata of the original data.

[0054] In this step, the original data can be source data in the storage system database or source data accessed externally. This embodiment does not limit this. Meta information refers to descriptive information about the data, which may be contained in file names, titles, tags, descriptions, classification fields, and other locations. Therefore, all category identifiers that can be used for classification can be collected from the meta information. Specifically, natural language processing techniques (such as stem extraction, entity recognition) or regular expression matching can be used to accurately extract category identifiers from the meta information. It should be noted that in order to ensure the accuracy and consistency of the category identifier, the collected meta information can be preprocessed before extracting the category identifier, including cleaning, normalization, and deduplication, and then the category identifier can be accurately extracted from the preprocessed meta information.

[0055] 202. Classify each category identifier, and construct a hierarchical classification structure corresponding to the original data according to the hierarchical affiliation corresponding to each category identifier after classification.

[0056] In this step, one or more classification criteria may be predefined. Specifically, the classification dimensions may be determined based on different data characteristics and business requirements in the original data, and specific classification rules may be defined for each classification dimension, thereby obtaining the classification criteria. For example, product classification may be performed according to dimensions such as function and brand, and may be divided into "office supplies" or "entertainment equipment" according to function. After determining the classification criteria, each category identifier may be classified according to its corresponding classification dimension and the classification rules of each classification dimension. Alternatively, a classification algorithm such as decision tree, hierarchical clustering, or K-means may be selected to classify each category identifier according to the data characteristics and classification requirements of the original data. This embodiment does not limit this. It should be noted that for data without a clear category, a default classification may be pre-set or manually modified by the user.

[0057] After obtaining each category identification after classification, the hierarchy can be defined manually, or a hierarchical relationship model can be constructed using a machine learning algorithm to define the hierarchy. The upper category and lower category of each category identification are determined according to the defined hierarchy and classification to obtain a clear hierarchical affiliation of each category identification. The broadest upper category in the hierarchical affiliation is defined as the upper classification layer, and the lower category belonging to the upper category is defined as the lower classification layer, and so on until the lower classification layer is defined to determine the position of each category identification in the hierarchical classification structure. At this time, each category identification is mapped to the corresponding position to obtain the hierarchical classification structure corresponding to the original data. Specifically, the hierarchical classification structure obtained in this embodiment includes one or more classification layers, each classification layer includes one or more category identifications, and any two adjacent classification layers include an upper classification layer and a lower classification layer. When the lower category identification of the lower classification layer belongs to the upper category identification of the upper classification layer, the category represented by the lower category is smaller than the category represented by the upper category identification.

[0058] For example, assuming that the original data is information about electronic products, the category identifiers extracted from it are: "electronic products", "mobile devices", "home appliances", "smart phones", "tablets", "TVs", "audios", and "smart watches". The levels determined according to the general classification method of electronic products are:

[0059] The top classification layer: "electronic products";

[0060] The next level of classification: “mobile devices” and “home appliances”;

[0061] The lowest level of classification: "Laptops", "Tablets", "TVs", "Speakers", and "Smartwatches".

[0062] The hierarchical classification structure is constructed as follows:

[0063] Electronic products (top level classification layer);

[0064] Mobile devices (the next level of classification);

[0065] Smartphones (the lowest classification level);

[0066] Tablets (lowest level of classification);

[0067] Smartwatch (lowest level of classification).

[0068] Household appliances (secondary classification level);

[0069] Television (the lowest classification level);

[0070] Audio (the lowest level of classification).

[0071] 203. Rebuild the index based on the lowest classification layer of the hierarchical classification structure in a tree structure to obtain a tree index structure.

[0072] Among them, the category represented by the category identifier corresponding to the lowest classification layer is the smallest category.

[0073] In this step, since the hierarchical classification structure is constructed according to the hierarchical affiliation of each category identifier, in order to ensure that each specific feature information can be accurately located at the end of the tree structure, so as to facilitate accurate retrieval. Specifically, the category identifiers corresponding to each lowest classification layer are used as leaf nodes to generate a tree structure; the classification path where each leaf node is located is obtained, and a branch index is established based on the classification path; the branch index is associated with the corresponding leaf node to obtain a tree index structure.

[0074] In one example, the construction process of the tree index structure is specifically as follows: according to the hierarchical structure of the hierarchical classification structure, a corresponding tree structure is generated, wherein the leaf node of the tree structure is the lowest classification layer of the hierarchical classification structure, and the category represented by the category identifier of the lowest classification layer is the smallest category; for the target lowest classification layer corresponding to any leaf node, the classification path from the highest classification layer to the target lowest classification layer in the hierarchical classification structure is obtained, and the classification path is used to represent or create a branch index; any leaf node and branch index are associated to generate a tree index structure.

[0075] It should be noted that, since the lowest classification layer in the identification hierarchical classification structure corresponds to the smallest category, and the smallest category is the most specific classification unit, an empty tree structure can be constructed, and each category identifier corresponding to the lowest classification layer is added to the tree structure as a leaf node. Since each category identifier has a clear classification and hierarchy, the complete path from the root node to each leaf node can be determined according to the classification and hierarchy, that is, the classification path of each leaf node. For example, the classification path of "smartphone" may be "electronic products->mobile devices->smartphone".

[0076] For each leaf node, a branch index is created according to its classification path. The branch index can be an array, list, or dictionary, which contains all node names from the root node to the leaf node. By associating each leaf node with its corresponding branch index, a tree index structure can be obtained. By directly associating the leaf node with all the feature information in the specific data instance of its corresponding smallest category, the feature information at the end of the tree structure can be accurately located, thereby improving the accuracy of retrieval.

[0077] 204. Check whether there is a category identifier in the search content.

[0078] As can be seen from step 101, the search content obtained by parsing the search request may contain keywords, phrases or more complex query statements, and the tree index structure is constructed based on the smallest category in the hierarchical classification structure of the original data, that is, in order to quickly confirm whether the search content can hit a certain branch index to ensure the scope and depth of the search, it can be detected whether the keywords or phrases in the search content contain specific category identifiers that can represent the category. That is, extract keywords and phrases in the search content, and use natural language processing technology, such as part-of-speech tagging, named entity recognition, etc., to identify keywords that may indicate classification or category, so as to identify the category identifiers that may be contained therein.

[0079] Since the branch index in the tree index structure corresponds to the minimum category one by one, if the category identifier is identified from the search content, and the category identifier is the branch index corresponding to a minimum category in the tree index structure, then step 205 is executed. If the category identifier is not identified from the search content or the identified category identifier is not the branch index corresponding to any minimum category in the tree index structure, then step 206 is executed.

[0080] It should be noted that when determining whether there is a hit branch index for a category identifier, specifically, the identified category identifier may be used as a query condition to search in the tree index structure to match the corresponding branch index.

[0081] 205. When there is a branch index matching the category identifier in the tree index structure, determine whether there is a hit branch index in the tree index structure for the search content.

[0082] In this step, if a branch index matching the category identifier is found in the tree index structure, it can be determined that the search content has a hit branch index in the tree index structure. At this time, the branch index can be used as the target branch index, and the search result matching the search content can be determined from the feature information corresponding to the target branch index, thereby improving the efficiency and accuracy of the search.

[0083] 206. Determine that no hit branch index exists for the search content in the tree index structure.

[0084] In this step, if no branch index matching the category identifier is found in the tree index structure, it can be determined that the search content does not have a hit branch index in the tree index structure. At this time, the search results matching the search content can be determined in parallel in the feature information corresponding to all branch indexes of the tree index structure. That is, all branch indexes of the tree index structure are searched to capture all possible related data, thereby achieving full-range search to avoid missing information.

[0085] 207. Determine, in the target branch index, a search result matching the search content according to the first search logic.

[0086] This step is combined with the description of step 103 in the above method, and the same content will not be repeated here. It should be noted that the specific execution process of determining the search results that match the search content in the target branch index according to the first search logic is: calculating the correlation between the search content and each feature information corresponding to the target branch index; sorting each feature information corresponding to the target branch index based on the correlation, and taking the feature information that meets the quantity filtering requirements in the sorted feature information corresponding to the target branch index as the valid result; recalling the valid results according to the classification path corresponding to the target branch index to obtain the search result.

[0087] In this step, keywords and phrases can be extracted from the search content, and corresponding feature information can be extracted from the data instance of the classification path corresponding to the target branch index, including but not limited to natural language processing technologies such as stem extraction, stop word filtering, and part-of-speech tagging. And use a suitable similarity measurement algorithm to calculate the correlation between the keywords or phrases in the search content and each feature information. The similarity measurement algorithm includes but is not limited to cosine similarity or Euclidean distance, etc., and deep learning models such as BERT and Transformer can also be used to calculate deeper semantic similarity. This embodiment does not limit this.

[0088] After obtaining the relevance of each feature information, the feature information is sorted in descending order based on the calculated relevance to ensure that the most relevant feature information is placed in front, and at the same time, a pre-set quantity filtering requirement is obtained, which is used to characterize the quantity limit of valid results, such as selecting the first N from the sorted feature information as valid results, where N is a positive integer. The sorted and filtered valid results are collected as a result set that has a clear classification path match and recall this time, and this result set is the search result. It should be noted that if multiple keywords or phrases are extracted from the search content, the valid results for each keyword or phrase need to be integrated to form the final search result. On this basis, the integrated valid results are sorted according to relevance and displayed to the user. In addition, additional contextual information, such as source, summary, etc., can be provided during the display to help users quickly judge the relevance of the search results.

[0089] 208. Determine, in all branch indexes of the tree index structure, a search result matching the search content according to the second search logic.

[0090] This step is combined with the description of step 104 in the above method, and the same content will not be repeated here. It should be noted that the specific execution process of determining the search results that match the search content according to the multi-way recall rule in all branch indexes of the tree index structure is: parallel calculation of the correlation between the search content and each feature information corresponding to each branch index; sorting the relevance of each feature information corresponding to each branch index based on the correlation, and taking the feature information that meets the quantity filtering requirements in the sorted feature information corresponding to each branch index as the valid result; recalling the valid results according to the classification path corresponding to each branch index to obtain the search results.

[0091] In this step, a parallel computing architecture can be designed in advance, and the computing tasks can be distributed to multiple processing units for simultaneous execution using multi-core CPUs, GPUs or distributed computing resources. That is, the relevance calculation tasks between the retrieval content and the feature information corresponding to each branch index are evenly distributed to different processing units according to the branch index. On each processing unit, the similarity measurement algorithm mentioned in step 207 is used to calculate the relevance of the retrieval content and each feature information corresponding to each branch index. When all processing units complete the calculation, the relevance of each feature information corresponding to each branch index is summarized, and the feature information corresponding to each branch index is sorted in descending order according to the calculated relevance. Similarly, the number filtering requirements mentioned in step 207 are used to obtain the valid results corresponding to each branch index. The sorted and filtered valid results are collected, and the valid results corresponding to each branch index are used as the result set that does not have a clear classification path this time and is matched and recalled with each classification path respectively. The result set is the retrieval result, which contains the retrieval results queried based on the retrieval request for each minimum category.

[0092] It should be noted that if multiple keywords or phrases are extracted from the search content, the valid results for each keyword or phrase need to be integrated to form the final search results. On this basis, the integrated valid results are sorted according to relevance and displayed to the user. And when displaying, additional contextual information, such as source, summary, etc., can be provided to help users quickly judge the relevance of the search results. It should be noted that when sorting the valid results according to relevance, they can be sorted separately according to the classification path, or they can be sorted globally without considering the classification path. This embodiment does not limit this.

[0093] In addition, in order to further improve the accuracy of the retrieval results, all feature information in the multi-way recall result set can also be globally sorted by relevance to ensure that the most relevant feature information is at the top of the result set. At the same time, duplicate feature information is removed to avoid redundancy in the final result. According to the final recall quantity requirement, the first M feature information is selected from the globally sorted result set as the final retrieval result. M is a positive integer.

[0094] Furthermore, in order to ensure that the search results corresponding to each minimum category are above a certain relevance standard with the user's search request during multi-path recall, thereby avoiding irrelevant information and ensuring the effectiveness of the search. The specific execution process of recalling the valid results according to the classification path corresponding to each branch index to obtain the search results is as follows: determine in turn whether the relevance of the valid results is higher than the preset relevance threshold; if not, delete the valid results with a relevance lower than or equal to the preset relevance threshold, and recall the remaining valid results according to the classification path corresponding to each branch index to obtain the search results; if so, recall the valid results according to the classification path corresponding to each branch index to obtain the search results.

[0095] In this step, the preset relevance threshold can be set according to historical data and business needs, or can be customized according to the user's search experience, which is not limited in this embodiment. By comparing the relevance of valid results with the preset relevance threshold one by one, the valid results that are lower than or equal to the preset relevance threshold are removed from the original result set, and the remaining valid results that are higher than the preset relevance threshold are recalled according to the classification path corresponding to each branch index, thereby ensuring that most of the truly relevant feature information can be included in the search results of this recall, while excluding most of the irrelevant feature information, to ensure the effectiveness of the search.

[0096] like Figure 5 As shown, when you need to retrieve the top1 result most relevant to feature A, refer to Figure 5 From the left part, we can see that in the prior art, in the process of single-dimensional recall, the semantic search and inverted index search are used to retrieve the three corresponding results of number 1, number 2 and number 3 related to feature A, and then the result is sorted by relevance: number 1> number 2> number 3. However, since this search only requires the most relevant top1 result, the search results of this search only include the result corresponding to number 1. Figure 5As can be seen from the middle right half, the tree structure provided in this embodiment has divided multiple branch indexes according to each category. Based on semantic retrieval and inverted index retrieval, three corresponding results of number 1, number 2 and number 3 related to feature A are obtained from category 1, category 2 and category 3 respectively. Although after precise sorting according to relevance, it is obtained that: number 1> number 2> number 3, but since number 1, number 2 and number 3 are the most relevant results to feature A in category 1, category 2 and category 3 respectively, the search results of this search will include the three corresponding results of number 1, number 2 and number 3 respectively. It can be seen that when searching for highly homogeneous feature A, the tree index structure adopted in this embodiment can find the most relevant results under the category containing feature A respectively, effectively eliminating the omission of information caused by the limitation of a single path, and ensuring that as comprehensive relevant information as possible is obtained from different dimensions and levels, thereby providing more accurate and comprehensive result coverage and improving the recall rate of retrieval.

[0097] Furthermore, as a response to the above Figure 1-2 The implementation of the method embodiment shown in the figure, the embodiment of the present application provides a retrieval device based on a tree structure, which is used to ensure that the retrieval results are comprehensive and accurate, so as to improve the retrieval recall rate. The embodiment of the device corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. Specifically, Figure 3 As shown, the device comprises:

[0098] The acquisition unit 31 is used to receive and analyze the search request of the user to obtain the search content corresponding to the search request;

[0099] A detection unit 32, used to detect whether the search content obtained by the acquisition unit 31 has a hit branch index in the tree index structure, wherein the tree index structure is constructed based on the smallest category in the hierarchical classification structure of the original data, and the smallest category corresponds to the branch index one by one;

[0100] A first processing unit 33 is configured to determine a search result matching the search content in the target branch index according to a search logic if the detection unit 32 detects that there is a hit branch index in the tree index structure of the search content, wherein the target branch index is the hit branch index, and the first search logic is configured to characterize matching and recalling the search result in the classification path corresponding to the target branch index;

[0101] The second processing unit 34 is used to determine the search results matching the search content according to the second search logic in all the branch indexes of the tree index structure if the detection unit 32 detects that there is no hit branch index for the search content in the tree index structure. The second search logic is used to represent the matching and recall of the search results in the classification paths corresponding to all the branch indexes respectively.

[0102] Further, such as Figure 4 As shown, the device also includes:

[0103] An extraction unit 35, configured to extract a category identifier from the meta information of the original data;

[0104] A first construction unit 36 ​​is used to classify the category identifiers obtained by the extraction unit 35, and to construct a hierarchical classification structure corresponding to the original data according to the hierarchical affiliation corresponding to the classified category identifiers;

[0105] The second construction unit 37 is used to rebuild the index in a tree structure based on the lowest classification layer of the hierarchical classification structure obtained by the first construction unit 36 ​​to obtain the tree index structure, and the category represented by the category identifier corresponding to the lowest classification layer is the smallest category.

[0106] Further, such as Figure 4 As shown, the second construction unit 37 includes:

[0107] A generating module 371, configured to use the category identifiers corresponding to the lowest classification layers as leaf nodes to generate the tree structure;

[0108] A construction module 372 is used to obtain the classification path where each leaf node obtained by the generation module 371 is located, and establish the branch index based on the classification path;

[0109] The association module 373 is used to associate the branch index obtained by the construction module 372 with the corresponding leaf node obtained by the generation module 371 to obtain the tree index structure.

[0110] Further, such as Figure 4 As shown, the detection unit 32 includes:

[0111] A detection module 321, used to detect whether there is category information in the search content;

[0112] A first determination module 322 is configured to determine that there is a hit branch index in the tree index structure for the search content if the detection module 321 detects that there is category information in the search content and there is a branch index matching the category information in the tree index structure;

[0113] The second determination module 323 is configured to determine that there is no hit branch index in the tree index structure for the search content if the detection module 321 detects that there is no category information in the search content.

[0114] Further, such as Figure 4 As shown, the first processing unit 33 includes:

[0115] A first calculation module 331 is used to calculate the correlation between the search content and each feature information corresponding to the target branch index;

[0116] A first processing module 332 is used to sort the relevance of each of the feature information corresponding to the target branch index based on the relevance obtained by the first calculation module 331, and take the feature information that meets the quantity filtering requirement among the sorted feature information corresponding to the target branch index as the valid result;

[0117] The first recall module 333 is used to recall the valid results obtained by the first processing module 332 according to the classification path corresponding to the target branch index to obtain the search result.

[0118] Further, such as Figure 4 As shown, the second processing unit 34 includes:

[0119] A second calculation module 341 is used to calculate in parallel the relevance between the search content and each feature information corresponding to each branch index;

[0120] The second processing module 342 is used to sort the relevance of each of the feature information corresponding to each of the branch indexes based on the relevance obtained by the second calculation module 341, and take the feature information that meets the quantity filtering requirement among the sorted feature information corresponding to each of the branch indexes as a valid result;

[0121] The second recall module 343 is used to recall the valid results respectively according to the classification path corresponding to each branch index to obtain the search result.

[0122] Further, such as Figure 4 As shown, the second recall module 343 includes:

[0123] The judging submodule 3431 is used to judge in turn whether the relevance of the valid results is higher than a preset relevance threshold;

[0124] The first recall submodule 3432 is configured to delete the valid results whose relevance is less than or equal to the preset relevance threshold if the judgment submodule 3431 judges in sequence that the relevance of the valid results is not all higher than the preset relevance threshold, and to recall the remaining valid results according to the classification paths corresponding to each of the branch indexes to obtain the search result;

[0125] The second recall submodule 3433 is used to recall the valid results respectively according to the classification paths corresponding to each branch index to obtain the retrieval results if the judgment submodule 3431 judges in turn that the relevance of the valid results is higher than the preset relevance threshold.

[0126] Furthermore, an embodiment of the present application further provides a storage medium, wherein the storage medium is used to store a computer program, wherein when the computer program is running, the device where the storage medium is located is controlled to execute the above Figure 1-2 The tree-structured retrieval method described in .

[0127] Furthermore, the embodiment of the present application also provides a processor, the processor is used to run a program, wherein the program executes the above Figure 1-2 The tree-structured retrieval method described in .

[0128] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0129] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, and do not represent the advantages and disadvantages of the embodiments.

[0130] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0131] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the application is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the application described herein, and the description of the specific language above is for the purpose of disclosing the best mode of implementation of the application.

[0132] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0138] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0139] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include transitory media such as modulated data signals and carrier waves.

[0140] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0141] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0142] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A tree-structure based retrieval method, characterized in that: The method comprises: Receive and analyze the user's search request to obtain the search content corresponding to the search request; Detecting whether there is a hit branch index in the search content in a tree index structure, wherein the tree index structure is constructed based on the smallest category in the hierarchical classification structure of the original data, and the smallest category corresponds to the branch index one by one; If so, determining the search result matching the search content in the target branch index according to the first search logic, the target branch index being the hit branch index, the first search logic being used to characterize matching and recalling the search result in the classification path corresponding to the target branch index; If not, then the search results matching the search content are determined in all branch indexes of the tree index structure according to a second search logic, where the second search logic is used to represent matching and recalling the search results in the classification paths corresponding to all branch indexes respectively.

2. The method according to claim 1, characterized in that Before receiving and parsing the user's search request, the method further includes: extracting a category identifier from the metadata of the original data; Classifying each category identifier, and constructing a hierarchical classification structure corresponding to the original data according to the hierarchical affiliation corresponding to each classified category identifier; The index is rebuilt in a tree structure based on the lowest classification layer of the hierarchical classification structure to obtain the tree index structure, and the category represented by the category identifier corresponding to the lowest classification layer is the smallest category.

3. The method according to claim 2, characterized in that Rebuilding the index in a tree structure based on the lowest classification layer of the hierarchical classification structure to obtain the tree index structure includes: The category identifiers corresponding to the lowest classification layers are used as leaf nodes to generate the tree structure; Obtaining the classification path where each leaf node is located, and establishing the branch index based on the classification path; The branch index is associated with the corresponding leaf node to obtain the tree index structure.

4. The method according to any one of claims 1 to 3, characterized in that Detecting whether the search content has a hit branch index in the tree index structure includes: Detecting whether there is a category identifier in the search content; If yes, when there is a branch index matching the category identifier in the tree index structure, it is determined that there is a hit branch index in the tree index structure for the search content; If not, it is determined that there is no hit branch index for the search content in the tree index structure.

5. The method according to any one of claims 1 to 3, characterized in that: Determining, in the target branch index, a search result matching the search content according to the first search logic, including: Calculating the correlation between the search content and each feature information corresponding to the target branch index; Sort the relevance of each of the feature information corresponding to the target branch index based on the relevance, and take the feature information that meets the quantity filtering requirement among the sorted feature information corresponding to the target branch index as the valid result; The valid results are recalled according to the classification path corresponding to the target branch index to obtain the retrieval result.

6. The method according to any one of claims 1 to 3, characterized in that: Determining a search result matching the search content in all branch indexes of the tree index structure according to a second search logic includes: Calculating in parallel the relevance between the search content and each feature information corresponding to each branch index; Based on the relevance, each of the feature information corresponding to each of the branch indexes is relevance-sorted, and the feature information that meets the quantity filtering requirement among the sorted feature information corresponding to each of the branch indexes is taken as a valid result; The valid results are respectively recalled according to the classification paths corresponding to each of the branch indexes to obtain the retrieval results.

7. The method according to claim 6, characterized in that Recalling the valid results respectively according to the classification paths corresponding to each of the branch indexes to obtain the search results, including: Determining in turn whether the relevance of the valid results is higher than a preset relevance threshold; If not, then deleting the valid results whose relevance is less than or equal to the preset relevance threshold, and respectively recalling the remaining valid results according to the classification paths corresponding to each of the branch indexes to obtain the retrieval result; If so, the valid results are respectively recalled according to the classification paths corresponding to each of the branch indexes to obtain the search results.

8. A retrieval device based on a tree structure, characterized in that: The device comprises: An acquisition unit, used to receive and analyze a user's search request to obtain search content corresponding to the search request; a detection unit, configured to detect whether the search content obtained by the acquisition unit has a hit branch index in a tree index structure, wherein the tree index structure is constructed based on the smallest category in the hierarchical classification structure of the original data, and the smallest category corresponds to the branch index one by one; A first processing unit is configured to determine, if the detection unit detects that the search content has a hit branch index in the tree index structure, a search result matching the search content in the target branch index according to a first search logic, wherein the target branch index is the hit branch index, and the first search logic is configured to characterize matching and recalling the search result in a classification path corresponding to the target branch index; A second processing unit is configured to determine, if the detection unit detects that there is no hit branch index for the search content in the tree index structure, a search result matching the search content in all branch indexes of the tree index structure according to a second search logic, wherein the second search logic is configured to represent and recall the search result in the classification paths corresponding to all the branch indexes respectively.

9. A storage medium, characterized in that: The storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the tree-structure-based retrieval method according to any one of claims 1 to 7.

10. A processor, characterized in that: The processor is used to run a program, wherein the program, when running, executes the tree-structure-based retrieval method according to any one of claims 1 to 7.

Images