A method, apparatus, electronic device, and storage medium for recommending retrieval strategies.

Abstract

This application relates to the field of data storage technology, and more particularly to a method, apparatus, electronic device, and storage medium for recommending retrieval strategies, to accurately select knowledge segmentation and knowledge retrieval strategies and improve the accuracy of retrieval results. It includes: for each paragraph in the original text, determining the distribution value of hot words in the paragraph based on the frequency of occurrence of multiple hot words in the original text and the text length of the paragraph; determining the dispersion of the distribution of multiple hot words based on the distribution distribution values, selecting a target segmentation strategy from multiple knowledge segmentation strategies to be selected based on the dispersion, and segmenting the original text into multiple knowledge blocks according to the target segmentation strategy; obtaining the semantic concentration of each knowledge block, and determining the target retrieval strategy based on the average of the obtained semantic concentrations and the semantic concentration of the retrieved text; the semantic concentration is used to reflect the information density carried by a unit of linguistic symbols.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a method, apparatus, electronic device, and storage medium for recommending a retrieval strategy. Background Technology

[0002] Knowledge segmentation and knowledge retrieval are the core of knowledge retrieval enhancement technology (RAG). When a user asks a question, the massive amount of information in the knowledge base first needs to be rationally segmented into semantically coherent fragments according to knowledge segmentation methods. The granularity and method (such as by sentence, by paragraph, by topic, or by utilizing semantic boundaries) directly affect the efficiency and accuracy of subsequent retrieval. Subsequently, based on the user's question, the most relevant knowledge content needs to be selected from these fragments using the chosen knowledge retrieval technology (such as keyword matching, vector similarity search, etc.) to achieve knowledge retrieval.

[0003] However, current methods for building and using RAG knowledge bases, whether for knowledge segmentation or retrieval, heavily rely on experience or trial-and-error, lacking universality. Therefore, existing technologies, when performing retrieval, suffer from two main problems: firstly, inappropriate segmentation and retrieval strategies can severely impair the efficiency of the knowledge base and affect the quality of search results returned to users; secondly, to achieve better search results, more complex segmentation and retrieval strategies or extensive parameter adjustments are often employed, significantly increasing the technical complexity and cost of building, maintaining, and using the knowledge base.

[0004] In summary, how to accurately select knowledge segmentation and retrieval strategies to improve the accuracy of search results is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method, apparatus, electronic device, and storage medium for recommending retrieval strategies, which can accurately select knowledge segmentation and knowledge retrieval strategies to improve the accuracy of retrieval results.

[0006] This application provides a method for recommending retrieval strategies, the method comprising:

[0007] For each paragraph in the original text, the hot word distribution value of the paragraph is determined based on the frequency of occurrence of multiple hot words in the original text and the text length of the paragraph;

[0008] After determining the degree of dispersion of the multiple hot words based on their distribution values, a target segmentation strategy is selected from multiple knowledge segmentation strategies to be selected based on the degree of dispersion, and the original text is segmented into multiple knowledge blocks based on the target segmentation strategy.

[0009] The semantic concentration of each knowledge block is obtained, and the target retrieval strategy is determined based on the average of the obtained semantic concentrations and the semantic concentration of the retrieved text; the semantic concentration is used to reflect the information density carried by a unit of language symbol.

[0010] This application provides a retrieval strategy recommendation device, the device comprising:

[0011] A determining unit is used to determine the hot word distribution value of each paragraph in the original text based on the occurrence frequency of multiple hot words in the original text and the text length of the paragraph.

[0012] The segmentation strategy selection unit is used to determine the degree of dispersion of the distribution of the multiple hot words based on the distribution values ​​of the multiple hot words, select a target segmentation strategy from the multiple knowledge segmentation strategies to be selected based on the degree of dispersion of the distribution, and segment the original text into multiple knowledge blocks according to the target segmentation strategy.

[0013] The retrieval strategy selection unit is used to obtain the semantic concentration of each knowledge block and determine the target retrieval strategy based on the average of the obtained semantic concentrations and the semantic concentration of the retrieval text; the semantic concentration is used to reflect the information density carried by a unit of language symbols.

[0014] Optionally, the determining unit is specifically used for:

[0015] Based on the word frequency information of the original text, the multiple hot words are determined;

[0016] The frequency of occurrence of each of the multiple hot words is normalized to obtain the frequency weight coefficient of each of the multiple hot words;

[0017] For each hot word, obtain the product of the frequency weight coefficient corresponding to the hot word and the number of times the hot word appears;

[0018] The ratio of the sum of multiple products to the text length is used as the hot word distribution value.

[0019] Optionally, the degree of dispersion of the distribution includes the degree of dispersion of hot word distribution; the segmentation strategy selection unit is specifically used for:

[0020] Determine the mean and standard deviation of the hot word distribution values;

[0021] The ratio of the mean of the hot word distribution to the standard deviation of the hot word distribution is used as the degree of dispersion of the hot word distribution.

[0022] Optionally, the degree of distribution dispersion also includes the degree of text length dispersion, and the segmentation strategy selection unit is specifically used for:

[0023] When the dispersion of the hot word distribution is less than a first preset threshold, the target segmentation strategy is determined to be a semantic segmentation strategy;

[0024] When the dispersion of the hot word distribution is greater than or equal to the first preset threshold, the text length dispersion of the original text is determined, and a target segmentation strategy is selected based on the text length dispersion.

[0025] Optionally, the segmentation strategy selection unit is specifically used for:

[0026] Determine the text length of each paragraph in the original text;

[0027] Determine the mean and standard deviation of text lengths for multiple text lengths;

[0028] The ratio of the mean text length to the standard deviation of the text length is used as the degree of dispersion of the text length.

[0029] The segmentation strategy selection unit is specifically used for:

[0030] When the text length dispersion is greater than the second preset threshold, the target segmentation strategy is determined to be a recursive segmentation strategy;

[0031] When the text length dispersion is less than or equal to the second preset threshold, the target segmentation strategy is determined to be a sliding window segmentation strategy.

[0032] Optionally, the retrieval strategy selection unit is specifically used to obtain the semantic concentration in the following manner:

[0033] For each hot word, the hot word information entropy is determined based on the frequency weight coefficient of the hot word; the hot word information entropy is used to reflect the degree of randomness in the distribution position of the hot word;

[0034] For each hot word, the term frequency-inverse document frequency is determined based on the number of times the hot word appears and the number of knowledge blocks.

[0035] The weighted average of the first sum and the second sum is taken as the semantic concentration, wherein the first sum is determined based on the sum of the information entropy of multiple hot words, and the second sum is determined based on the sum of multiple word frequencies and inverse document frequencies.

[0036] Optionally, the retrieval strategy selection unit is specifically used for:

[0037] Determine the standardized difference of the semantic concentration of the retrieved text relative to the mean of the plurality of semantic concentrations;

[0038] When the standardization difference is greater than a third preset threshold, the target retrieval strategy is determined to prioritize full-text retrieval results over vector retrieval results.

[0039] When the standardized difference equals the third preset threshold, the target retrieval strategy is determined to be: a hybrid retrieval strategy that combines the full-text retrieval strategy and the vector retrieval strategy;

[0040] When the standardization difference is less than the third preset threshold, the target retrieval strategy is determined to be the vector retrieval strategy.

[0041] An electronic device provided in this application includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the recommendation method for any of the above-described retrieval strategies.

[0042] This application provides a computer-readable storage medium including a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the recommendation method for any of the above-described retrieval strategies.

[0043] This application provides a computer program product, which includes a computer program stored in a computer-readable storage medium. When a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the recommendation method of any of the above-described retrieval strategies.

[0044] The beneficial effects of this application are as follows:

[0045] This application provides a method, apparatus, electronic device, and storage medium for recommending retrieval strategies. For each paragraph in the original text, this application determines the hot word distribution value of the paragraph based on the frequency of occurrence of multiple hot words in the original text and the text length of the paragraph. After determining the distribution dispersion of multiple hot words based on the distribution dispersion values, a target segmentation strategy is selected from multiple knowledge segmentation strategies to be selected based on the distribution dispersion, and the original text is segmented into multiple knowledge blocks according to the target segmentation strategy. Through this step, this application achieves the construction of a corresponding document knowledge distribution based on the hot word distribution characteristics of the original text and recommends a corresponding knowledge segmentation strategy. The semantic concentration of each knowledge block is obtained, and a target retrieval strategy is determined based on the average of the obtained multiple semantic concentrations and the semantic concentration of the retrieved text. In this step, this application dynamically adjusts the retrieval mode based on the semantic concentration of the retrieved text and recommends a suitable retrieval strategy.

[0046] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0047] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0048] Figure 1 A flowchart illustrating the implementation of a retrieval strategy recommendation method provided in this application embodiment;

[0049] Figure 2 A flowchart illustrating a method for recommending a retrieval strategy as provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of a retrieval strategy device according to an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the hardware structure of an electronic device using an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the hardware structure of a computing device according to an embodiment of this application. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0054] The design concept of the embodiments of this application is briefly introduced below:

[0055] Knowledge segmentation and knowledge retrieval are the core of knowledge retrieval enhancement technologies. Different knowledge segmentation and retrieval techniques will affect the final answer generation, but there are currently no clear guidelines for knowledge base construction and usage strategies, resulting in poor knowledge base performance or high usage complexity.

[0056] In existing technologies, during retrieval, neural network models can be used to fuse semantic and keyword features. Dense vectors and sparse weight vectors are extracted by segmenting text fragments, and then combined with a large language model to calculate the semantic features and keyword weights of the query statement, establishing a hierarchical index to achieve ranking and integration of retrieval results. However, in this method, the two vectors are calculated independently, lacking the ability to customize retrieval based on the semantic and key features of the query, and there is no method for knowledge block segmentation. Existing technologies can also vectorize the user-input search phrases, calculate the cosine similarity between the phrases and the knowledge base questions and keywords, and combine semantic matching and word frequency matching to optimize retrieval results. However, this method does not provide corresponding recommendation logic.

[0057] In view of this, embodiments of this application provide a method, apparatus, electronic device, and storage medium for recommending retrieval strategies. For each paragraph in the original text, this application determines the hot word distribution value of the paragraph based on the frequency of occurrence of multiple hot words in the original text and the text length of the paragraph; after determining the distribution dispersion of multiple hot words based on the distribution dispersion values, a target segmentation strategy is selected from multiple knowledge segmentation strategies to be selected based on the distribution dispersion, and the original text is segmented into multiple knowledge blocks according to the target segmentation strategy; through this step, this application achieves the construction of a corresponding document knowledge distribution based on the hot word distribution characteristics of the original text and recommends a corresponding knowledge segmentation strategy. The semantic concentration of each knowledge block is obtained, and a target retrieval strategy is determined based on the average of the obtained multiple semantic concentrations and the semantic concentration of the retrieved text. In this step, this application dynamically adjusts the retrieval mode based on the semantic concentration of the retrieved text and recommends a suitable retrieval strategy.

[0058] See Figure 1 The diagram shown is a flowchart of a retrieval strategy recommendation method provided in this application embodiment. The specific implementation process of this method includes S101~S103:

[0059] S101: For each paragraph in the original text, determine the hot word distribution value of the paragraph based on the frequency of occurrence of multiple hot words in the original text and the text length of the paragraph.

[0060] The retrieval strategy recommendation method of this application can be applied to a knowledge base containing multiple original texts, and the retrieval strategy recommendation method of this application is executed for each original text in the knowledge base.

[0061] The paragraphs in the original text were obtained by dividing the text into paragraphs based on the paragraph separator "\n".

[0062] In one optional implementation, when determining the hot word distribution value for each paragraph in an original text, this application implements the following specific method:

[0063] Step 1: Identify multiple hot words based on the word frequency information of the original text.

[0064] In this step, based on the jieba word segmentation model, each paragraph is processed cyclically, and word segmentation is performed on each paragraph to obtain meaningful words. Regular words, such as commas, periods, and words without meaning, are removed. The word frequency information for each paragraph is then calculated. n The word frequency information is denoted as Among them, word frequency information It contains multiple meaningful words, as well as the number of times each meaningful word appears.

[0065] Analyze the word frequency information of all paragraphs in the original text to obtain the word frequency distribution of the original text. f =[ ， ，…， ], N This represents the total number of paragraphs in the original text. Based on the word frequency information of the original text, all meaningful words are sorted in descending order of frequency, and the top K meaningful words are selected as hot words. The value of K is an empirical value, which can be 50, 60, etc.

[0066] Step 2: Normalize the occurrence counts of each of the multiple hot words to obtain the frequency weight coefficients of each hot word;

[0067] Based on the word frequency distribution of the original text, the word frequency information of each hot word in the original text is obtained. The word frequency information of the hot word reflects the number of times the hot word appears in the entire original text. Alternatively, the word frequency information of each hot word in the knowledge base can be obtained based on the word frequency distribution of each original text in the knowledge base. In this case, the word frequency information of the hot word reflects the number of times the hot word appears in the entire knowledge base.

[0068] Specifically, taking the word frequency information of hot words as reflecting the number of times the hot word appears in the entire original text as an example, for each hot word, the number of times the hot word appears is normalized to obtain the frequency weight coefficient of the hot word, which is calculated using the following formula 1:

[0069] (Formula 1)

[0070] in, This represents the word frequency information of the i-th hot word. This represents the frequency weight coefficient of the k-th hot word, and the frequency weight coefficient of the K hot words is denoted as... .

[0071] Step 3: For each hot word, obtain the product of the frequency weight coefficient corresponding to the hot word and the number of times the hot word appears; use the ratio of the sum of multiple products to the text length as the hot word distribution value.

[0072] Specifically, for paragraphs, the calculation is performed using the following formula 2. n Hot word distribution value :

[0073] (Formula 2)

[0074] in, It represents the number of times the k-th hot word appears. Let n be the text length of paragraph n, and let n be the text length of all paragraphs. The distribution values ​​of hot words in all paragraphs are denoted as follows: .

[0075] S102: After determining the distribution dispersion of multiple hot words based on their distribution values, select the target segmentation strategy from among the multiple knowledge segmentation strategies to be selected based on the distribution dispersion, and segment the original text into multiple knowledge blocks according to the target segmentation strategy.

[0076] In this application, the multiple knowledge segmentation strategies to be selected include: sliding window segmentation strategy, semantic segmentation strategy, and recursive segmentation strategy.

[0077] Among them, the sliding window segmentation strategy is a segmentation strategy based on a fixed-size window used to extract text segments from text sequence data. This method generates a series of overlapping or non-overlapping text subsequences by moving a fixed-length window across a text sequence in increments of a certain step size. Semantic segmentation strategies focus on understanding the meaning of the text and segmenting it accordingly. They consider not only lexical information but also context and semantic relationships to determine the optimal way to divide knowledge blocks. Recursive segmentation strategies recursively decompose the text by repeatedly applying the same rules or algorithms until a certain stopping condition is met.

[0078] Therefore, these three knowledge segmentation strategies each have their own characteristics and application scenarios. Thus, after obtaining the hot word distribution values, this application can determine the distribution of hot words throughout the entire original document or the entire knowledge base based on these values, and then select an appropriate knowledge segmentation strategy based on the distribution.

[0079] In one optional implementation, before determining the knowledge segmentation strategy, this application first uses the following steps to determine the dispersion of hot word distribution:

[0080] Determine the mean and standard deviation of the distribution of multiple hot words; use the ratio of the mean to the standard deviation as the degree of dispersion of the hot word distribution.

[0081] Specifically, the dispersion of hot word distribution can be determined using the following formula 3:

[0082] (Formula 3)

[0083] in, The degree of dispersion of the distribution of hot words for all hot words. The mean of the hot word distribution. The standard deviation of the hot word distribution.

[0084] in, The degree of dispersion of the hot words distribution of all hot words in the original text can be used as a basis for the subsequent knowledge segmentation strategy in this application to segment the original text. Alternatively, the distribution of hot words in the knowledge base can be discrete. In this case, the subsequent knowledge segmentation strategy in this application is used to segment the knowledge base.

[0085] by Taking the dispersion of hot word distribution of all hot words in the original text as an example, this application specifically adopts the following method to determine the target segmentation strategy of the original text:

[0086] (1) When the dispersion of hot word distribution is less than the first preset threshold, the target segmentation strategy is determined to be the semantic segmentation strategy.

[0087] Specifically, in this application, the first preset threshold is an empirical value, which can be set to 0.4. At this time, when... When the similarity is less than 0.4, it is assumed that the knowledge is evenly distributed in each paragraph. It is recommended to segment the knowledge using paragraph separators and determine the semantic segmentation strategy as the target segmentation strategy. The similarity between adjacent paragraphs is calculated, and paragraph merging is performed when the similarity is greater than 0.75.

[0088] The similarity between adjacent paragraphs can be calculated by using the cosine similarity of word frequency information between the two paragraphs. When the cosine similarity is greater than 0.75, the paragraphs are merged. Alternatively, the Accard similarity can be determined based on the intersection and union of the paragraphs. When the Accard similarity is greater than 0.75, the paragraphs are merged. Semantic similarity between paragraphs can also be calculated, and so on.

[0089] (2) When the dispersion of hot word distribution is greater than or equal to the first preset threshold, determine the text length dispersion of the original text and select the target segmentation strategy according to the text length dispersion.

[0090] Specifically, when If the value is ≥0.4, then the knowledge is considered to be distributed across paragraphs. In this case, the dispersion of text length can be further calculated to determine which paragraphs of text length are more dispersed in terms of knowledge.

[0091] In the above embodiments, this application preliminarily determines whether the distribution of knowledge in the text is uniform based on the degree of dispersion of hot word distribution, and determines the target segmentation strategy accordingly.

[0092] In one optional implementation, this application employs the following steps when calculating the text discreteness:

[0093] Determine the text length of each paragraph in the original text; determine the mean and standard deviation of the text lengths; use the ratio of the mean to the standard deviation as the text length dispersion.

[0094] Specifically, with Similarly, the degree of dispersion in text length can be determined using the following formula 4:

[0095] (Formula 4)

[0096] in, The degree of text length dispersion for all paragraphs, The average text length. This represents the standard deviation of the text length.

[0097] Because the above content is based on This introduction uses the dispersion of the distribution of all hot words in the original text as an example. Therefore, at this point... This also represents the degree of text length dispersion across all paragraphs of the original text. When considering the degree of dispersion of the distribution of all hot words in the knowledge base, then This also represents the degree of text length dispersion across all paragraphs in the knowledge base.

[0098] The following is based on Taking the text length dispersion of all paragraphs in the original text as an example, we will further introduce the knowledge segmentation selection strategy:

[0099] (3) When the text length dispersion is greater than the second preset threshold, the target segmentation strategy is determined to be a recursive segmentation strategy.

[0100] In this application, the second preset threshold is also an empirical value, which can be set to 0.4. At this time, >0.4, the overall text length distribution is discrete, and the knowledge distribution is also discrete ( >0.4), the target segmentation strategy is determined to be a recursive segmentation strategy, which segments long paragraphs into shorter text blocks to better capture knowledge.

[0101] (4) When the text length dispersion is less than or equal to the second preset threshold, the target segmentation strategy is determined to be the sliding window segmentation strategy.

[0102] Specifically, When the value is ≤0.4, the target segmentation strategy is determined to be the sliding window segmentation strategy.

[0103] In the above embodiments, based on the degree of dispersion of hot word distribution, this application further determines the degree of dispersion of text length, determines the comparison between the knowledge distribution and the text paragraph length distribution in the entire document, and determines the target segmentation strategy accordingly.

[0104] S103: Obtain the semantic concentration of each knowledge block, and determine the target retrieval strategy based on the average of the multiple semantic concentrations obtained and the semantic concentration of the retrieved text.

[0105] Semantic concentration is used to reflect the information density carried by a unit of linguistic symbols.

[0106] In this application, the semantic concentration of the knowledge block and the semantic concentration of the retrieved text are determined in the same way.

[0107] The following section uses the semantic concentration of retrieved text as an example to introduce the calculation method of semantic concentration in this application:

[0108] Step 1: For each hot word, determine the hot word information entropy based on the frequency weight coefficient of the hot word.

[0109] Among them, the hot word information entropy is used to reflect the randomness of the distribution of hot words. The more random the distribution of hot words, the greater the hot word information entropy; otherwise, the hot word information entropy is smaller.

[0110] Specifically, the information entropy of hot words is calculated using the following formula 5:

[0111] (Formula 5)

[0112] in, Let k be the hot word information entropy.

[0113] Step 2: For each hot word, determine the term frequency-inverse document frequency based on the number of times the hot word appears and the number of knowledge blocks.

[0114] Among them, Term Frequency-Inverse Document Frequency (TF-IDF) can reflect the discriminative power of hot words. If a hot word appears frequently in a specific article (i.e., high TF) and rarely appears in other articles (i.e., high IDF), then this hot word is considered to have good discriminative power.

[0115] Specifically, the term frequency minus the inverse document frequency is calculated using the following formula 6:

[0116] (Formula 6)

[0117] in, The term frequency minus the inverse document frequency of the hot word k. It is the frequency of hot word k in the search text j. This represents the total number of words in the searched text, where N represents the number of knowledge blocks. This indicates the number of times the hot word k appears in all knowledge blocks.

[0118] Specifically, since the above content is based on an original document, N represents the number of knowledge blocks in the original document.

[0119] It should be noted that when calculating the semantic concentration of retrieved text and knowledge blocks, the semantic concentration can be calculated based on hot words or directly based on words, as shown in Formula 5. It can also be the frequency weighting coefficient of words. In this case, in Formula 6... It should also be the frequency of the word's appearance in the search text / knowledge base. This represents the number of times a word appears in all knowledge blocks.

[0120] Step 3: Use the weighted average of the first and second sums as the semantic concentration.

[0121] The first sum is determined based on the sum of the information entropy of multiple hot words, and the second sum is determined based on the sum of the word frequency and inverse document frequency of multiple terms.

[0122] Specifically, the first sum is calculated using the following formula 7:

[0123] (Formula 7)

[0124] in, This is the first sum.

[0125] Specifically, the second sum is calculated using the following formula 8:

[0126] (Formula 8)

[0127] The semantic concentration of the retrieved text is a weighted sum of the first and second sums, calculated as shown in Formula 9 below:

[0128] (Formula 9)

[0129] in, To retrieve the semantic concentration of text j, α is a dynamic weighting coefficient. Since hot word information entropy and word frequency-inverse document frequency both evaluate semantic concentration from an independent perspective, α=0.5 is recommended.

[0130] Similarly, using the above method to calculate the semantic concentration of a knowledge block, the semantic concentration of knowledge block a is denoted as... For an original document containing A knowledge blocks, its overall semantic concentration can be denoted as: .

[0131] The search strategies to be selected in this application include vector search strategy, full-text search strategy, and hybrid search strategy.

[0132] Vector retrieval utilizes mathematical vectors to represent data and calculates the similarity or distance between data points. It can handle semantic relationships, context, and rich semantic information of the data, and is suitable for processing various data types such as images, audio, and video, providing more accurate and relevant search results.

[0133] Full-text search is used to find exact keyword or phrase matches. It mainly relies on keyword matching to provide search results and is suitable for simple queries and exact match scenarios. However, it cannot handle semantic relationships and complex data types.

[0134] A combination of hybrid search and keyword search.

[0135] Generally, the accuracy of search results obtained using a hybrid search strategy is higher than that obtained using vector search strategies or full-text search strategies alone. However, when using a hybrid search strategy, it is crucial to carefully configure the priority of each individual search strategy. This application considers that when the search text itself lacks a certain level of semantic meaning, the results obtained using a vector search strategy are generally unreliable, and in such cases, a full-text search strategy is relatively reliable.

[0136] Therefore, in one optional implementation, this application adopts the following steps when selecting a target retrieval strategy:

[0137] (1) Determine the standardized difference of the semantic concentration of the retrieved text relative to the mean of multiple semantic concentrations.

[0138] Specifically, the standardized differences can be calculated using the following formula 10:

[0139] (Formula 10)

[0140] Wherein, β is the standardized difference, which can also be called the reordering weight coefficient of hybrid retrieval, used to balance the priority of the retrieval results of full-text retrieval and the retrieval results of vector retrieval; for The mean, for Standard deviation

[0141] After obtaining the standardized differences, the target retrieval strategy, when balancing the retrieval results of full-text retrieval and vector retrieval, can be represented by the following formula 11:

[0142] (Formula 11)

[0143] Where r is the rearranged result of the mixed retrieval. Knowledge block score coefficient for full-text search It is the knowledge block score coefficient of vector retrieval.

[0144] (2) When the standardization difference is greater than the third preset threshold, the target retrieval strategy is determined to be that the priority of full-text retrieval results is higher than that of vector retrieval results.

[0145] Since standardized differences are used to dynamically adjust the priority of the retrieval strategy, the third preset threshold can be set to 0.5.

[0146] like This indicates that the reordering of the hybrid search is biased towards full-text search results, and the priority of full-text search results is higher than that of vector search results.

[0147] (3) When the standardization difference is equal to the third preset threshold, the target retrieval strategy is determined to be a hybrid retrieval strategy of full-text retrieval strategy and vector retrieval strategy.

[0148] like When the value is 0, it indicates that the search results from full-text search and vector search have the same importance.

[0149] (4) When the standardization difference is less than the third preset threshold, the target retrieval strategy is determined to be the vector retrieval strategy.

[0150] like This indicates that the rearrangement of the mixed search is biased towards vector search results, and the priority of full-text search results is lower than that of vector search results.

[0151] like Figure 2The flowchart shown is a method for recommending a retrieval strategy according to an embodiment of this application, specifically including S201~S210:

[0152] S201: Importing the original document;

[0153] S202: Extract hot words from the original document process and determine the degree of dispersion of the distribution;

[0154] S203: Construct the knowledge distribution of the original document based on the degree of dispersion of the distribution;

[0155] S204: Recommendations for knowledge segmentation strategies;

[0156] S205: Retrieve the object's query;

[0157] S206: Extract semantic concentration from the query;

[0158] S207: Determine the standardization differences based on the semantic concentration of the knowledge block and the semantic concentration of the query;

[0159] S208: Recommend search strategies based on standardization differences.

[0160] Based on the same inventive concept, embodiments of this application also provide a retrieval strategy recommendation device 300. For example... Figure 3 As shown, this is a schematic diagram of the structure of the retrieval strategy recommendation device 300, which may include:

[0161] The determining unit 301 is used to determine the hot word distribution value of each paragraph in the original text based on the number of occurrences of multiple hot words in the original text and the text length of the paragraph.

[0162] The segmentation strategy selection unit 302 is used to determine the distribution dispersion of multiple hot words based on the distribution values ​​of multiple hot words, select the target segmentation strategy from multiple knowledge segmentation strategies to be selected based on the distribution dispersion, and segment the original text into multiple knowledge blocks according to the target segmentation strategy.

[0163] The retrieval strategy selection unit 303 is used to obtain the semantic concentration of each knowledge block and determine the target retrieval strategy based on the average of the multiple semantic concentrations obtained and the semantic concentration of the retrieval text; the semantic concentration is used to reflect the information density carried by a unit of language symbol.

[0164] Optionally, the determining unit 301 is specifically used for:

[0165] Based on the word frequency information of the original text, identify multiple hot words;

[0166] The frequency of occurrence of each of the multiple hot words is normalized to obtain the frequency weight coefficient of each hot word.

[0167] For each hot word, obtain the product of the frequency weight coefficient corresponding to the hot word and the number of times the hot word appears;

[0168] The ratio of the sum of multiple products to the text length is used as the hot word distribution value.

[0169] Optionally, the degree of dispersion of the distribution includes the degree of dispersion of the hot word distribution; the segmentation strategy selection unit 302 is specifically used for:

[0170] Determine the mean and standard deviation of the distribution of multiple hot words;

[0171] The ratio of the mean of the hot word distribution to the standard deviation of the hot word distribution is used as the degree of dispersion of the hot word distribution.

[0172] Optionally, the degree of dispersion in the distribution also includes the degree of dispersion in the text length, and the segmentation strategy selection unit 302 is specifically used for:

[0173] When the dispersion of hot word distribution is less than the first preset threshold, the target segmentation strategy is determined to be a semantic segmentation strategy;

[0174] When the dispersion of hot word distribution is greater than or equal to the first preset threshold, the text length dispersion of the original text is determined, and the target segmentation strategy is selected based on the text length dispersion.

[0175] Optionally, the segmentation strategy selection unit 302 is specifically used for:

[0176] Determine the text length of each paragraph in the original text;

[0177] Determine the mean and standard deviation of text lengths for multiple text lengths;

[0178] The ratio of the mean text length to the standard deviation of text length is used as the degree of text length dispersion.

[0179] The segmentation strategy selection unit 302 is specifically used for:

[0180] When the text length dispersion is greater than the second preset threshold, the target segmentation strategy is determined to be a recursive segmentation strategy;

[0181] When the text length dispersion is less than or equal to the second preset threshold, the target segmentation strategy is determined to be the sliding window segmentation strategy.

[0182] Optionally, the retrieval strategy selection unit 303 is specifically used to obtain semantic concentration in the following manner:

[0183] For each hot word, the hot word information entropy is determined based on the frequency weight coefficient of the hot word; the hot word information entropy is used to reflect the degree of randomness in the distribution of hot words;

[0184] For each hot word, the term frequency-inverse document frequency is determined based on the number of times the hot word appears and the number of knowledge blocks.

[0185] The weighted average of the first and second sums is used as the semantic concentration, where the first sum is determined based on the sum of the information entropy of multiple hot words, and the second sum is determined based on the sum of multiple term frequencies and inverse document frequencies.

[0186] Optionally, the retrieval strategy selection unit 303 is specifically used for:

[0187] Determine the standardized difference of the semantic concentration of the retrieved text relative to the mean of multiple semantic concentrations;

[0188] When the standardization difference is greater than the third preset threshold, the target retrieval strategy is determined to prioritize full-text retrieval results over vector retrieval results.

[0189] When the standardization difference equals the third preset threshold, the target retrieval strategy is determined to be a hybrid retrieval strategy that combines full-text retrieval and vector retrieval strategies.

[0190] When the standardization difference is less than the third preset threshold, the target retrieval strategy is determined to be a vector retrieval strategy.

[0191] Based on the same inventive concept, this application also provides an electronic device. In this embodiment, the structure of the electronic device can be as follows: Figure 4 As shown, it includes a memory 401, a communication module 403, and one or more processors 402.

[0192] The memory 401 is used to store computer programs executed by the processor 402. The memory 401 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0193] Memory 401 may be volatile memory, such as random-access memory (RAM); memory 401 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 401 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 401 may be a combination of the above-described memories.

[0194] Processor 402 may include one or more central processing units (CPUs) or digital processing units, etc. Processor 402 is used to implement the recommended method of the above-mentioned retrieval strategy when calling the computer program stored in memory 401.

[0195] The communication module 403 is used to communicate with electronic devices and other servers.

[0196] This application embodiment does not limit the specific connection medium between the memory 401, communication module 403, and processor 402 described above. This application embodiment... Figure 4 The memory 401 and the processor 402 are connected via a bus 404, and the bus 404 is in Figure 4 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 404 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 4 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0197] The memory 401 stores a computer storage medium containing computer-executable instructions for implementing the retrieval strategy recommendation method of this application embodiment. The processor 402 executes the aforementioned retrieval strategy recommendation method. Based on the same inventive concept, this application embodiment provides a computer-readable storage medium, the computer program product including: computer program code, which, when run on a computer, causes the computer to execute any of the retrieval strategy recommendation methods discussed above. Since the principle of the problem solved by the aforementioned computer-readable storage medium is similar to the retrieval strategy recommendation method, the implementation of the aforementioned computer-readable storage medium can refer to the implementation of the method, and repeated details will not be elaborated further.

[0198] The following reference Figure 5 To describe a computing device 500 according to this embodiment of the present application. Figure 5 The computing device 500 is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0199] like Figure 5 The computing device 500 is manifested in the form of a general-purpose computing device. The components of the computing device 500 may include, but are not limited to: at least one processing unit 501, at least one storage unit 502, and a bus 503 connecting different system components (including storage unit 502 and processing unit 501).

[0200] Bus 503 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0201] Storage unit 502 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 521 and / or cache memory 522, and may further include read-only memory (ROM) 523.

[0202] Storage unit 502 may also include a program / utility 525 having a set (at least one) of program modules 524, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0203] The computing device 500 can also communicate with one or more external devices 504 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with the computing device 500, and / or with any device that enables the computing device 500 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via the input / output (I / O) interface 505. Furthermore, the computing device 500 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter 506. Figure 5 As shown, network adapter 506 communicates with other modules used in computing device 500 via bus 503. It should be understood that, although not shown in the figures, other hardware and / or software modules may be used in conjunction with computing device 500, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0204] This application also provides a computer program product. The methods in this application can be implemented, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in this application are executed, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, network equipment, user equipment, core network equipment, OAM, or other programmable devices.

[0205] A computer-readable storage medium can be an implementation of a computer program product. That is, this application embodiment also provides a computer-readable storage medium that includes a computer program, which, when executed by a processor, implements a recommendation method for any of the retrieval strategies described above.

[0206] The computer program or instructions may be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions may be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium may be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; or an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0207] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0208] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0209] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0210] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0211] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for recommending retrieval strategies, characterized in that, The method includes: For each paragraph in the original text, the hot word distribution value of the paragraph is determined based on the frequency of occurrence of multiple hot words in the original text and the text length of the paragraph; After determining the degree of dispersion of the multiple hot words based on their distribution values, a target segmentation strategy is selected from multiple knowledge segmentation strategies to be selected based on the degree of dispersion, and the original text is segmented into multiple knowledge blocks based on the target segmentation strategy. The semantic concentration of each knowledge block is obtained, and the target retrieval strategy is determined based on the average of the obtained semantic concentrations and the semantic concentration of the retrieved text; the semantic concentration is used to reflect the information density carried by a unit of language symbol.

2. The method as described in claim 1, characterized in that, The step of determining the hot word distribution value of a paragraph based on the frequency of occurrence of multiple hot words in the original text and the text length of the paragraph includes: Based on the word frequency information of the original text, the multiple hot words are determined; The frequency of occurrence of each of the multiple hot words is normalized to obtain the frequency weight coefficient of each of the multiple hot words; For each hot word, obtain the product of the frequency weight coefficient corresponding to the hot word and the number of times the hot word appears; The ratio of the sum of multiple products to the text length is used as the hot word distribution value.

3. The method as described in claim 1, characterized in that, The degree of dispersion of the distribution includes the degree of dispersion of hot word distribution; The step of determining the dispersion of the distribution of multiple hot words based on their distribution values ​​includes: Determine the mean and standard deviation of the hot word distribution values; The ratio of the mean of the hot word distribution to the standard deviation of the hot word distribution is used as the degree of dispersion of the hot word distribution.

4. The method as described in claim 1, characterized in that, The degree of dispersion of the distribution also includes the degree of dispersion of text length. The step of selecting a target segmentation strategy from among several knowledge segmentation strategies to be selected based on the degree of dispersion of the distribution includes: When the dispersion of the hot word distribution is less than a first preset threshold, the target segmentation strategy is determined to be a semantic segmentation strategy; When the dispersion of the hot word distribution is greater than or equal to the first preset threshold, the text length dispersion of the original text is determined, and a target segmentation strategy is selected based on the text length dispersion.

5. The method as described in claim 4, characterized in that, Determining the text length discreteness of the original text includes: Determine the text length of each paragraph in the original text; Determine the mean and standard deviation of text lengths for multiple text lengths; The ratio of the mean text length to the standard deviation of the text length is used as the degree of dispersion of the text length. The step of selecting the target segmentation strategy based on the text length dispersion includes: When the text length dispersion is greater than the second preset threshold, the target segmentation strategy is determined to be a recursive segmentation strategy; When the text length dispersion is less than or equal to the second preset threshold, the target segmentation strategy is determined to be a sliding window segmentation strategy.

6. The method as described in claim 1, characterized in that, The semantic concentration is obtained in the following way: For each hot word, the hot word information entropy is determined based on the frequency weight coefficient of the hot word; the hot word information entropy is used to reflect the degree of randomness in the distribution position of the hot word; For each hot word, the term frequency-inverse document frequency is determined based on the number of times the hot word appears and the number of knowledge blocks. The weighted average of the first sum and the second sum is taken as the semantic concentration, wherein the first sum is determined based on the sum of the information entropy of multiple hot words, and the second sum is determined based on the sum of multiple word frequencies and inverse document frequencies.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the target retrieval strategy based on the average of multiple semantic concentrations and the semantic concentration of the retrieved text includes: Determine the standardized difference of the semantic concentration of the retrieved text relative to the mean of the plurality of semantic concentrations; When the standardization difference is greater than a third preset threshold, the target retrieval strategy is determined to prioritize full-text retrieval results over vector retrieval results. When the standardized difference equals the third preset threshold, the target retrieval strategy is determined to be: a hybrid retrieval strategy that combines the full-text retrieval strategy and the vector retrieval strategy; When the standardization difference is less than the third preset threshold, the target retrieval strategy is determined to be the vector retrieval strategy.

8. A device for recommending retrieval strategies, characterized in that, The device includes: A determining unit is used to determine the hot word distribution value of each paragraph in the original text based on the occurrence frequency of multiple hot words in the original text and the text length of the paragraph. The segmentation strategy selection unit is used to determine the degree of dispersion of the distribution of the multiple hot words based on the distribution values ​​of the multiple hot words, select a target segmentation strategy from the multiple knowledge segmentation strategies to be selected based on the degree of dispersion of the distribution, and segment the original text into multiple knowledge blocks according to the target segmentation strategy. The retrieval strategy selection unit is used to obtain the semantic concentration of each knowledge block and determine the target retrieval strategy based on the average of the obtained semantic concentrations and the semantic concentration of the retrieval text; the semantic concentration is used to reflect the information density carried by a unit of language symbols.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 7.

11. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 7.

Images