Retrieval method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a retrieval method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. According to the similarity between the target retrieval statement and preset text blocks, a first text block is determined from the preset text blocks, and the preset text blocks are obtained by dividing a preset text according to multiple different division granularities; and generating a retrieval result based on the first text block. By applying the scheme provided by the embodiment of the invention, the accuracy of the retrieval result can be improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a retrieval method, device, electronic device and storage medium. Background Art

[0002] At present, with the development of Internet technology, people are increasingly acquiring knowledge and information through online searches in the fields of academic research, commercial applications or daily life. The online search system in the related art mainly relies on a single search method, such as simple semantic understanding based on keywords and then searching. The final search results may not satisfy the users and the accuracy of the search results is low. Summary of the invention

[0003] The purpose of the embodiments of the present invention is to provide a search method, device, electronic device and storage medium to improve the accuracy of search results. The specific technical solution is as follows: In a first aspect, an embodiment of the present application provides a retrieval method, the method comprising: Get the target search statement; Determining a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks, wherein the preset text blocks are obtained by dividing the preset text according to a plurality of different division granularities; A search result is generated based on the first text block.

[0004] In one embodiment of the present application, determining the first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks includes: Using a plurality of preset vectorization methods respectively, the target search sentence is vectorized and converted to obtain a plurality of search vectors; For each search vector, according to the similarity between the search vector and the target text block vector, a first text block vector is determined from the target text block vector, wherein the target text block vector is a text block vector obtained by processing the search vector in the same vectorization manner, and the text block vector is obtained by performing vectorization conversion on a preset text block in the plurality of preset vectorization manners respectively; The generating of the search result based on the first text block includes: A retrieval result is generated based on the first text block corresponding to the first text block vector.

[0005] In one embodiment of the present application, after obtaining the target search statement, the method further includes: Segmenting the target search sentence to obtain a target term; Determining a second text block including the target term from the preset text blocks; The generating of the search result based on the first text block includes: A search result is generated based on the first text block and the second text block.

[0006] In one embodiment of the present application, the determining of the second text block containing the target term from the preset text block includes: A first number of preset text blocks containing the most target terms are determined as second text blocks.

[0007] In one embodiment of the present application, for each preset term, there is an inverted index list corresponding to the preset term, wherein the inverted index list corresponding to the preset term is composed of identifiers of preset text blocks containing the preset term; The step of determining a second text block including the target term from the preset text block includes: It is determined that the inverted index list corresponding to the target term contains a preset text block corresponding to the identifier as the second text block.

[0008] In one embodiment of the present application, an inverted index list corresponding to a preset term is generated in the following manner: For each preset text block, segment the preset text block to obtain preset word items; For each preset term, a target preset text block containing the preset term is determined in the preset text blocks, and an inverted index list consisting of identifiers of the preset text blocks containing the preset term is generated.

[0009] In one embodiment of the present application, determining the first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks includes: A second number of preset text blocks having the highest similarity to the target search sentence are determined as the first text block.

[0010] In one embodiment of the present application, generating a search result based on the first text block includes: reordering the first text block according to the relevance between the target search sentence and the first text block; selecting a third text block from the first text blocks according to the reordered order; A search result is generated based on the third text block.

[0011] In one embodiment of the present application, the obtaining of the target search statement includes: Get the original search statement; Inputting the original search sentence into the sentence rewriting macromodel to obtain a target search sentence generated by rewriting the original search sentence by the sentence rewriting macromodel; The sentence rewriting large model is a large language model, which is used to replace characters of the original search sentence and / or add context information to the original search sentence according to the search sentence obtained in the history, and the context information is used to represent the search intent of the original search sentence.

[0012] In a second aspect, an embodiment of the present application provides a retrieval device, the device comprising: A sentence acquisition module, used for acquiring a target search sentence; A first text block determination module, configured to determine a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks, wherein the preset text blocks are obtained by dividing the preset text according to a plurality of different division granularities; A result generating module is used to generate a search result based on the first text block.

[0013] In one embodiment of the present application, the first text block determination module is specifically used to: Using a plurality of preset vectorization methods respectively, the target search sentence is vectorized and converted to obtain a plurality of search vectors; For each search vector, according to the similarity between the search vector and the target text block vector, a first text block vector is determined from the target text block vector, wherein the target text block vector is a text block vector obtained by processing the search vector in the same vectorization manner, and the text block vector is obtained by performing vectorization conversion on a preset text block in the plurality of preset vectorization manners respectively; Generating a search result based on the target text block includes: A retrieval result is generated based on the first text block corresponding to the first text block vector.

[0014] In one embodiment of the present application, the device further includes: A term acquisition module, used for segmenting the target search sentence to obtain a target term; A second text block determining module, configured to determine a second text block containing the target term from the preset text blocks; The result generation module is specifically used for: A search result is generated based on the first text block and the second text block.

[0015] In one embodiment of the present application, the second text block determination module is specifically used to: A first number of preset text blocks containing the most target terms are determined as second text blocks.

[0016] In one embodiment of the present application, for each preset term, there is an inverted index list corresponding to the preset term, wherein the inverted index list corresponding to the preset term is composed of identifiers of preset text blocks containing the preset term; The second text block determination module is specifically used to: It is determined that the inverted index list corresponding to the target term contains a preset text block corresponding to the identifier as the second text block.

[0017] In one embodiment of the present application, an inverted index list corresponding to a preset term is generated by the following modules: A word segmentation module is used to segment each preset text block to obtain preset word items; The list generation module is used to determine, for each preset term, a target preset text block containing the preset term in the preset text block, and generate an inverted index list composed of identifiers of the preset text blocks containing the preset term.

[0018] In one embodiment of the present application, the first text block determination module is specifically used to: A second number of preset text blocks having the highest similarity to the target search sentence are determined as the first text block.

[0019] In one embodiment of the present application, the result generation module is specifically used to: reordering the first text block according to the relevance between the target search sentence and the first text block; selecting a third text block from the first text blocks according to the reordered order; A search result is generated based on the third text block.

[0020] In one embodiment of the present application, the statement acquisition module is specifically used to: Get the original search statement; Inputting the original search sentence into the sentence rewriting macromodel to obtain a target search sentence generated by rewriting the original search sentence by the sentence rewriting macromodel; The sentence rewriting large model is a large language model, which is used to replace characters of the original search sentence and / or add context information to the original search sentence according to the search sentence obtained in the history, and the context information is used to represent the search intent of the original search sentence.

[0021] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement any method step in the first aspect when executing a program stored in the memory.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps in any one of the second aspects are implemented.

[0023] In a fifth aspect, an embodiment of the present application provides a computer program product, which prompts the processor to implement any method step of the first aspect.

[0024] Beneficial effects of the embodiments of the present invention: As can be seen from the above, in the solution provided by the embodiment of the present invention, after the target search sentence is obtained, the first text block is selected from the preset text blocks according to the similarity between the target search sentence and the preset text blocks. Since the preset text blocks are obtained by dividing the preset text according to different division granularities. For the same preset text, the preset text blocks obtained from the preset text according to different division granularities contain different amounts of information and different levels of detail of the contained information. Therefore, by comparing the target search sentence with different preset text blocks, a variety of first text blocks with different amounts of information and levels of detail can be obtained. This makes the obtained first text blocks richer, and the search results obtained based on the rich first text blocks more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0026] Figure 1 A schematic diagram of a flow chart of a first retrieval method provided in an embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a second retrieval method provided in an embodiment of the present invention; Figure 3 A schematic diagram of a flow chart of a third retrieval method provided in an embodiment of the present invention; Figure 4 A schematic diagram of a fourth retrieval method provided in an embodiment of the present invention; Figure 5 A schematic diagram of a fifth retrieval method provided in an embodiment of the present invention; Figure 6 A schematic diagram of the structure of a retrieval device provided by an embodiment of the present invention; Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field based on the present invention belong to the scope of protection of the present invention.

[0028] The accuracy of the search results of the network search system in the related art is low, which may make users dissatisfied with the search results.

[0029] In order to solve the above problems, the embodiments of the present invention provide a retrieval method, an apparatus, an electronic device and a storage medium, which are described in detail below.

[0030] First, a search method provided by an embodiment of the present invention is described.

[0031] See also Figure 1 , is a flow chart of a first retrieval method provided in an embodiment of the present invention. The method can be applied to electronic devices with computing capabilities, such as computers, servers, etc. The method includes the following steps S101 to S103.

[0032] S101: Obtain a target search statement.

[0033] The target search sentence reflects the query demand, and the electronic device searches for information according to the target search sentence. The target search sentence can be in the form of text such as a character, a word or a sentence. For example, in one case, the target search sentence is: "people" or a string of characters in other languages ​​with the same meaning, such as English: "people"; in another case, the target search sentence is: "universe" or a string of characters in other languages ​​with the same meaning, such as English: "universe"; in another case, the target search sentence is: "the number of countries in the world" or a sentence in other languages ​​with the same meaning, such as English: "Number of countries in theworld"; and so on.

[0034] The above-mentioned target search sentence can be read from the search text by the execution subject of the embodiment of the present application, or obtained from the outside by the execution subject of the embodiment of the present application. The target search sentence can be directly input from the outside, or other forms of data such as voice can be input, and the execution subject recognizes the data to obtain the target search sentence. The target search sentence obtained from the outside can be a sentence directly input by the user to the execution subject, or it can be a sentence sent by other electronic devices received by the execution subject. For example, the execution subject can be a server, and the target search sentence can be a sentence sent by the client to the server.

[0035] S102: Determine a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks.

[0036] The above-mentioned preset text blocks are obtained by dividing the preset text according to a plurality of different division granularities.

[0037] Specifically, the preset text may be a pre-collected variety of text resources, including text files in various formats, such as text files in .txt, .pdf, and .doc formats. Afterwards, the preset text is divided according to different division granularities. For example, the division granularity may include chapters, paragraphs, sentences, etc. from large to small. For the same preset text, the preset text is divided using a variety of different division granularities. For example, preset text 1 contains 3 chapters, and each chapter contains 3 paragraphs. After dividing preset text 1 with chapters and paragraphs as the division granularity, 3 preset text blocks of chapters and 9 preset text blocks of paragraphs can be obtained. The preset text blocks with large division granularity contain preset text blocks with small division granularity.

[0038] In the division process described above, the semantic integrity of the preset text blocks obtained after the division is guaranteed.

[0039] In one embodiment of the present application, the target search sentence can be vectorized to obtain a target search vector, and the preset text block can be vectorized in advance to obtain a text block vector, and the cosine distance between the target search vector and the text block vector is calculated as the similarity between the target search sentence and the preset text block. The larger the cosine distance between the target search vector and the text block vector, the higher the similarity between the target search sentence and the preset text block.

[0040] Specifically, if a search vector is recorded as , and record a text block vector as , then for the cosine distance between the two , the calculation formula is as follows: ; After calculating the cosine distance between each text block vector and the search vector, the set of text block vectors sorted by cosine distance can be recorded as S e , S e = ,in, The function returns the index value of the sorted array elements.

[0041] Alternatively, other similarity calculation methods in related technologies may be used, such as calculating Manhattan distance and Pearson correlation coefficient as the similarity between the target search sentence and the preset text block, which is not limited here.

[0042] In addition, since a higher similarity indicates a higher degree of association between the preset text block and the target search statement, selecting a preset text block with a high degree of association with the target search statement as the first text block can make the search results generated based on the first text block more strongly associated with the target search statement, and the obtained search results are more accurate.

[0043] Therefore, in one embodiment, a preset text block with a similarity greater than a preset similarity may be selected as the first text block.

[0044] In another embodiment, step S102 may be implemented by following step A below.

[0045] Step A: Determine a second number of preset text blocks having the highest similarity to the target search sentence as the first text block.

[0046] In one embodiment of the present application, the preset text blocks may be sorted according to similarity, so that the second number of preset text blocks at the top of the sort are selected as the first text block.

[0047] S103: Generate search results based on the first text block.

[0048] In one case, after obtaining the first text block, the first text block may be sorted in descending order of similarity between the first text block and the target search sentence, and the sorted first text block is used as the search result.

[0049] In another case, the target search sentence is used as a control sentence, and the first text block is used as background knowledge, which are input into the large language model to control the large language model to implement the search process indicated by the target search sentence based on the background knowledge and output the search result.

[0050] As can be seen from the above, in the solution provided by the embodiment of the present invention, after the target search sentence is obtained, the first text block is selected from the preset text blocks according to the similarity between the target search sentence and the preset text blocks. Since the preset text blocks are obtained by dividing the preset text according to different division granularities. For the same preset text, the preset text blocks obtained from the preset text according to different division granularities contain different amounts of information and different levels of detail of the contained information. Therefore, by comparing the target search sentence with different preset text blocks, a variety of first text blocks with different amounts of information and levels of detail can be obtained. This makes the obtained first text blocks richer, and the search results obtained based on the rich first text blocks more accurate.

[0051] See also Figure 2 , is a flow chart of the second retrieval method provided in the embodiment of the present application, which is similar to the aforementioned Figure 1 Compared with the illustrated embodiment, the above step S102 can be implemented by the following steps S102A-S102B, and the above step S103 can be implemented by the following steps S103A.

[0052] S102A: Using a plurality of preset vectorization methods respectively, the target search sentence is vectorized and converted to obtain a plurality of search vectors.

[0053] In one embodiment of the present application, a vectorized model may be used to process a target search statement to generate a search vector.

[0054] For the above target search sentence, multiple different vectorization models can be used to process the target search sentence. For example, for a target search sentence, two vectorization models are used: BGE (Beijing Academy of Artificial Intelligence General Embedding, general semantic vector model) and BCE (Bilingual and Crosslingual Embedding, bilingual and cross-lingual semantic representation algorithm model), respectively, to process the target search sentence. Finally, two search vectors can be obtained.

[0055] Of course, as needed, three or four different vectorization methods can be used to process each target search sentence, and so on. Using different vectorization models to vectorize the same target search sentence at the same time can obtain rich vectorization results based on the unique algorithms and feature extraction methods of different vectorization models.

[0056] S102B: For each search vector, according to the similarity between the search vector and the target text block vector, determine a first text block vector from the target text block vectors.

[0057] The target text block vector is a text block vector obtained by processing the search vector in the same vectorization manner, and the text block vector is obtained by vectorizing the preset text block using the multiple preset vectorization manners.

[0058] In one embodiment of the present application, the above-mentioned multiple preset vectorization methods can be used in advance to process each preset text block to obtain multiple text block vectors.

[0059] For example, there are preset text blocks .

[0060] The preset text block can be vectorized according to the following formula: ; in, It is a vectorized method The text vector obtained after vectorization processing, i is the number of the preset text block, Indicates vectorization processing. After vectorization processing, a vector library can be constructed .

[0061] That is, for each vectorization method, there are retrieval vectors and text block vectors obtained by processing using the vectorization method in the embodiment of the present application. When calculating the similarity, for each vectorization method, the similarity between the retrieval vector and the text block vector obtained by processing using the vectorization method is calculated, and then the first text block vector targeted by the vectorization method is determined. That is, for different vectorization methods, different first text block vectors can be determined in the embodiment of the present application, and the text blocks corresponding to the first text block vectors determined for different vectorization methods can be the same or different.

[0062] Specifically, the method of calculating the similarity can be found in the above description, which will not be repeated here.

[0063] S103A: Generate a search result based on the first text block corresponding to the first text block vector.

[0064] In the embodiment of the present application, the union of text blocks corresponding to the first text block vectors determined in different vectorization methods may be used as the first text block.

[0065] After obtaining the first text block, the method of generating the search results can refer to the above description of step S103, which will not be repeated here.

[0066] As can be seen from the above, the scheme provided in the embodiment of the present application uses different vectorization methods to vectorize the target search sentence and the preset text block to obtain the search vector and the text block vector. The same target search sentence and text block vector are vectorized using different vectorization methods at the same time. Based on the unique algorithms and feature extraction methods of the different vectorization methods, the different search vectors obtained contain different features, and the different text block vectors obtained also contain different features. By calculating the similarity of the search vector and the text block vector with different features respectively, and then selecting the first text block vector according to the similarity, the selected first text block vector can be made richer, thereby improving the richness of the obtained first text block and making the generated search results more accurate.

[0067] See also Figure 3 , is a flow chart of the third retrieval method provided in the embodiment of the present application, which is similar to the aforementioned Figure 1 Compared with the embodiment shown, the step S101 includes the following steps S104 to S105, and the step S103 can be implemented by the following step S103B.

[0068] S104: Segment the target search sentence to obtain target terms.

[0069] After obtaining the target search sentence, the target search sentence is segmented, and the obtained words are the target terms. For example, if the target search sentence is: "the longitude and latitude of city A", the target terms obtained after segmentation are: city A, longitude and latitude. For another example, if the target search sentence is: "the weather of city B", the target terms obtained after segmentation are: city B, weather.

[0070] The specific word segmentation rules required for word segmentation of the target search sentence can be set according to the needs. Specifically, the target search sentence can be segmented using relevant word segmentation tools. For example, the word segmentation rule can be the maximum positive matching method, that is, scanning the target search sentence from left to right, taking the longest possible string each time to match with the words in the preset dictionary, if the match is successful, the string is segmented as a word; if the match is unsuccessful, the last character of the string is removed, and the remaining string continues to be matched until the match is successful or the string is segmented into single words.

[0071] For another example, the word segmentation tool used may be pkuseg or HanLP.

[0072] S105: Determine a second text block including the target term from the preset text blocks.

[0073] After the target term is obtained in step S104, a search is performed in the preset text blocks based on the target term to retrieve a second text block containing the target term.

[0074] Specifically, for each target term, it can be confirmed in each preset text block whether the target term is included. A corresponding confirmation parameter value can be set for each preset text block. At the beginning of each search based on the target search sentence, the initial value of the confirmation parameter value is 0. Whenever a preset text block is confirmed to contain the target term, the confirmation parameter value of the preset text block is increased by 1.

[0075] For example, if there are two target terms: target term t1 and target term t2, when searching based on target term t1, it is found that preset text block D1 and preset text block D2 both contain target term t1, then the confirmation parameter value of preset text block D1 changes from 0 to 1, and the confirmation parameter value of preset text block D2 changes from 0 to 1.

[0076] When searching based on the target term t2, it is found that the preset text block D1 and the preset text block D3 both contain the target term t2, then the confirmation parameter value of the preset text block D1 changes from 1 to 2, and the confirmation parameter value of the preset text block D3 changes from 0 to 1.

[0077] That is to say, for a preset text block, the corresponding confirmation parameter value is equal to the number of target terms contained in the preset text block. In the above example, the confirmation parameter value of the preset text block D1 is 2, the confirmation parameter value of the preset text block D2 is 1, and the confirmation parameter value of the preset text block D3 is 1.

[0078] Then, the preset text block whose confirmation parameter value is greater than 0 may be determined as the second text block.

[0079] In addition, since the preset text block containing more target terms has a higher degree of association with the target search statement, a preset text block containing more target terms can be selected as the second text block, so that the second text block has a higher degree of association with the target search statement, thereby making the search results generated based on the second text block more accurate.

[0080] Therefore, in one embodiment, a preset text block including a number of target terms greater than a preset number may be selected as the second text block.

[0081] In another embodiment, step S105 may be implemented by following step B below.

[0082] Step B: Determine a first number of preset text blocks containing the most target terms as the second text block.

[0083] In one embodiment of the present application, the preset text blocks may be sorted in descending order according to the number of target terms contained therein, so that the first number of preset text blocks at the top of the sort are selected as the second text block.

[0084] As shown above, the embodiment of the present application can count the confirmation parameter value for each preset text block. The above confirmation parameter value can represent the number of target terms contained in the preset text block, so the preset text blocks can be sorted in descending order based on the confirmation parameter value.

[0085] In yet another embodiment, for each preset term, there is an inverted index list corresponding to the preset term, wherein the inverted index list corresponding to the preset term is composed of a preset text block containing the preset term.

[0086] For example, if the default text blocks containing term t1 are D1, D3, D4, D 12 , D 13 , then the inverted index list of term t1 can be expressed as .

[0087] When there is an inverted index list corresponding to each preset term, step S105 can be implemented through step C.

[0088] Step C: Determine that the inverted index list corresponding to the target term contains a preset text block corresponding to the identifier as the second text block.

[0089] Since the identifier included in the inverted index list corresponding to the target term is the identifier of the preset text block including the target term, the second text block can be determined according to the predetermined inverted index list.

[0090] For example, if the inverted index list of target term t2 is , then the preset text blocks determined are D2, D3, D4, D 10 .

[0091] In addition, the number of times that the identifier of a second text block appears in the inverted index list corresponding to the target term is the number of target terms included in the second text block. Therefore, the number of target terms included in the second text block can be determined according to the inverted index list, and thus the second text block can be determined according to the number of target terms included in the preset text block.

[0092] For example, there are three target terms: target term t1, target term t2, target term t3, and the inverted index list of target term t1 is From the inverted index list, it can be determined that the preset text blocks containing the target term t1 are D1, D3, D4, D 12, D 13 . It indicates that the number of target terms contained in the preset text block D1 is currently accumulated to 1, the number of target terms contained in the preset text block D3 is currently accumulated to 1, the number of target terms contained in the preset text block D4 is currently accumulated to 1, and the preset text block D 12 The number of target terms currently included is 1, and the preset text block D 13 The number of included target terms currently accumulated is 1.

[0093] The inverted index list of target term t2 is From the inverted index list, it can be determined that the preset text blocks containing the target term t2 are D2, D3, D4, D 10 At this time, the number of target terms contained in the preset text block D1 is currently accumulated to 1, the number of target terms contained in the preset text block D2 is currently accumulated to 1, the number of target terms contained in the preset text block D3 is currently accumulated to 2, the number of target terms contained in the preset text block D4 is currently accumulated to 2, and the preset text block D 10 The number of target terms currently included is 1, and the preset text block D 12 The number of target terms currently included is 1, and the preset text block D 13 The number of included target terms currently accumulated is 1.

[0094] The inverted index list of target term t3 is From the inverted index list, it can be determined that the preset text blocks containing the target term t3 are D4, D 16 At this time, the number of target terms contained in the preset text block D1 is currently accumulated to 1, the number of target terms contained in the preset text block D2 is currently accumulated to 1, the number of target terms contained in the preset text block D3 is currently accumulated to 2, the number of target terms contained in the preset text block D4 is currently accumulated to 3, and the preset text block D 10 The number of target terms currently included is 1, and the preset text block D 12 The number of target terms currently included is 1, and the preset text block D 13 The number of target terms currently included is 1, and the preset text block D 16 The number of included target terms currently accumulated is 1.

[0095] Finally, it can be learned that the number of target terms contained in each preset text block can be determined.

[0096] Specifically, the process of determining the second text block according to the number of target terms included in the preset text block can be referred to the above description, which will not be repeated here.

[0097] In one embodiment of the present application, an inverted index list corresponding to a preset term may be generated through steps D to E, which will not be described in detail here.

[0098] S103B: Generate search results based on the first text block and the second text block.

[0099] In one embodiment of the present application, the search results can be generated in a manner similar to the description of step S103 above, with the only difference being that in step S103B, when generating the search results, not only the first text block is used but also the second text block is added, and the processing method for the second text block is the same as the processing method for the first text block, which will not be repeated here.

[0100] In another embodiment of the present application, the first text block and the second text block may be merged first, and then the search result may be generated.

[0101] Specifically, a fusion algorithm can be used to process multiple result sets to generate a unified and more optimized result set. The relevant fusion algorithms that can be used include weighted average method, Borda Count method, Rank Aggregation Methods, etc. One result set contains the first text block, and the other result set contains the second text block.

[0102] In one embodiment, the first text block and the second text block are processed using an RRF (Reciprocal Rank Fusion) algorithm. t , and record the result set containing the second text block as S e , using the RRF algorithm to fuse S t With S e , and get the fusion result set ,in, The fusion results are obtained for each text block contained in the first text block and the second text block.

[0103] The fusion result can be directly used as the retrieval result. Alternatively, the target retrieval sentence can be used as the control sentence and the fusion result can be used as the background knowledge and input into the large language model to control the large language model to implement the retrieval process indicated by the target retrieval sentence based on the background knowledge and output the retrieval result.

[0104] In addition, it should be noted that step S104 to step S105 can be executed in parallel with step S102, thereby improving the retrieval efficiency.

[0105] As can be seen from the above, in addition to searching the first text block, the solution provided in the embodiment of the present application also searches for the second text block containing the target term in the target search sentence, and combines the first text block and the second text block to jointly generate the search result, so that the information based on which the search result is generated is richer and the search result obtained is more accurate.

[0106] In one embodiment of the present application, an inverted index list corresponding to a preset term may be generated through steps D to E.

[0107] Step D: For each preset text block, segment the preset text block to obtain preset terms.

[0108] For each preset text block, the preset text block is segmented to obtain a series of preset terms: t1, t2, ..., t k Specifically, the method of segmenting the preset text block is the same as the method of segmenting the target search sentence in step S104.

[0109] Step E: for each preset term, determine a target preset text block containing the preset term in the preset text blocks, and generate an inverted index list consisting of identifiers of the preset text blocks containing the preset term.

[0110] For each preset term determined in step D, the preset text block containing the preset term in each preset text block is determined as the target preset text block. In addition, a list corresponding to the preset term is established to record the identifiers of all preset text blocks containing the preset term, thereby obtaining an inverted index list.

[0111] See also Figure 4 , is a flow chart of the fourth retrieval method provided in the embodiment of the present application, which is similar to the aforementioned Figure 1 Compared with the embodiment shown, the above step S103 can be implemented by the following steps S103C to S103E.

[0112] S103C: Reorder the first text block according to the relevance between the target search sentence and the first text block.

[0113] In one embodiment of the present application, the first text blocks may be reordered according to the relevance from low to high or from high to low.

[0114] In addition, it should be noted that if the search result is finally generated based on the first text block and the second text block, the first text block and the second text block can be reordered together.

[0115] In one embodiment of the present application, the above-mentioned relevance degree can be represented by the similarity between the target search sentence and the text block, and the reordering can be performed by sorting according to the similarity.

[0116] In another embodiment of the present application, a re-ranking model can be used for re-ranking. The re-ranking model takes the target search sentence and the text block as input, outputs a relevance score, and the relevance score reflects the relevance between the text block and the target search sentence, and ranks the text blocks according to the relevance score to obtain a re-ranking result.

[0117] The large re-ranking model can be optimized using cross-entropy loss, and the output similarity score is not limited to a specific range and can even be negative. This flexibility enables the large re-ranking model to more accurately evaluate the relevance between a text block and a target retrieval sentence.

[0118] The implementation of the re-ranking model can be implemented using a cross encoder or a dual encoder based on deep learning. In one embodiment, a BGE-reranker is used to re-rank the text blocks. Specifically, if Represents a large model for reordering, which collects text blocks And the target search sentence is input into the re-ranking model to generate the re-ranking result .

[0119] S103D: Select a third text block from the first text block according to the reordered order.

[0120] In one embodiment of the present application, according to the re-sorted order, a third number of first text blocks with the highest relevance may be selected as the third text block.

[0121] If the order after reordering is that the text blocks with higher relevance are ranked higher, then the third number of text blocks at the front of the reordered text blocks are selected as the third text block.

[0122] If the order after reordering is that the text blocks with higher relevance are ranked lower, then the third number of text blocks at the end of the reordered text blocks are selected as the third text block.

[0123] S103E: Generate search results based on the third text block.

[0124] Specifically, the method of implementing step S103E is similar to the method of implementing step S103 in the previous text, with the only difference being that the search result is generated based on the third text block in step S103E, which will not be described in detail here.

[0125] As can be seen from the above, in the solution provided by the embodiment of the present invention, the preset text blocks are reordered in fine granularity according to the relevance between the search statement and the preset text blocks, and then the third text block is selected according to the reordered order. The selected third text block is related to the relevance of the target search statement. This can further improve the search quality and increase the user's satisfaction with the search results.

[0126] In another embodiment of the present application, step S101 is implemented by the following steps F-G.

[0127] Step F: Obtain the original search statement.

[0128] The above original search statement is a search statement directly obtained by the execution subject of the embodiment of the present application.

[0129] The above-mentioned original search sentence can be directly read from the search text by the execution subject of the embodiment of the present application, or directly obtained from the outside by the execution subject of the embodiment of the present application. The original search sentence can be directly input from the outside, or other forms of data such as voice can be input, and the execution subject recognizes the data to obtain the original search sentence. The original search sentence obtained from the outside can be a sentence directly input by the user to the execution subject, or it can be a sentence sent by other electronic devices received by the execution subject. For example, the execution subject can be a server, and the original search sentence can be a sentence sent by the client to the server.

[0130] Step G: Input the original search sentence into the sentence rewriting macromodel to obtain the target search sentence generated after the sentence rewriting macromodel rewrites the original search sentence.

[0131] Among them, the above sentence rewriting large model is a large language model, which is used to replace characters of the above original search sentence and / or add context information to the above original search sentence according to the search sentence obtained in the history, and the above context information is used to represent the search intent of the above original search sentence.

[0132] In order to more accurately respond to the user's query needs, the original search sentence input by the user can be appropriately rewritten. Specifically, the original search sentence can be input into the sentence rewriting model, and the sentence rewriting model can rewrite the original search sentence.

[0133] In one case, in an embodiment of the present application, the received multiple consecutive original search sentences can be continuously input into the sentence rewriting model. The sentence rewriting model can learn the context information of the multiple consecutive original search sentences and perform semantic understanding of the multiple consecutive original search sentences. After obtaining the newly input original search sentence, the sentence rewriting model can replace the pronouns in the original search sentence with the objects they actually refer to based on the context information of the original search sentence and other original search sentences obtained historically, the semantic understanding of the original search sentence, etc.

[0134] For example, in one example, the original search sentence is: "Li Shimin is a very powerful person. What is his power?" The sentence rewriting model can replace "he" in the original search sentence with "Li Shimin" based on contextual information, semantic understanding, etc., so the target search sentence obtained after rewriting is: "Li Shimin is a very powerful person. What is Li Shimin's power?"

[0135] In another example, the first original search sentence is: "What is the weather like in District C today?", and the second original search sentence is: "Is there a traffic jam here?". Then, the sentence rewriting model can replace "here" in the second original search sentence with "District C" based on context information, semantic understanding, etc., so the target search sentence obtained after rewriting is: "Is there a traffic jam in District C?" In another case, if the original search sentence contains content describing time, the content describing time is replaced with a specific date, or a 12-hour or 24-hour time. For example, if the original search sentence is: "How is the weather today?", and today is July 22, 2024, the sentence rewriting model can rewrite the original search sentence and generate a target search sentence: "How is the weather on July 22, 2024?"

[0136] In another case, the sentence rewriting model adds contextual information to the original search sentence to express the query intent based on the semantic understanding of the original search sentence. For example, if the original search sentence input by the user is: "Vehicle restrictions in D City on July 22, and restrictions on July 23", then according to the semantic understanding, "July 23" refers to "D City on July 23", and "Restrictions on July 23" refers to "Vehicle restrictions on July 23", so the original search sentence can be added with contextual information for query rewriting, and the generated target search sentence is: "Vehicle restrictions in D City on July 22, and vehicle restrictions in D City on July 23", among which "D City" before "July 23" and "Vehicle" after "July 23" are the added contextual information.

[0137] In one case, the sentence rewriting model can add context information to the newly obtained original search sentence based on the original search sentence obtained in the past. For example, if the original query sentence obtained in the past is: "The weather in E City", and the newly obtained original search sentence is: "Tomorrow's weather", then according to the previous original query sentence, it can be inferred that "tomorrow" refers to "Tomorrow in E City", so the target search sentence generated after rewriting the original search sentence can be: "Tomorrow's weather in E City".

[0138] Of course, the situation where the sentence rewriting model rewrites the original search sentence to generate a target search sentence is not limited to the above example. In the scenario of the previous example, the target search sentence actually output by the sentence rewriting model may also be different from the target search sentence in the previous example.

[0139] As can be seen from the above, the solution provided by the embodiment of the present application can rewrite the original search statement to generate a target search statement, so that the target search statement can more clearly point to a specific object or concept, and can more accurately reflect the search intention, thereby improving the accuracy and efficiency of the search.

[0140] See also Figure 5 , which is a flow chart of the fifth retrieval method provided in an embodiment of the present application.

[0141] The process consists of two steps, Figure 5 The left side of the dotted line is the preprocessing process, and the processing object is the preset document. Document preprocessing refers to dividing the preset document into preset text blocks and performing word segmentation on the preset text blocks. Vectorization refers to performing vectorization processing on the preset text blocks. Building an inverted index refers to building an inverted index list corresponding to the preset terms.

[0142] Figure 5 The right side of the dotted line is the retrieval process. The processing object is the original query statement. Input processing refers to rewriting the original query statement into the target query statement. Retrieval refers to retrieving the first text block and the second text block. Reordering refers to reordering the first text block and the second text block.

[0143] Corresponding to the aforementioned retrieval method, an embodiment of the present invention further provides a retrieval device.

[0144] See also Figure 6 , is a schematic diagram of the structure of a retrieval device provided by an embodiment of the present invention, the device is applied to an electronic device with computing capabilities, and the above device includes: A sentence acquisition module 601 is used to acquire a target search sentence; A first text block determination module 602, configured to determine a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks, wherein the preset text blocks are obtained by dividing the preset text according to a plurality of different division granularities; The result generating module 603 is used to generate a search result based on the first text block.

[0145] As can be seen from the above, in the solution provided by the embodiment of the present invention, after the target search sentence is obtained, the first text block is selected from the preset text blocks according to the similarity between the target search sentence and the preset text blocks. Since the preset text blocks are obtained by dividing the preset text according to different division granularities. For the same preset text, the preset text blocks obtained from the preset text according to different division granularities contain different amounts of information and different levels of detail of the contained information. Therefore, by comparing the target search sentence with different preset text blocks, a variety of first text blocks with different amounts of information and levels of detail can be obtained. This makes the obtained first text blocks richer, and the search results obtained based on the rich first text blocks more accurate.

[0146] In one embodiment of the present application, the first text block determination module 602 is specifically configured to: Using a plurality of preset vectorization methods respectively, the target search sentence is vectorized and converted to obtain a plurality of search vectors; For each search vector, according to the similarity between the search vector and the target text block vector, a first text block vector is determined from the target text block vector, wherein the target text block vector is a text block vector obtained by processing the search vector in the same vectorization manner, and the text block vector is obtained by performing vectorization conversion on a preset text block in the plurality of preset vectorization manners respectively; Generating a search result based on the target text block includes: A retrieval result is generated based on the first text block corresponding to the first text block vector.

[0147] As can be seen from the above, the scheme provided in the embodiment of the present application uses different vectorization methods to vectorize the target search sentence and the preset text block to obtain the search vector and the text block vector. The same target search sentence and text block vector are vectorized using different vectorization methods at the same time. Based on the unique algorithms and feature extraction methods of the different vectorization methods, the different search vectors obtained contain different features, and the different text block vectors obtained also contain different features. By calculating the similarity of the search vector and the text block vector with different features respectively, and then selecting the first text block vector according to the similarity, the selected first text block vector can be made richer, thereby improving the richness of the obtained first text block and making the generated search results more accurate.

[0148] In one embodiment of the present application, the device further includes: A term acquisition module, used for segmenting the target search sentence to obtain a target term; A second text block determining module, configured to determine a second text block containing the target term from the preset text blocks; The result generation module is specifically used for: A search result is generated based on the first text block and the second text block.

[0149] As can be seen from the above, in addition to searching the first text block, the solution provided in the embodiment of the present application also searches for the second text block containing the target term in the target search sentence, and combines the first text block and the second text block to jointly generate the search result, so that the information based on which the search result is generated is richer and the search result obtained is more accurate.

[0150] In one embodiment of the present application, the second text block determination module is specifically used to: A first number of preset text blocks containing the most target terms are determined as second text blocks.

[0151] In one embodiment of the present application, for each preset term, there is an inverted index list corresponding to the preset term, wherein the inverted index list corresponding to the preset term is composed of identifiers of preset text blocks containing the preset term; The second text block determination module is specifically used to: It is determined that the inverted index list corresponding to the target term contains a preset text block corresponding to the identifier as the second text block.

[0152] In one embodiment of the present application, an inverted index list corresponding to a preset term is generated by the following modules: A word segmentation module is used to segment each preset text block to obtain preset word items; The list generation module is used to determine, for each preset term, a target preset text block containing the preset term in the preset text block, and generate an inverted index list composed of identifiers of the preset text blocks containing the preset term.

[0153] In one embodiment of the present application, the first text block determination module 602 is specifically configured to: A second number of preset text blocks having the highest similarity to the target search sentence are determined as the first text block.

[0154] In one embodiment of the present application, the result generation module 603 is specifically used to: reordering the first text block according to the relevance between the target search sentence and the first text block; selecting a third text block from the first text blocks according to the reordered order; A search result is generated based on the third text block.

[0155] As can be seen from the above, in the solution provided by the embodiment of the present invention, the preset text blocks are reordered in fine granularity according to the relevance between the search statement and the preset text blocks, and then the third text block is selected according to the reordered order. The selected third text block is related to the relevance of the target search statement. This can further improve the search quality and increase the user's satisfaction with the search results.

[0156] In one embodiment of the present application, the statement acquisition module 601 is specifically used to: Get the original search statement; Inputting the original search sentence into the sentence rewriting macromodel to obtain a target search sentence generated by rewriting the original search sentence by the sentence rewriting macromodel; The sentence rewriting large model is a large language model, which is used to replace characters of the original search sentence and / or add context information to the original search sentence according to the search sentence obtained in the history, and the context information is used to represent the search intent of the original search sentence.

[0157] As can be seen from the above, the solution provided by the embodiment of the present application can rewrite the original search statement to generate a target search statement, so that the target search statement can more clearly point to a specific object or concept, and can more accurately reflect the search intention, thereby improving the accuracy and efficiency of the search.

[0158] See also Figure 7, is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, including a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704; Memory 703, used for storing computer programs; The processor 701 is used to implement any step of the aforementioned retrieval method when executing the program stored in the memory 703.

[0159] As can be seen from the above, in the solution provided by the embodiment of the present invention, after the target search sentence is obtained, the first text block is selected from the preset text blocks according to the similarity between the target search sentence and the preset text blocks. Since the preset text blocks are obtained by dividing the preset text according to different division granularities. For the same preset text, the preset text blocks obtained from the preset text according to different division granularities contain different amounts of information and different levels of detail of the contained information. Therefore, by comparing the target search sentence with different preset text blocks, a variety of first text blocks with different amounts of information and levels of detail can be obtained. This makes the obtained first text blocks richer, and the search results obtained based on the rich first text blocks more accurate.

[0160] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0161] The communication interface is used for communication between the above terminal and other devices.

[0162] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0163] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0164] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned retrieval methods are implemented.

[0165] When the computer program stored in the computer-readable storage medium provided by the embodiment of the present invention is used for retrieval, after the target retrieval statement is obtained, the first text block is selected from the preset text blocks according to the similarity between the target retrieval statement and the preset text blocks. Since the preset text blocks are obtained by dividing the preset text according to different division granularities. For the same preset text, the preset text blocks obtained from the preset text according to different division granularities contain different amounts of information and different levels of detail of the contained information. Therefore, by comparing the target retrieval statement with different preset text blocks, a variety of first text blocks with different amounts of information and levels of detail can be obtained. This makes the obtained first text blocks richer, and the retrieval results obtained based on the rich first text blocks more accurate.

[0166] In another embodiment of the present invention, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute any of the retrieval methods in the above embodiments.

[0167] When the computer program product provided by the embodiment of the present invention is used for retrieval, after obtaining the target retrieval statement, the first text block is selected from the preset text blocks according to the similarity between the target retrieval statement and the preset text blocks. Since the preset text blocks are obtained by dividing the preset text according to different division granularities. For the same preset text, the preset text blocks obtained from the preset text according to different division granularities contain different amounts of information and different levels of detail of the contained information. Therefore, by comparing the target retrieval statement with different preset text blocks, a variety of first text blocks with different amounts of information and levels of detail can be obtained. This makes the obtained first text blocks richer, and the retrieval results obtained based on the rich first text blocks more accurate.

[0168] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0169] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0170] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0171] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. A search method, characterized in that: The method comprises: Get the target search statement; Determining a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks, wherein the preset text blocks are obtained by dividing the preset text according to a plurality of different division granularities; A search result is generated based on the first text block.

2. The method according to claim 1, characterized in that The step of determining a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks includes: Using a plurality of preset vectorization methods respectively, the target search sentence is vectorized and converted to obtain a plurality of search vectors; For each search vector, according to the similarity between the search vector and the target text block vector, a first text block vector is determined from the target text block vector, wherein the target text block vector is a text block vector obtained by processing the search vector in the same vectorization manner, and the text block vector is obtained by performing vectorization conversion on a preset text block in the plurality of preset vectorization manners respectively; The generating of the search result based on the first text block includes: A retrieval result is generated based on the first text block corresponding to the first text block vector.

3. The method according to claim 1, characterized in that After obtaining the target search statement, the method further includes: Segmenting the target search sentence to obtain a target term; Determining a second text block including the target term from the preset text blocks; The generating of the search result based on the first text block includes: A search result is generated based on the first text block and the second text block.

4. The method according to claim 3, characterized in that The step of determining a second text block including the target term from the preset text block includes: A first number of preset text blocks containing the most target terms are determined as second text blocks.

5. The method according to claim 3, characterized in that: For each preset term, there is an inverted index list corresponding to the preset term, wherein the inverted index list corresponding to the preset term is composed of identifiers of preset text blocks containing the preset term; The step of determining a second text block including the target term from the preset text block includes: It is determined that the inverted index list corresponding to the target term contains a preset text block corresponding to the identifier as the second text block.

6. The method according to claim 5, characterized in that Generate the inverted index list corresponding to the preset terms in the following way: For each preset text block, segment the preset text block to obtain preset word items; For each preset term, a target preset text block containing the preset term is determined in the preset text blocks, and an inverted index list consisting of identifiers of the preset text blocks containing the preset term is generated.

7. The method according to any one of claims 1 to 6, characterized in that The step of determining a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks includes: A second number of preset text blocks having the highest similarity to the target search sentence are determined as the first text block.

8. The method according to any one of claims 1 to 6, characterized in that The generating of the search result based on the first text block includes: reordering the first text block according to the relevance between the target search sentence and the first text block; selecting a third text block from the first text blocks according to the reordered order; A search result is generated based on the third text block.

9. The method according to any one of claims 1 to 6, characterized in that: The obtaining of the target search statement includes: Get the original search statement; Inputting the original search sentence into the sentence rewriting macromodel to obtain a target search sentence generated by rewriting the original search sentence by the sentence rewriting macromodel; The sentence rewriting large model is a large language model, which is used to replace characters of the original search sentence and / or add context information to the original search sentence according to the search sentence obtained in the history, and the context information is used to represent the search intent of the original search sentence.

10. A search device, characterized in that: The device comprises: A sentence acquisition module, used for acquiring a target search sentence; A first text block determination module, configured to determine a first text block from the preset text blocks according to the similarity between the target search sentence and the preset text blocks, wherein the preset text blocks are obtained by dividing the preset text according to a plurality of different division granularities; A result generating module is used to generate a search result based on the first text block.

11. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor is used to implement any method step in claims 1-9 when executing a program stored in a memory.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps in any one of claims 1 to 9 are implemented.

Images