Text retrieval method, apparatus, device, and storage medium

Abstract

The present disclosure relates to a text retrieval method, device, equipment and storage medium, the method comprising: in response to a text retrieval request, performing feature extraction on the text retrieval request to obtain a first feature vector; performing keyword matching on the text in a text library based on a target keyword corresponding to the text retrieval request to obtain a first text and obtain a second feature vector of the first text; calculating a first similarity and a second similarity between the feature vector of the text and the first feature vector and the second feature vector for the text in the text library; sorting the text according to the comprehensive similarity determined by the first similarity and the second similarity corresponding to the text, and determining a second text based on the sorting result. The present disclosure can determine the similarity between the text and the first text matched successfully by the text retrieval request and the keyword, so that the text meeting the user's demand but not containing the keyword corresponding to the retrieval request can also be used as the retrieval result, and the matching degree of the retrieval result and the user's demand is improved.

Description

Technical Field

[0001] This disclosure relates to the field of text processing technology, and in particular to a text retrieval method, apparatus, device, and storage medium. Background Technology

[0002] Enterprise knowledge bases often contain a large number of documents with a wealth of useful information. When searching for a specific document, current technologies typically employ keyword matching methods for document retrieval, such as searching for documents containing keywords in a question. However, in reality, the keywords in the user's original search query may not perfectly match the documents in the enterprise knowledge base, resulting in the inability to retrieve documents that meet the user's needs. Therefore, improving the matching degree between search results and user requirements is a technical problem that needs to be solved. Summary of the Invention

[0003] To address the aforementioned technical problems, this disclosure provides a text retrieval method, apparatus, device, and storage medium.

[0004] A first aspect of this disclosure provides a text retrieval method, the method comprising:

[0005] In response to a text retrieval request, feature extraction is performed on the text retrieval request to obtain a first feature vector of the text retrieval request;

[0006] Based on the target keywords corresponding to the text retrieval request, keyword matching is performed with the text in the text library. The first text that matches the target keywords is then filtered from the text library, and the second feature vector of the first text is obtained.

[0007] For the text in the text library, calculate the first similarity and the second similarity between the feature vector of the text and the first feature vector and the second feature vector, respectively;

[0008] The texts in the text library are sorted according to their comprehensive similarity to obtain a first sorting result, and a second text is determined based on the first sorting result, wherein the comprehensive similarity is determined based on the first similarity and the second similarity of the text.

[0009] A second aspect of this disclosure provides a text retrieval device, the device comprising:

[0010] The first extraction module is used to extract features from the text retrieval request in response to the text retrieval request, and obtain the first feature vector of the text retrieval request.

[0011] The matching module is used to perform keyword matching between the target keyword corresponding to the text retrieval request and the text in the text library, filter out the first text that matches the target keyword from the text library, and obtain the second feature vector of the first text;

[0012] The first calculation module is used to calculate, for the text in the text library, the first similarity and the second similarity between the feature vector of the text and the first feature vector and the second feature vector, respectively;

[0013] The determination module is used to sort the texts in the text library according to the comprehensive similarity of the texts to obtain a first sorting result, and determine a second text based on the first sorting result, wherein the comprehensive similarity is determined based on the first similarity and the second similarity of the texts.

[0014] A third aspect of this disclosure provides a computer device including a memory and a processor, and a computer program, wherein the memory stores the computer program, and when the computer program is executed by the processor, it implements the text retrieval method of the first aspect described above.

[0015] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the text retrieval method of the first aspect described above.

[0016] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0017] In the text retrieval method, apparatus, device, and storage medium provided in this disclosure embodiment, in response to a text retrieval request, feature extraction is performed on the text retrieval request to obtain a first feature vector of the text retrieval request. Based on the target keywords corresponding to the text retrieval request, keyword matching is performed with the text in the text library. A first text matching the target keywords is selected from the text library, and a second feature vector of the first text is obtained. For the text in the text library, a first similarity and a second similarity are calculated between the feature vector of the text and the first and second feature vectors, respectively. The texts in the text library are sorted according to the comprehensive similarity of the texts to obtain a first sorting result. A second text is determined based on the first sorting result. The comprehensive similarity is determined based on the first and second similarity of the texts. After the first text is obtained by matching the target keywords corresponding to the text retrieval request, the feature vector of the text is compared with the first feature vector of the text retrieval request and the second feature vector of the first text, respectively. The text with the higher comprehensive similarity is determined as the final retrieval result. Through the above-mentioned broader retrieval method, texts that meet the user's retrieval needs but do not directly contain the keywords in the text retrieval request can also be detected and presented to the user as the final retrieval result. Meanwhile, since keywords account for a smaller proportion in long documents, the document's feature vector contains more interfering features. Directly matching the document's feature vector with the feature vector of the document retrieval request for similarity results is easily affected by other interfering keywords in the long document. However, the retrieval method that combines keyword matching and similarity calculation can highlight keyword features and output more comprehensive retrieval results, with a higher degree of matching between the retrieval results and user needs. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0019] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a text retrieval method provided in an embodiment of this disclosure;

[0021] Figure 2 This is a flowchart of a method for filtering a first text provided in an embodiment of this disclosure;

[0022] Figure 3This is a flowchart of a method for determining target keywords provided in an embodiment of this disclosure;

[0023] Figure 4 This is a flowchart illustrating a method for determining the comprehensive similarity of texts according to an embodiment of this disclosure;

[0024] Figure 5 This is a flowchart of a text preprocessing method provided in an embodiment of this disclosure;

[0025] Figure 6 This is a schematic diagram of the structure of a text retrieval device provided in an embodiment of this disclosure;

[0026] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0027] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0028] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0029] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0030] Figure 1 This is a flowchart illustrating a text retrieval method provided in an embodiment of this disclosure. The method can be executed by a text retrieval device, which can be implemented in software and / or hardware. The text retrieval device can be configured in an electronic device, such as a server or terminal, where the terminal specifically includes a mobile phone, computer, or tablet computer. Figure 1 As shown, the text retrieval method provided in this embodiment includes the following steps:

[0031] S101. In response to a text retrieval request, feature extraction is performed on the text retrieval request to obtain the first feature vector of the text retrieval request.

[0032] The text retrieval request in this embodiment can be understood as a request to retrieve text from a text library and find the corresponding text. For example, the text retrieval request can be in text form, voice form, etc., and is not limited here.

[0033] In this embodiment of the disclosure, after receiving a text retrieval request input by a user, the text retrieval device can perform feature extraction processing on the text retrieval request to obtain a first feature vector corresponding to the text retrieval request.

[0034] In one exemplary embodiment of this disclosure, the text retrieval device can input a text retrieval request into a pre-trained feature extraction model for feature extraction. For example, the feature extraction model can be a word vector model (word2vec).

[0035] In another exemplary embodiment of the present disclosure, the text retrieval device may use the term frequency–inverse document frequency (TF-IDF) algorithm to extract a first feature vector from the text retrieval request.

[0036] S102. Based on the target keywords corresponding to the text retrieval request, perform keyword matching with the text in the text library, filter out the first text that matches the target keywords from the text library, and obtain the second feature vector of the first text.

[0037] In this embodiment of the disclosure, the text in the text library can be a complete document or text obtained by splitting a document into paragraphs, and there is no limitation on this.

[0038] The target keywords in this embodiment can be understood as keywords related to the text retrieval request. For example, the target keywords can be keywords contained in the text retrieval request, or keywords contained in the text retrieval request and other synonymous keywords, which are not limited here.

[0039] In this embodiment of the present disclosure, after receiving a text retrieval request, the text retrieval device can determine the target keyword corresponding to the request based on the text retrieval request, perform keyword matching between the target keyword and the text in the text library, filter out the first text containing the target keyword from the text library, and obtain the second feature vector of the first text after determining the first text.

[0040] In one exemplary embodiment of this disclosure, the text retrieval device may extract a second feature vector of the first text based on a method similar to that used in S101 to extract the first feature vector corresponding to the text retrieval request.

[0041] In another exemplary embodiment of this disclosure, the text retrieval device can perform word segmentation on the text retrieval request, removing words without actual meaning such as modal particles and conjunctions, and determining the remaining segmented words as keywords contained in the text retrieval request. Alternatively, it can perform keyword matching on the text retrieval request based on a preset keyword database to determine the keywords contained in the text retrieval request. After determining the keywords contained in the text retrieval request, the text retrieval device can determine other keywords with the same meaning as these keywords based on a preset thesaurus, and determine the keywords contained in the text retrieval request and other keywords with the same meaning as the target keywords corresponding to the text retrieval request.

[0042] S103. For the text in the text library, calculate the first similarity and the second similarity between the text's feature vector and the first and second feature vectors, respectively.

[0043] The similarity in the embodiments of this disclosure can be understood as cosine similarity.

[0044] In this embodiment of the disclosure, after obtaining the first feature vector of the text retrieval request and the second feature vector of the first text, the text retrieval device can obtain the feature vectors of each text in the text library, and calculate the first similarity between the feature vector of each text and the first feature vector, and the second similarity between the feature vector of each text and the second feature vector.

[0045] S104. Sort the texts in the text library according to the comprehensive similarity of the texts to obtain the first sorting result, and determine the second text based on the first sorting result, wherein the comprehensive similarity is determined based on the first similarity and the second similarity of the texts.

[0046] In this embodiment of the present disclosure, after obtaining the first similarity and second similarity between the feature vectors of each text in the text library and the first feature vector and the second feature vector, the text retrieval device can calculate the comprehensive similarity of each text based on the first similarity and second similarity of each text. For example, the average of the first similarity and the second similarity can be used as the comprehensive similarity of the corresponding text, and the texts in the text library can be sorted in descending order of comprehensive similarity to obtain a first sorting result. Based on the first sorting result, the second text with a higher comprehensive similarity is determined, and the second text is returned to the user as the retrieval result.

[0047] In one exemplary embodiment of this disclosure, the text retrieval device may, after obtaining the first ranking result, identify the text with a comprehensive similarity higher than a preset similarity as the second text.

[0048] In another exemplary embodiment of the present disclosure, the text retrieval device may, after obtaining the first sorting result, determine the text with a serial number less than a preset value as the second text.

[0049] In this embodiment, in response to a text retrieval request, features are extracted from the text retrieval request to obtain a first feature vector. Keyword matching is performed between the target keywords corresponding to the text retrieval request and texts in a text library. First texts matching the target keywords are selected from the text library, and a second feature vector of the first text is obtained. For each text in the text library, a first similarity and a second similarity are calculated between the text's feature vector and the first and second feature vectors, respectively. The texts in the text library are sorted according to their comprehensive similarity to obtain a first sorting result. A second text is determined based on the first sorting result. The comprehensive similarity is determined based on the first and second similarities of the texts. This method allows for the calculation of similarity between the text's feature vector and the first and second feature vectors of the first text after matching the first text with the target keywords corresponding to the text retrieval request. Texts with higher comprehensive similarity are then identified as the final retrieval result. This broader retrieval method enables texts that meet the user's retrieval needs but do not directly contain the keywords in the text retrieval request to be detected and presented to the user as the final retrieval result. Meanwhile, since keywords account for a smaller proportion in long documents, the document's feature vector contains more interfering features. Directly matching the document's feature vector with the feature vector of the document retrieval request for similarity results is easily affected by other interfering keywords in the long document. However, the retrieval method that combines keyword matching and similarity calculation can highlight keyword features and output more comprehensive retrieval results, with a higher degree of matching between the retrieval results and user needs.

[0050] Figure 2 This is a flowchart of a method for filtering a first text provided in an embodiment of this disclosure, such as... Figure 2 As shown, based on the above embodiments, the first text can be filtered using the following method.

[0051] S201. Obtain the inverted index associated with the target keyword.

[0052] The inverted index in this embodiment can be understood as a text index built according to keywords. Specifically, it stores the information of all texts containing a certain keyword into the inverted index associated with that keyword, and there is a one-to-one correspondence between keywords and inverted indexes.

[0053] In this embodiment of the disclosure, the text retrieval device can obtain an inverted index associated with the target keyword after determining the target keyword corresponding to the text retrieval request.

[0054] S202. Based on the text identifiers of each text in the inverted index, filter out the first text that matches the target keyword from the text library.

[0055] In this embodiment of the disclosure, the text identifier can be understood as a unique identifier used to distinguish text from other texts. For example, the text identifier can be the storage address of the text.

[0056] In this embodiment of the present disclosure, after obtaining the inverted index associated with the target keyword, the text retrieval device can, according to preset filtering rules, filter out the first text that matches the target keyword from the text library based on the text identifiers of each text contained in the inverted index.

[0057] In one exemplary embodiment of this disclosure, the text retrieval device can take the union of the text identifiers of each text contained in the inverted index associated with each keyword, and determine the text containing at least one target keyword as the first text.

[0058] In another exemplary embodiment of this disclosure, the text retrieval device can find the intersection of the text identifiers of each text contained in the inverted index associated with each keyword, and determine the text containing all target keywords as the first text.

[0059] This embodiment of the disclosure obtains an inverted index associated with the target keyword, and based on the text identifiers of each text in the inverted index, filters out the first text that matches the target keyword from the text library. It can quickly determine the first text that matches the target keyword through the inverted index without having to traverse all texts in the text library, thus improving text retrieval efficiency.

[0060] Figure 3 This is a flowchart of a method for determining target keywords provided in an embodiment of this disclosure, such as... Figure 3 As shown, based on the above embodiments, target keywords can be determined by the following method.

[0061] S301. Expand the text retrieval request to obtain an expanded statement with the same semantics as the text retrieval request.

[0062] In this embodiment of the disclosure, after receiving a text retrieval request input by a user, the text retrieval device can perform synonym expansion on the text retrieval request to obtain an expanded statement with the same semantics as the text retrieval request.

[0063] In one exemplary embodiment of this disclosure, after receiving a text retrieval request, the text retrieval device may input the text retrieval request as a question (query) along with a preset prompt into a pre-trained question-answering model. For example, the prompt may be "Please output 5 common expressions with different wording but the same meaning". The question-answering model may be a generative model, such as a Generative Adversarial Network (GAN), which is not limited here.

[0064] S302. The keywords extracted from the text retrieval request and the expanded statement are determined as the target keywords corresponding to the text retrieval request.

[0065] In this embodiment of the disclosure, after obtaining the extended statement corresponding to the text search request, the text search device can extract keywords from the text search request and the extended statement. The specific extraction method is similar to S102, and the extracted keywords are determined as the target keywords corresponding to the text search request.

[0066] This embodiment of the disclosure expands the text retrieval request to obtain an expanded statement with the same semantics as the text retrieval request. Keywords extracted from the text retrieval request and the expanded statement are determined as the target keywords corresponding to the text retrieval request. Synonyms not directly contained in the text retrieval request can also be used as target keywords. Subsequently, keyword matching is performed with text in the text library to increase the breadth of the retrieval, so that the first text obtained by matching is more in line with the user's needs, and further improves the accuracy of the final retrieval results.

[0067] Figure 4 This is a flowchart illustrating a method for determining the comprehensive similarity of text provided in an embodiment of this disclosure. Figure 4 As shown, based on the above embodiments, the overall similarity of texts can be determined by the following method.

[0068] S401. Obtain the first weight and the second weight corresponding to the first feature vector and the second feature vector, respectively.

[0069] In this embodiment of the disclosure, the text retrieval device can obtain the first weight and the second weight corresponding to the first feature vector and the second feature vector respectively after obtaining the first similarity between the feature vector of the text in the text library and the first feature vector of the text retrieval request, and the second similarity between the feature vector of the text and the second feature vector of the first text.

[0070] In one exemplary embodiment of this disclosure, the specific values ​​of the first weight and the second weight can be set by the user as needed.

[0071] S402. Based on the first similarity and first weight of the corresponding text, and the second similarity and second weight of the corresponding text, calculate the weighted sum of the similarities of the corresponding text, and determine the weighted sum of the similarities as the comprehensive similarity of the corresponding text.

[0072] In this embodiment of the disclosure, after obtaining the first weight and the second weight corresponding to the first feature vector and the second feature vector respectively, the text retrieval device calculates the product of the first similarity and the first weight for each text in the text library, and then calculates the product of the second similarity and the second weight for each text. The sum of the weighted similarity results obtained from the two calculations is determined as the comprehensive similarity of the text.

[0073] In one exemplary embodiment of this disclosure, there may be multiple first texts that match the target keyword. The second similarity corresponding to the second feature vectors of each first text may be the same or different. For example, when there are first texts A, B, and C, the second weights corresponding to the first texts A, B, and C are k1, k2, and k3, respectively. The second similarity between the feature vector x of text X in the text library and the second feature vector a of the first text A is s1, the second similarity between the feature vector x of text X and the second feature vector b of the first text B is s2, and the second similarity between the feature vector x of text X and the second feature vector c of the first text C is s3. When calculating the product of the second similarity and the second weight corresponding to text X, s1×k1+s2×k2+s3×k3 can be calculated, and the calculation result can be summed with the product of the first similarity and the first weight corresponding to text X to obtain the comprehensive similarity corresponding to text X.

[0074] The text in the text library includes the first text. When calculating the comprehensive similarity of the texts in the text library, it is also necessary to calculate the comprehensive similarity of the first text. That is, for the first text A, the second similarity between the feature vector a of the first text A (i.e., the second feature vector of the first text A) and the second feature vector a of the first text A is s4 (which can take the value 1). The second similarity between the feature vector a of the first text A and the second feature vector b of the first text B is s5. The second similarity between the feature vector a of the first text A and the second feature vector c of the first text C is s6. When calculating the product of the second similarity and the second weight corresponding to text A, we can calculate s4×k1+s5×k2+s6×k3, and sum the calculation result with the product of the first similarity and the first weight corresponding to text A to obtain the comprehensive similarity of text A.

[0075] This embodiment of the disclosure obtains the first weight and the second weight corresponding to the first feature vector and the second feature vector, respectively. Based on the first similarity and the first weight of the text, as well as the second similarity and the second weight of the text, it can calculate the weighted sum of similarity of the text and determine the comprehensive similarity of the text, which facilitates the subsequent determination of the final search result based on the comprehensive similarity.

[0076] In some embodiments of this disclosure, the inverted index also includes the number of keywords associated with the inverted index contained in each text. After the text retrieval device obtains the first text that matches the target keyword from the text library, it sorts the first text in descending order of the sum of the number of each target keyword contained in each first text to obtain a second sorting result. When obtaining the second weight corresponding to the second feature vector, it determines the weight corresponding to each first text based on the pre-obtained correspondence between the sequence number and the weight and the second sorting result, and determines the weight corresponding to each first text as the second weight corresponding to the second feature vector of each first text.

[0077] Specifically, after filtering out the first texts matching the target keywords from the text library, the text retrieval device can obtain the number of corresponding target keywords contained in each first text from the inverted index associated with the target keywords. For each first text, the device calculates the sum of the number of all target keywords contained in that first text and sorts the first texts in descending order of the sum of the number of target keywords contained, obtaining a second sorting result. Then, based on the pre-obtained correspondence between the sequence number and the weight, the device determines the weight corresponding to the sequence number of each first text in the second sorting result, and determines it as the second weight corresponding to the second feature vector of each first text. The weight decreases as the corresponding sequence number increases, which allows the first texts containing a larger total number of target keywords to have a greater impact on the overall similarity of the finally determined texts, while the first texts containing a smaller total number of target keywords have a smaller impact, further improving the matching degree between the finally determined second texts and the user's needs.

[0078] Optionally, when the text retrieval device performs keyword matching between the target keywords and the text in the text library to obtain the first text, it can sort the first text based on the sum of the number of each target keyword contained in each first text in descending order to obtain a second sorting result, and determine the text with the sequence number less than a preset value as the first text. Thus, when obtaining the second weight corresponding to the second feature vector, it can directly determine the weight corresponding to each first text based on the sequence number corresponding to each first text and the correspondence between the sequence number and the weight, and determine the weight corresponding to each first text as the second weight corresponding to the second feature vector of each first text.

[0079] Figure 5 is a flowchart of a text preprocessing method provided by an embodiment of the present disclosure. As Figure 5 shown, based on the above embodiment, text preprocessing can be performed through the following method.

[0080] S501. Perform word segmentation on the texts in the text library to determine the keywords included in each text.

[0081] In an embodiment of the present disclosure, after establishing the text library, the text retrieval device can perform word segmentation on the texts in the text library, or when there are texts to be added to the text library, perform word segmentation on such texts to determine the keywords included in each text.

[0082] In an exemplary implementation manner of an embodiment of the present disclosure, the text retrieval device can use the Jieba Chinese word segmentation tool to perform word segmentation on the text, and then perform stop word filtering to filter out some commonly used meaningless words such as "de", "a", "le", etc., and determine the remaining word segmentation results as the keywords included in the text.

[0083] In another exemplary implementation manner of an embodiment of the present disclosure, when the texts in the text library are texts obtained by splitting the original document by paragraphs, the text retrieval device can perform word segmentation on the texts obtained after splitting the original document by paragraphs to determine the keywords included in the text.

[0084] S502. Add the text identifier of the text to the inverted index associated with the keywords included in the text.

[0085] In an embodiment of the present disclosure, after determining the keywords included in each text, the text retrieval device can, for each keyword, determine whether there is an inverted index associated with the keyword. If so, add the text identifier of the text containing the keyword to the corresponding inverted index. If not, create an inverted index associated with the keyword and add the text identifier of the text containing the keyword to the newly created inverted index.

[0086] S503. Extract features of the text to obtain a feature vector of the text, and store the feature vector of the text.

[0087] In an embodiment of the present disclosure, after adding the text identifier to the inverted index associated with the keywords included in the text, the text retrieval device can extract features of the text to obtain a feature vector of the text, and store the feature vector of the text. Optionally, the text retrieval device can establish a correspondence between the text identifier of the text and the feature vector, which is convenient for directly obtaining the feature vector of the text according to the text identifier when processing a text retrieval request later.

[0088] This embodiment of the disclosure performs word segmentation on the text in the text library, determines the keywords contained in each text, adds the text identifier of the text to the inverted index associated with the keywords contained in the text, extracts features from the text to obtain the feature vector of the text, and stores the feature vector of the text. It can complete the relevant processing of the text in the text library in advance, so that after receiving a text retrieval request, it can directly obtain the processing results of the text and apply them, further improving the efficiency of text retrieval.

[0089] In other embodiments of this disclosure, after extracting features from the text to obtain the text's feature vectors, similarity calculations can be performed on the feature vectors corresponding to different paragraphs in the same original document. Multiple paragraphs with similarity higher than a preset threshold can be merged. Specifically, the average value of the feature vectors corresponding to multiple paragraphs can be calculated, and this average value can be determined as the feature vector corresponding to these multiple paragraphs. The inverted index can then be modified based on the unified text identifier of the merged text.

[0090] In some embodiments of this disclosure, the first feature vector and the second feature vector are extracted based on a large language model.

[0091] Large language models can be understood as language models with large parameter sizes and training data. They are trained using large amounts of data from different domains and languages, and possess cross-domain and cross-linguistic feature extraction capabilities. Large language models primarily employ a transformer architecture, where the attention mechanism allows them to capture the contextual features of text. Due to their massive parameters, they can capture complex contextual features, understand the deep semantics, syntax, and context of long text documents, and obtain accurate word embeddings containing deep features. By extracting the first and second feature vectors from the text retrieval request and the first text, the large language model can better summarize the text features, further improving retrieval accuracy.

[0092] Figure 6 This is a schematic diagram of the structure of a text retrieval device provided in an embodiment of this disclosure. For example... Figure 6As shown, the text retrieval device 600 includes: a first extraction module 610, a matching module 620, a first calculation module 630, and a determination module 640. The first extraction module 610, in response to a text retrieval request, performs feature extraction on the text retrieval request to obtain a first feature vector of the text retrieval request. The matching module 620, based on the target keywords corresponding to the text retrieval request, performs keyword matching with texts in a text library, filters out first texts matching the target keywords from the text library, and obtains a second feature vector of the first text. The first calculation module 630, for texts in the text library, calculates a first similarity and a second similarity between the feature vector of the text and the first and second feature vectors, respectively. The determination module 640, sorts the texts in the text library according to their comprehensive similarity to obtain a first sorting result, and determines a second text based on the first sorting result, wherein the comprehensive similarity is determined based on the first and second similarities of the texts.

[0093] Optionally, the matching module 620 includes: an acquisition unit for acquiring an inverted index associated with the target keyword; and a filtering unit for filtering the first text that matches the target keyword from the text library based on the text identifiers of each text in the inverted index.

[0094] Optionally, the matching module 620 further includes: an expansion unit, used to expand the text retrieval request to obtain an expanded statement with the same semantics as the text retrieval request; and an extraction unit, used to determine the keywords extracted from the text retrieval request and the expanded statement as the target keywords corresponding to the text retrieval request.

[0095] Optionally, the text retrieval device 600 further includes: an acquisition module, configured to acquire the first weight and the second weight corresponding to the first feature vector and the second feature vector, respectively; and a second calculation module, configured to calculate a weighted sum of similarities corresponding to the text based on the first similarity and the first weight, and the second similarity and the second weight corresponding to the text, and determine the weighted sum of similarities as the comprehensive similarity corresponding to the text.

[0096] Optionally, the inverted index also includes the number of keywords associated with the inverted index contained in each text. The matching module 620 further includes: a sorting unit, used to sort the first texts in descending order based on the sum of the number of each target keyword contained in each first text, to obtain a second sorting result; the acquisition module is specifically used to determine the weight corresponding to each first text based on the pre-acquired correspondence between the sequence number and the weight and the second sorting result, and to determine the weight corresponding to each first text as the second weight corresponding to the second feature vector of each first text.

[0097] Optionally, the text retrieval device 600 further includes: a word segmentation module for segmenting texts in the text library to determine the keywords contained in each text; an adding module for adding the text identifier of the text to the inverted index associated with the keywords contained in the text; and a second extraction module for extracting features from the text to obtain the feature vector of the text and storing the feature vector of the text.

[0098] Optionally, the first feature vector and the second feature vector are extracted based on a large language model.

[0099] The text retrieval device provided in this embodiment can execute the method described in any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0100] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure.

[0101] like Figure 7 As shown, the computer device may include a processor 710 and a memory 720 storing computer program instructions.

[0102] Specifically, the processor 710 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0103] Memory 720 may include mass storage for information or instructions. For example, and not limitingly, memory 720 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 720 may include removable or non-removable (or fixed) media. Where appropriate, memory 720 may be internal or external to the integrated gateway device. In a particular embodiment, memory 720 is non-volatile solid-state memory. In a particular embodiment, memory 720 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0104] The processor 710 reads and executes computer program instructions stored in the memory 720 to perform the steps of the text retrieval method provided in the embodiments of this disclosure.

[0105] In one example, the computer device may also include a transceiver 730 and a bus 740. Wherein, as... Figure 7 As shown, the processor 710, memory 720 and transceiver 730 are connected via bus 740 and communicate with each other.

[0106] Bus 740 may include hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 740 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0107] This disclosure also provides a computer-readable storage medium that can store a computer program, which, when executed by a processor, enables the processor to implement the text retrieval method provided in this disclosure.

[0108] The aforementioned storage medium may, for example, include a memory 720 containing computer program instructions, which can be executed by the processor 710 of the text retrieval device to complete the text retrieval method provided in the embodiments of this disclosure. Optionally, the storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), compact disc ROM (CD-ROM), magnetic tape, floppy disk, and optical data storage device. The aforementioned computer program may be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code may be executed entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0109] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A text retrieval method, characterized in that, The method includes: In response to a text retrieval request, feature extraction is performed on the text retrieval request to obtain a first feature vector of the text retrieval request; Based on the target keywords corresponding to the text retrieval request, keyword matching is performed with the text in the text library. The first text that matches the target keywords is then filtered from the text library, and the second feature vector of the first text is obtained. For the text in the text library, calculate the first similarity and the second similarity between the feature vector of the text and the first feature vector and the second feature vector, respectively; The texts in the text library are sorted according to their comprehensive similarity to obtain a first sorting result, and a second text is determined based on the first sorting result, wherein the comprehensive similarity is determined based on the first similarity and the second similarity of the text.

2. The method according to claim 1, characterized in that, The step of performing keyword matching between the target keyword corresponding to the text retrieval request and the text in the text library, and filtering out the first text that matches the target keyword from the text library, includes: Obtain the inverted index associated with the target keyword; Based on the text identifiers of each text in the inverted index, the first text that matches the target keyword is selected from the text library.

3. The method according to claim 2, characterized in that, Before obtaining the inverted index associated with the target keyword, the method further includes: The text retrieval request is expanded to obtain an expanded statement with the same semantics as the text retrieval request; The keywords extracted from the text retrieval request and the expanded statement are determined as the target keywords corresponding to the text retrieval request.

4. The method according to claim 2, characterized in that, Before sorting the texts in the text library according to their comprehensive similarity, the method includes: Obtain the first weight and the second weight corresponding to the first feature vector and the second feature vector, respectively; Based on the first similarity and the first weight corresponding to the text, and the second similarity and the second weight corresponding to the text, a weighted sum of similarity corresponding to the text is calculated, and the weighted sum of similarity is determined as the comprehensive similarity corresponding to the text.

5. The method according to claim 4, characterized in that, The inverted index also includes the number of keywords associated with the inverted index contained in each text. After filtering out the first text matching the target keyword from the text library, the method further includes: The first texts are sorted in descending order of the sum of the number of each target keyword contained in each first text to obtain a second sorting result; Obtaining the second weight corresponding to the second feature vector includes: Based on the pre-obtained correspondence between the sequence number and the weight, and the second sorting result, the weight corresponding to each first text is determined, and the weight corresponding to each first text is determined as the second weight corresponding to the second feature vector of each first text.

6. The method according to claim 2, characterized in that, Before performing feature extraction on the text retrieval request in response to the text retrieval request, the method further includes: The text in the text library is segmented to determine the keywords contained in each text. Add the text identifier of the text to the inverted index associated with the keywords contained in the text; The text is subjected to feature extraction to obtain a feature vector, and the feature vector is stored.

7. The method according to any one of claims 1-6, characterized in that, The first feature vector and the second feature vector are extracted based on a large language model.

8. A text retrieval device, characterized in that, The device includes: The first extraction module is used to extract features from the text retrieval request in response to the text retrieval request, and obtain the first feature vector of the text retrieval request. The matching module is used to perform keyword matching between the target keyword corresponding to the text retrieval request and the text in the text library, filter out the first text that matches the target keyword from the text library, and obtain the second feature vector of the first text; The first calculation module is used to calculate, for the text in the text library, the first similarity and the second similarity between the feature vector of the text and the first feature vector and the second feature vector, respectively; The determination module is used to sort the texts in the text library according to the comprehensive similarity of the texts to obtain a first sorting result, and determine a second text based on the first sorting result, wherein the comprehensive similarity is determined based on the first similarity and the second similarity of the texts.

9. A computer device, characterized in that, include: Memory; processor; And a computer program; wherein the computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the text retrieval method as described in any one of claims 1-7.

Images