Text recall method and device, computer equipment and storage medium

Abstract

The invention relates to a text recall method and device, computer equipment and a storage medium. The method belongs to the technical field of text retrieval, and comprises the following steps: based on each text embedding model, respectively determining a first vector representation of a retrieval statement; and respectively selecting each first vector from the candidate vector databases to represent the associated target vector database. According to each first vector representation, a second vector representation in a target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determining the text length of the reference text corpus; determining a similarity analysis result of the retrieval statement and the reference text corpus corresponding to the second vector representation in each target database; and recalling the target text corpora according to a similarity analysis result of the retrieval statement and the reference text corpora corresponding to the second vector representation in each target vector database. Based on the method, the accuracy of the recalled target text corpus is effectively improved, and the user retrieval service experience is improved.

Description

Technical Field

[0001] The present application relates to the technical field of text retrieval, and in particular to a text recall method, apparatus, computer equipment and storage medium. Background Art

[0002] A traditional search engine is a computer program whose main function is to search and find relevant information on the Internet. It converts resources such as web pages and documents into searchable keyword lists through indexing technology, and sorts the search results according to the degree of keyword matching.

[0003] Although traditional search engines are widely used in information search, they still have certain limitations in dealing with issues such as semantically related content. In particular, in the field of text document retrieval, the recalled text results are inaccurate, affecting the user experience. Summary of the invention

[0004] Based on this, it is necessary to provide a text recall method, device, computer equipment and storage medium that can effectively improve the accuracy of text recall in response to the above technical problems.

[0005] In a first aspect, the present application provides a text recall method. The method comprises:

[0006] Based on each text embedding model, determine the first vector representation of the search sentence respectively;

[0007] Selecting a target vector database associated with each first vector representation from the candidate vector database;

[0008] Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0009] The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0010] In one embodiment, each text embedding model corresponds to a candidate vector database, and the method further includes:

[0011] Segmenting the reference text corpus to obtain at least one first segment corpus;

[0012] Concatenate the text feature information of the reference text corpus with each of the first segment corpora to obtain the second segment corpora corresponding to each of the first segment corpora;

[0013] For each text embedding model, the text embedding model is used to encode each second segment corpus respectively to obtain a segment vector representation of each second segment corpus;

[0014] Determining a second vector representation of the reference text corpus according to the segment vector representation of each second corpus segment;

[0015] The second vector representation of the reference text corpus is added to a candidate vector database associated with the text embedding model.

[0016] In one embodiment, determining a similarity analysis result between a search statement and a reference text corpus corresponding to the second vector representation in each target vector database according to each first vector representation, a second vector representation in a target vector database associated with each first vector representation, and a text length of a reference text corpus corresponding to the second vector representation in each target vector database includes:

[0017] For each first vector representation, determining a similarity between the first vector representation and a segment vector representation of a second corpus segment included in each second vector representation in a target vector database associated with the first vector representation;

[0018] Based on the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database is determined.

[0019] In one embodiment, determining a similarity analysis result between a search statement and a reference text corpus corresponding to each second vector representation in a target database according to the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in a target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, includes:

[0020] For each second corpus segment, determining a similarity analysis result of the second corpus segment according to a similarity between the first vector representation and a segment vector representation of the second corpus segment, and a text length of the second corpus segment;

[0021] According to the similarity analysis results of each second corpus segment included in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined.

[0022] In one embodiment, each text embedding model corresponds to a candidate vector database;

[0023] Selecting each first vector to represent an associated target vector database from the candidate vector database, including:

[0024] For each first vector representation, a candidate vector database associated with the text embedding model of the first vector representation is processed as a target vector database associated with the first vector representation.

[0025] In one embodiment, recalling the target text corpus according to the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database includes:

[0026] For each target vector database, sorting the reference text corpus corresponding to each second vector representation in the target vector database according to the similarity analysis result;

[0027] According to the sorting results and the quantity threshold, the target text corpus is recalled from each reference text corpus.

[0028] In a second aspect, the present application also provides a text recall device. The device comprises:

[0029] A first determination module, used to determine first vector representations of search sentences based on each text embedding model;

[0030] A selection module, used for selecting a target vector database associated with each first vector representation from the candidate vector database;

[0031] A second determination module is used to determine the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0032] The recall module is used to recall the target text corpus according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0033] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program.

[0034] Based on each text embedding model, determine the first vector representation of the search sentence respectively;

[0035] Selecting a target vector database associated with each first vector representation from the candidate vector database;

[0036] Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0037] The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0038] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0039] Based on each text embedding model, determine the first vector representation of the search sentence respectively;

[0040] Selecting a target vector database associated with each first vector representation from the candidate vector database;

[0041] Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0042] The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0043] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0044] Based on each text embedding model, determine the first vector representation of the search sentence respectively;

[0045] Selecting a target vector database associated with each first vector representation from the candidate vector database;

[0046] Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0047] The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0048] The above-mentioned text recall method, device, computer equipment and storage medium respectively determine the first vector representation of the search sentence based on each text embedding model. Select the target vector database associated with each first vector representation from the candidate vector database. According to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determine the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target database; according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database, recall the target text corpus. When performing text recall, the present application adopts multiple text embedding models and multiple candidate vector databases, and according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determine the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target database, and recall the target text corpus based on the similarity analysis result, effectively improving the accuracy of the recalled target text corpus and improving the user retrieval service experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] FIG1 is a diagram showing an application environment of the text recall method provided by this embodiment;

[0050] FIG2 is a schematic diagram of a flow chart of a first text recall method provided in this embodiment;

[0051] Figure 3 A schematic diagram of the process of constructing a candidate vector database provided in this embodiment;

[0052] Figure 4 A schematic diagram of a process for determining similarity analysis results provided in this embodiment;

[0053] Figure 5 A schematic diagram of a flow chart of a second text recall method provided in this embodiment;

[0054] Figure 6 A structural block diagram of a text recall device provided in this embodiment;

[0055] FIG. 7 is a diagram showing the internal structure of the computer device provided in this embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] The vehicle state analysis method provided in the embodiment of the present application can be executed by a terminal device, or can be executed by a server. That is, the method can be applied to the application environment shown in Figure 1. Take the application to the server as an example for explanation, wherein the server 104 receives the search statement sent by the terminal device 102. The server 104 determines the first vector representation of the search statement based on each text embedding model; selects the target vector database associated with each first vector representation from the candidate vector database; determines the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database; recalls the target text corpus according to the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database. The server 104 sends the recalled target text corpus to the terminal device 102.

[0058] The server 104 may be implemented as an independent server or a server cluster composed of multiple servers. The terminal device 102 may be a smart terminal such as a mobile phone or a computer.

[0059] In one embodiment, as shown in FIG2 , a text recall method is provided, and the method is applied to the server in FIG1 as an example for explanation, as shown in FIG2 , including the following steps:

[0060] S201, determining first vector representations of search sentences based on respective text embedding models.

[0061] The text embedding model refers to a machine learning model that converts text data into a numerical vector. The search sentence refers to a tool with recall text keywords for recalling the required text. The first vector representation refers to the vector representation corresponding to the search sentence.

[0062] As an optional implementation of the embodiment of the present application, the search sentence is input into each text embedding model respectively, and each text embedding model outputs a first vector representation of the search sentence respectively.

[0063] Another optional implementation of the present application is to preprocess the search sentence and input the preprocessed search sentence into each text embedding model, and each text embedding model outputs the first vector representation of the search sentence. The preprocessing method includes but is not limited to word segmentation, filtering, deletion, semantic analysis, etc.

[0064] Exemplarily, the present application includes T lightweight text embedding models, respectively [m1, m2, ..., mT ],m T represents the Tth lightweight text embedding model; after encoding the search sentence, the first vector obtained is represented as [q1,q2,…, q T ]; where q T The first vector representation obtained after the T-th lightweight text embedding model encodes the search sentence.

[0065] S202: Select a target vector database associated with each first vector representation from the candidate vector database.

[0066] The candidate vector database refers to a database used as an alternative to store the second vector representation corresponding to the reference text corpus; it should be noted that each text embedding model in this application corresponds to a candidate vector database. The target vector database refers to a candidate vector database associated with the first vector representation.

[0067] As an optional implementation of the embodiment of the present application, for each first vector representation, the candidate vector database associated with the text embedding model obtained by processing the first vector representation is used as the target vector database associated with the first vector representation. Specifically, the text embedding model associated with each candidate vector database is determined; the candidate vector database associated with the text embedding model obtained by processing the first vector representation is used as the target vector database associated with the first vector representation.

[0068] It should be noted that since there are multiple first text vector representations, all of which are obtained by processing different text embedding models, each text vector representation has a corresponding target vector database.

[0069] S203, determining the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database based on each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database.

[0070] The reference text corpus refers to the text corpus used as a reference, mainly the text corpus that has been made public. The second vector representation refers to the vector representation obtained after encoding the reference text corpus based on the text embedding model. The text length refers to the text length corresponding to the reference text corpus. The similarity analysis result refers to the correlation result between the search statement and the reference text corpus.

[0071] Optionally, in this embodiment, for each first vector representation, the similarity between the first vector representation and each second vector representation in the target vector database associated with the first vector representation is determined. Based on the similarity between the first vector representation and each second vector representation in the target vector database associated with the first vector representation, and the text length of the reference text corpus corresponding to the second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in the target database is determined.

[0072] S204, recalling the target text corpus according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0073] Optionally, in this embodiment, for each target vector database, the reference text corpora corresponding to each second vector representation in the target vector database are sorted according to the similarity analysis result. According to the sorting result and the first quantity threshold, the first target text corpora are recalled from the reference text corpora corresponding to the target database. For example, the first quantity threshold is K. That is to say, the reference text corpora with the first K similarity analysis results are selected from the target vector database as the first target text corpora. The first target text corpora selected from each target vector database are summarized. If there are repeated first target text corpora (it should be noted that the reference text corpora corresponding to the second vector representations in each candidate vector database are the same, but because the associated text embedding models are different, the second vector representations obtained by encoding the reference text corpora may be different, so the first target text corpora finally recalled from each target vector database may be repeated), then the correlation analysis results of the repeated first target text corpora are added. Finally, according to the correlation analysis results, the summarized first target text corpora are sorted, and the second target text corpora are selected from each first target text corpus according to the sorting result and the second quantity threshold.

[0074] The above-mentioned text recall method determines the first vector representation of the search sentence based on each text embedding model. The target vector database associated with each first vector representation is selected from the candidate vector database. According to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target database is determined; according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database, the target text corpus is recalled. When performing text recall, the present application adopts a variety of text embedding models and a variety of candidate vector databases, and according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target database is determined, and based on the similarity analysis result, the target text corpus is recalled, which effectively improves the accuracy of the recalled target text corpus and improves the user retrieval service experience.

[0075] In one embodiment, each text embedding model corresponds to a candidate vector database. In order to construct the candidate vector database, Figure 3 As shown, an optional implementation of a text recall method includes:

[0076] S301, segmenting a reference text corpus to obtain at least one first segment corpus.

[0077] The first segment corpus refers to a corpus segment with finer granularity obtained by segmenting the reference text corpus.

[0078] Optionally, in this embodiment, a segmentation tool is used to segment the reference text corpus to obtain at least one first segment corpus. The segmentation tool may be a semantic analysis model or a text recognition model, or other text processing software.

[0079] S302: Concatenate the text feature information of the reference text corpus with each of the first segment corpora to obtain second segment corpora corresponding to each of the first segment corpora.

[0080] The text feature information includes, but is not limited to, the title, keywords, and yesterday's related information of the reference text corpus.

[0081] Optionally, in this embodiment, for each first segment corpus, the text feature information of the reference text corpus is concatenated with each first segment corpus to obtain a second segment corpus corresponding to each first segment corpus.

[0082] Specifically, the reference text corpus is S{s1,s2,…,s N}, the i-th reference text corpus is s i =(k i ,h i ,u i ,[p i1 ,…,p i* ]); one of the second segment corpora is s ij ’ =k i ||h i ||u i ||p ij , where k i is the reference text corpus i Keywords: h i is the reference text corpus i Title, u i is the reference text corpus i Other relevant information, p ij is the reference text corpus i The main content of the jth paragraph, || is the splicing operation.

[0083] S303: for each text embedding model, use the text embedding model to encode each second segment corpus respectively to obtain a segment vector representation of each second segment corpus.

[0084] The segment vector representation is a vector representation obtained after encoding each second segment corpus.

[0085] Optionally, for each text embedding model, each second segment corpus is input into the text embedding model respectively, and the text embedding model outputs a segment vector representation of each second segment corpus.

[0086] It should be noted that each text embedding model needs to encode the second segment corpus of the reference text corpus to obtain the segment vector representation corresponding to the text embedding model.

[0087] S304: Determine a second vector representation of the reference text corpus according to the segment vector representation of each second corpus segment.

[0088] Optionally, in this embodiment, for each text embedding model, the text embedding model is used to encode each second segment corpus respectively to obtain a segment vector representation of each second segment corpus. The segment vector representations of each second segment corpus of each reference text corpus encoded by each text embedding model are summarized to obtain a vector set, and the vector set is used as the second vector representation of the reference text corpus. Specifically, v i ={vi1 ,…,v ij ,…}, where v i Represents the use of the i-th text embedding model m i The vector set obtained by encoding each second segment corpus of a reference text corpus is the second vector representation; v i1 The segment vector representation corresponding to the second segment corpus of the first segment.

[0089] S305: Add the second vector representation of the reference text corpus to a candidate vector database associated with the text embedding model.

[0090] Optionally, in this embodiment, the second vector representation of the reference text corpus is added to a candidate vector database associated with the text embedding model that processes the second vector representation.

[0091] In this embodiment, the reference text corpus is segmented to obtain at least one first segment corpus. The text feature information of the reference text corpus is spliced ​​with each first segment corpus to obtain the second segment corpus corresponding to each first segment corpus. For each text embedding model, the text embedding model is used to encode each second segment corpus to obtain the segment vector representation of each second segment corpus. According to the segment vector representation of each second corpus segment, the second vector representation of the reference text corpus is determined. The second vector representation of the reference text corpus is added to the candidate vector database associated with the text embedding model. In this embodiment, different candidate vector databases are constructed based on the same reference text corpus, which not only increases the richness of the second vector representation, but also reduces the instability of semantic perception using a single text embedding model, which can effectively improve the accuracy of text recall. In this embodiment, when creating the candidate vector database, the reference text corpus is segmented to obtain a more fine-grained segment vector representation, which further improves the perception ability for the reference text corpus and further improves the accuracy of the recall result.

[0092] In one embodiment, in order to more accurately determine the similarity analysis result, such as Figure 4 As shown, an optional implementation of S203 includes:

[0093] S401 : For each first vector representation, determine the similarity between the first vector representation and a segment vector representation of a second corpus segment included in each second vector representation in a target vector database associated with the first vector representation.

[0094] Here, similarity refers to the degree of similarity, which can be represented by the distance between two vector representations.

[0095] Optionally, in this embodiment, for each first vector representation, based on a similarity confirmation algorithm, the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation is determined. The similarity confirmation algorithm includes but is not limited to Euclidean distance, cosine similarity, inner product, Manhattan distance, Hamming distance, and Jaccard similarity coefficient.

[0096] S402, determining the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database based on the similarity between the segment vector representations of the second corpus segments contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation.

[0097] Optionally, in this embodiment, for each second corpus segment, the similarity analysis result of the second corpus segment is determined based on the similarity between the first vector representation and the segment vector representation of the second corpus segment, as well as the text length of the second corpus segment. Based on the similarity analysis results of each second corpus segment contained in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined. In this embodiment, the similarity analysis results of the first vector representation and each second corpus segment of each reference text corpus are taken into account. Then, based on the similarity analysis results of each second corpus segment contained in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined, so that the obtained similarity analysis results are more accurate.

[0098] Optionally, in this embodiment, the similarity analysis result of the second corpus segment is determined according to the similarity between the first vector representation and the segment vector representation of the second corpus segment, and the text length of the second corpus segment, which can be determined in combination with the following formula (1):

[0099] Φ ik = (∑ |U ik | f(v i*, q i )•len(si' * )) / ( ∑ |U ik | len(si' * )) (1)

[0100] Among them, Φ in formula (1) ikrepresents the similarity analysis result between the first vector representation and the segment vector representation of the second segment corpus of the i-th segment in the K-th reference text corpus; |U ik | represents the number of segment vectors representing the second corpus segment corresponding to the k-th reference text corpus; f(v i*, q i ) represents the segment vector representation of the i-th second segment corpus corresponding to the k-th reference text corpus being recalled and the first vector representation q i Similarity; len(si' * ) represents the text length of the i-th second segment corpus corresponding to the k-th reference text corpus.

[0101] In this embodiment, for each first vector representation, the similarity between the first vector representation and the fragment vector representation of the second corpus fragment contained in each second vector representation in the target vector database associated with the first vector representation is determined. Based on the similarity between the fragment vector representation of the second corpus fragment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus fragment of the reference text corpus corresponding to each second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database is determined. When determining the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database, this embodiment takes into account the fragment vector representation of the second text corpus fragment with finer granularity and the text length of the second corpus fragment, which effectively improves the accuracy of the determined similarity analysis result, thereby improving the accuracy of text recall.

[0102] In one embodiment, if Figure 5 As shown, an optional implementation of a text recall method includes:

[0103] S501, segmenting a reference text corpus to obtain at least one first segment corpus.

[0104] S502: Concatenate the text feature information of the reference text corpus with each of the first segment corpora to obtain second segment corpora corresponding to each of the first segment corpora.

[0105] S503: For each text embedding model, use the text embedding model to encode each second segment corpus respectively to obtain a segment vector representation of each second segment corpus.

[0106] S504: Determine a second vector representation of the reference text corpus according to the segment vector representation of each second corpus segment.

[0107] S505, adding the second vector representation of the reference text corpus to a candidate vector database associated with the text embedding model, wherein each text embedding model corresponds to a candidate vector database.

[0108] S506: Determine first vector representations of the search sentences based on the text embedding models.

[0109] S507: For each first vector representation, a candidate vector database associated with the text embedding model of the first vector representation is processed as a target vector database associated with the first vector representation.

[0110] S508: For each first vector representation, determine the similarity between the first vector representation and the segment vector representation of the second corpus segment included in each second vector representation in the target vector database associated with the first vector representation.

[0111] S509 , for each second corpus segment, determine a similarity analysis result of the second corpus segment according to the similarity between the first vector representation and the segment vector representation of the second corpus segment, and the text length of the second corpus segment.

[0112] S510, determining the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database according to the similarity analysis result of each second corpus segment included in each reference text corpus.

[0113] S511, for each target vector database, sort the reference text corpora corresponding to each second vector representation in the target vector database according to the similarity analysis result.

[0114] S512: Recall the target text corpus from each reference text corpus according to the sorting result and the quantity threshold.

[0115] This embodiment determines the first vector representation of the search statement based on each text embedding model. Select the target vector database associated with each first vector representation from the candidate vector database. According to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determine the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database; according to the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database, recall the target text corpus. When performing text recall, the present application adopts multiple text embedding models and multiple candidate vector databases, and according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determine the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database, and recall the target text corpus based on the similarity analysis result, effectively improving the accuracy of the recalled target text corpus and improving the user search service experience.

[0116] Based on the same inventive concept, the embodiment of the present application also provides a text recall device for implementing the above-mentioned text recall method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the text recall device embodiment provided below can refer to the limitations of the text recall method above, and will not be repeated here.

[0117] In one embodiment, if Figure 6 As shown, a text recall device 1 is provided, comprising:

[0118] A first determination module 10, used to determine first vector representations of search sentences based on respective text embedding models;

[0119] A selection module 20, for selecting a target vector database associated with each first vector representation from the candidate vector database;

[0120] A second determination module 30 is used to determine a similarity analysis result between a search statement and a reference text corpus corresponding to the second vector representation in each target vector database according to each first vector representation, a second vector representation in a target vector database associated with each first vector representation, and a text length of a reference text corpus corresponding to the second vector representation in each target vector database;

[0121] The recall module 40 is used to recall the target text corpus according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0122] Based on the device in this embodiment, based on each text embedding model, the first vector representation of the search sentence is determined respectively. The target vector database associated with each first vector representation is selected from the candidate vector database. According to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target database is determined; according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database, the target text corpus is recalled. When performing text recall, the present application adopts a variety of text embedding models and a variety of candidate vector databases, according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target database is determined, and based on the similarity analysis result, the target text corpus is recalled, which effectively improves the accuracy of the recalled target text corpus and improves the user retrieval service experience.

[0123] In one embodiment, each text embedding model corresponds to a candidate vector database, based on which, Figure 6 As shown, the text recall device 1 also includes:

[0124] A segmentation module, used for segmenting the reference text corpus to obtain at least one first segment corpus;

[0125] A concatenation module, used for concatenating the text feature information of the reference text corpus with each first segment corpus to obtain a second segment corpus corresponding to each first segment corpus;

[0126] An encoding module, for encoding each second segment corpus using the text embedding model for each text embedding model, to obtain a segment vector representation of each second segment corpus;

[0127] A third determination module, configured to determine a second vector representation of the reference text corpus according to the segment vector representation of each second corpus segment;

[0128] An adding module is used to add the second vector representation of the reference text corpus to the candidate vector database associated with the text embedding model.

[0129] In one embodiment, the Figure 6 The second determining module 30 is further specifically configured to:

[0130] For each first vector representation, determining a similarity between the first vector representation and a segment vector representation of a second corpus segment included in each second vector representation in a target vector database associated with the first vector representation;

[0131] Based on the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database is determined.

[0132] In one embodiment, the Figure 6 The second determining module 30 is further specifically configured to:

[0133] For each second corpus segment, determining a similarity analysis result of the second corpus segment according to a similarity between the first vector representation and a segment vector representation of the second corpus segment, and a text length of the second corpus segment;

[0134] According to the similarity analysis results of each second corpus segment included in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined.

[0135] In one embodiment, each text embedding model corresponds to a candidate vector database; on this basis, Figure 6 The selection module 20 is further specifically used for:

[0136] For each first vector representation, a candidate vector database associated with the text embedding model of the first vector representation is processed as a target vector database associated with the first vector representation.

[0137] In one embodiment, the Figure 6 The recall module 40 is also specifically used for:

[0138] For each target vector database, sorting the reference text corpus corresponding to each second vector representation in the target vector database according to the similarity analysis result;

[0139] According to the sorting results and the quantity threshold, the target text corpus is recalled from each reference text corpus.

[0140] Each module in the above-mentioned text recall device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.

[0141] In one embodiment, a computer device is provided, and its internal structure diagram can be shown in FIG7. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a text recall method is implemented. The display unit of the computer device is used to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0142] Those skilled in the art will understand that the structure shown in FIG. 7 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0144] Based on each text embedding model, determine the first vector representation of the search sentence respectively;

[0145] Selecting a target vector database associated with each first vector representation from the candidate vector database;

[0146] Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0147] The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0148] In one embodiment, when the processor executes the computer program, the following steps are further implemented: each text embedding model corresponds to a candidate vector database, and the method further includes:

[0149] Segmenting the reference text corpus to obtain at least one first segment corpus;

[0150] Concatenate the text feature information of the reference text corpus with each of the first segment corpora to obtain the second segment corpora corresponding to each of the first segment corpora;

[0151] For each text embedding model, the text embedding model is used to encode each second segment corpus respectively to obtain a segment vector representation of each second segment corpus;

[0152] Determining a second vector representation of the reference text corpus according to the segment vector representation of each second corpus segment;

[0153] The second vector representation of the reference text corpus is added to a candidate vector database associated with the text embedding model.

[0154] In one embodiment, when the processor executes the computer program, the following steps are further implemented: according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determining the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database, including:

[0155] For each first vector representation, determining a similarity between the first vector representation and a segment vector representation of a second corpus segment included in each second vector representation in a target vector database associated with the first vector representation;

[0156] Based on the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database is determined.

[0157] In one embodiment, when the processor executes the computer program, the following steps are further implemented: according to the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, determining the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database, including:

[0158] For each second corpus segment, determining a similarity analysis result of the second corpus segment according to a similarity between the first vector representation and a segment vector representation of the second corpus segment, and a text length of the second corpus segment;

[0159] According to the similarity analysis results of each second corpus segment included in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined.

[0160] In one embodiment, when the processor executes the computer program, the following steps are further implemented: each text embedding model corresponds to a candidate vector database;

[0161] Selecting each first vector to represent an associated target vector database from the candidate vector database, including:

[0162] For each first vector representation, a candidate vector database associated with the text embedding model of the first vector representation is processed as a target vector database associated with the first vector representation.

[0163] In one embodiment, when the processor executes the computer program, the following steps are further implemented: according to the similarity analysis results between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database, recalling the target text corpus includes:

[0164] For each target vector database, sorting the reference text corpus corresponding to each second vector representation in the target vector database according to the similarity analysis result;

[0165] According to the sorting results and the quantity threshold, the target text corpus is recalled from each reference text corpus.

[0166] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0167] Based on each text embedding model, determine the first vector representation of the search sentence respectively;

[0168] Selecting a target vector database associated with each first vector representation from the candidate vector database;

[0169] Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0170] The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0171] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: each text embedding model corresponds to a candidate vector database, and the method further includes:

[0172] Segmenting the reference text corpus to obtain at least one first segment corpus;

[0173] Concatenate the text feature information of the reference text corpus with each of the first segment corpora to obtain the second segment corpora corresponding to each of the first segment corpora;

[0174] For each text embedding model, the text embedding model is used to encode each second segment corpus respectively to obtain a segment vector representation of each second segment corpus;

[0175] Determining a second vector representation of the reference text corpus according to the segment vector representation of each second corpus segment;

[0176] The second vector representation of the reference text corpus is added to a candidate vector database associated with the text embedding model.

[0177] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determining the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database, including:

[0178] For each first vector representation, determining a similarity between the first vector representation and a segment vector representation of a second corpus segment included in each second vector representation in a target vector database associated with the first vector representation;

[0179] Based on the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database is determined.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: according to the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, determining the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database, including:

[0181] For each second corpus segment, determining a similarity analysis result of the second corpus segment according to a similarity between the first vector representation and a segment vector representation of the second corpus segment, and a text length of the second corpus segment;

[0182] According to the similarity analysis results of each second corpus segment included in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined.

[0183] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: each text embedding model corresponds to a candidate vector database;

[0184] Selecting each first vector to represent an associated target vector database from the candidate vector database, including:

[0185] For each first vector representation, a candidate vector database associated with the text embedding model of the first vector representation is processed as a target vector database associated with the first vector representation.

[0186] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: according to the similarity analysis results between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database, recalling the target text corpus includes:

[0187] For each target vector database, sorting the reference text corpus corresponding to each second vector representation in the target vector database according to the similarity analysis result;

[0188] According to the sorting results and the quantity threshold, the target text corpus is recalled from each reference text corpus.

[0189] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0190] Based on each text embedding model, determine the first vector representation of the search sentence respectively;

[0191] Selecting a target vector database associated with each first vector representation from the candidate vector database;

[0192] Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database;

[0193] The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

[0194] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: each text embedding model corresponds to a candidate vector database, and the method further includes:

[0195] Segmenting the reference text corpus to obtain at least one first segment corpus;

[0196] Concatenate the text feature information of the reference text corpus with each of the first segment corpora to obtain the second segment corpora corresponding to each of the first segment corpora;

[0197] For each text embedding model, the text embedding model is used to encode each second segment corpus respectively to obtain a segment vector representation of each second segment corpus;

[0198] Determining a second vector representation of the reference text corpus according to the segment vector representation of each second corpus segment;

[0199] The second vector representation of the reference text corpus is added to a candidate vector database associated with the text embedding model.

[0200] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, determining the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database, including:

[0201] For each first vector representation, determining a similarity between the first vector representation and a segment vector representation of a second corpus segment included in each second vector representation in a target vector database associated with the first vector representation;

[0202] Based on the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database is determined.

[0203] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: according to the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, determining the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database, including:

[0204] For each second corpus segment, determining a similarity analysis result of the second corpus segment according to a similarity between the first vector representation and a segment vector representation of the second corpus segment, and a text length of the second corpus segment;

[0205] According to the similarity analysis results of each second corpus segment included in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined.

[0206] In one embodiment, when the computer program is executed by a processor, the following steps are further implemented: each text embedding model corresponds to a candidate vector database;

[0207] Selecting each first vector to represent an associated target vector database from the candidate vector database, including:

[0208] For each first vector representation, a candidate vector database associated with the text embedding model of the first vector representation is processed as a target vector database associated with the first vector representation.

[0209] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: according to the similarity analysis results between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database, recalling the target text corpus includes:

[0210] For each target vector database, sorting the reference text corpus corresponding to each second vector representation in the target vector database according to the similarity analysis result;

[0211] According to the sorting results and the quantity threshold, the target text corpus is recalled from each reference text corpus.

[0212] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0213] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0214] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A text recall method, characterized in that: The method comprises: Based on each text embedding model, determine the first vector representation of the search sentence respectively; Selecting a target vector database associated with each first vector representation from the candidate vector database; Determine a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database; The target text corpus is recalled according to the similarity analysis result between the search sentence and the reference text corpus corresponding to the second vector representation in each target vector database.

2. The method according to claim 1, characterized in that: Each text embedding model corresponds to a candidate vector database, and the method further includes: Segmenting the reference text corpus to obtain at least one first segment corpus; Concatenate the text feature information of the reference text corpus with each of the first segment corpora to obtain a second segment corpus corresponding to each of the first segment corpora; For each text embedding model, using the text embedding model, respectively encode each second segment corpus to obtain a segment vector representation of each second segment corpus; Determining a second vector representation of the reference text corpus according to the segment vector representation of each of the second corpus segments; The second vector representation of the reference text corpus is added to a candidate vector database associated with the text embedding model.

3. The method according to claim 2, characterized in that The determining, based on each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database, a similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database includes: For each of the first vector representations, determining a similarity between the first vector representation and a segment vector representation of a second corpus segment included in each of the second vector representations in a target vector database associated with the first vector representation; Based on the similarity between the first vector representation and the segment vector representation of the second corpus segment contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database is determined.

4. The method according to claim 3, characterized in that: The method of determining the similarity analysis result between the search statement and the reference text corpus corresponding to each second vector representation in the target database according to the similarity between the segment vector representations of the second corpus segments contained in each second vector representation in the target vector database associated with the first vector representation, and the text length of each second corpus segment of the reference text corpus corresponding to each second vector representation, comprises: For each second corpus segment, determining a similarity analysis result of the second corpus segment according to a similarity between the first vector representation and a segment vector representation of the second corpus segment, and a text length of the second corpus segment; According to the similarity analysis results of each second corpus segment included in each reference text corpus, the similarity analysis results of the search statement and the reference text corpus corresponding to each second vector representation in the target database are determined.

5. The method according to claim 1, characterized in that: Each text embedding model corresponds to a candidate vector database; Selecting each first vector to represent an associated target vector database from the candidate vector database, including: For each of the first vector representations, a candidate vector database associated with the text embedding model of the first vector representation is processed and used as a target vector database associated with the first vector representation.

6. The method according to claim 1, characterized in that The step of recalling the target text corpus according to the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database comprises: For each of the target vector databases, sorting the reference text corpora corresponding to each second vector representation in the target vector database according to the similarity analysis result; According to the sorting result and the quantity threshold, the target text corpus is recalled from each of the reference text corpora.

7. A text recall device, characterized in that: The device comprises: A first determination module, used to determine first vector representations of search sentences based on each text embedding model; A selection module, used for selecting a target vector database associated with each first vector representation from the candidate vector database; A second determination module is used to determine the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target database according to each first vector representation, the second vector representation in the target vector database associated with each first vector representation, and the text length of the reference text corpus corresponding to the second vector representation in each target vector database; The recall module is used to recall the target text corpus according to the similarity analysis result between the search statement and the reference text corpus corresponding to the second vector representation in each target vector database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the text recall method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images