Text retrieval method, device, electronic device and readable storage medium

Abstract

The present application provides a text retrieval method, device, electronic device and readable storage medium, which pre-process the acquired text query requirements and extract a keyword list from the text query requirements; use the keyword list to perform a preliminary search on the stored texts in the text library to determine multiple preliminary screening matching texts related to the text query requirements; use a pre-trained encoder model to determine the semantic matching probability between the text query requirements and each preliminary screening matching text, perform a secondary search on the multiple preliminary screening matching texts, and determine multiple candidate matching texts; based on the semantic matching probability between the text query requirements and each candidate matching text, use a pre-trained text classifier to determine the target matching text that matches the text query requirements. In this way, the precision of the retrieval results can be improved with the help of the semantic information of the text query requirements and the candidate matching texts, and thus the accuracy of the retrieval results can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a text retrieval method, device, electronic device, and readable storage medium. Background Art

[0002] With the development of Internet technology, people gradually rely on searching for content they are interested in on the Internet. However, as the number of documents on the Internet increases, when users search, they need to search for the content they want from a large number of documents.

[0003] At present, a keyword comparison method is usually adopted, and the text containing the search keyword entered by the user is pushed to the user as the search result. However, considering that the semantics represented by the same words are different in different language environments, it is undoubtedly not accurate to only use the text containing the search keyword as the search result. Therefore, the search results finally presented still have the problem of low accuracy. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a text retrieval method, device, electronic device and readable storage medium, which can combine the semantic matching probability between the text query requirement and the matching text during the retrieval process to determine the target matching text that matches the text query requirement, thereby improving the accuracy of the matching results.

[0005] The present invention provides a text search method, which includes:

[0006] Obtain the user's text query requirements;

[0007] Preprocessing the text query demand and extracting a keyword list from the text query demand;

[0008] Using the keyword list, a preliminary search is performed on the stored texts in the text library to determine a plurality of preliminarily screened matching texts relevant to the text query requirement;

[0009] Using a pre-trained encoder model, by determining the semantic matching probability between the text query requirement and each of the pre-screened matching texts, a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts;

[0010] Based on the semantic matching probability between the text query requirement and each candidate matching text, a pre-trained text classifier is used to cluster the multiple candidate matching texts to determine a target matching text that matches the text query requirement.

[0011] In a possible implementation, using the keyword list to perform a preliminary search of stored texts in a text library to determine a plurality of preliminarily screened matching texts relevant to the text query requirement includes:

[0012] Constructing an inverted index list of texts stored in the text library;

[0013] By matching the keyword list with the inverted index list, a preliminary search is performed on the stored texts in the text library to determine a plurality of pre-screened matching texts that reference any keyword included in the keyword list.

[0014] In one possible implementation, the pre-trained encoder model is used to determine the semantic matching probability between the text query requirement and each pre-screened matching text, and then a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts, including:

[0015] For each primary screening matching text, the primary screening matching text is concatenated with the keyword list to obtain a combined matching text;

[0016] Inputting the combined matching text into the BERT layer of the encoder model to determine a word representation vector of the combined matching text; wherein the word representation vector is used to represent the word semantics of each word in the combined matching text;

[0017] Performing average pooling processing on the word representation vectors to obtain a sentence representation vector of the combined matching text; wherein the sentence representation vector is used to represent the sentence semantics of the combined matching text;

[0018] Inputting the sentence representation vector into the linear layer of the encoder model to determine the semantic matching probability between the text query requirement and the pre-screened matching text;

[0019] Based on the semantic matching probability between the text query requirement and each of the pre-screened matching texts, a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts that meet preset conditions.

[0020] In a possible implementation, the secondary search of the multiple initially screened matching texts based on the semantic matching probability between the text query requirement and each initially screened matching text to determine multiple candidate matching texts that meet preset conditions includes:

[0021] Based on the semantic matching probability between the text query requirement and each pre-screened matching text, the multiple pre-screened matching texts are sorted, and the pre-screened matching text at the front preset position is determined as the candidate matching text that meets the preset conditions; or

[0022] The pre-screened matching texts whose semantic matching probability is greater than a preset matching threshold are determined as candidate matching texts that meet the preset conditions.

[0023] In a possible implementation, preprocessing the text query requirement to extract a keyword list from the text query requirement includes:

[0024] Performing word segmentation processing on the text query requirement, dividing the text query requirement into multiple text word segments;

[0025] performing stop word removal processing on the multiple text segmentations, and determining multiple candidate segmentations from the multiple text segmentations;

[0026] Using the tf / idf algorithm, extract at least one keyword from the multiple candidate word segmentations;

[0027] The at least one keyword is combined to obtain the keyword list.

[0028] In one possible implementation, the determining of a target matching text that matches the text query requirement by clustering the multiple candidate matching texts using a pre-trained text classifier based on the semantic matching probability between the text query requirement and each candidate matching text includes:

[0029] For each candidate matching text, based on the semantic matching probability between the text query requirement and the candidate matching text, a pre-trained text classifier is used to cluster the candidate matching text to determine the category to which the candidate matching text belongs and the probability of belonging to the category;

[0030] Based on the category and category belonging probability of each candidate matching text, a target matching text that matches the text query requirement is determined, and the target matching text is fed back to the user.

[0031] In a possible implementation, determining a target matching text that matches the text query requirement based on the category and category probability of each candidate matching text includes:

[0032] Based on the category belonging probability of each candidate matching text, sorting the multiple categories to which the multiple candidate matching texts belong, and determining the target category at the first priority;

[0033] The candidate matching texts belonging to the target category are determined as target matching texts fed back to the user.

[0034] The present application also provides a text search device, which includes:

[0035] Demand acquisition module, used to obtain user text query requirements;

[0036] A keyword extraction module, configured to pre-process the text query requirements and extract a keyword list from the text query requirements;

[0037] A preliminary search module, configured to perform a preliminary search of the stored texts in the text library using the keyword list to determine a plurality of preliminarily screened matching texts relevant to the text query requirement;

[0038] A secondary retrieval module is configured to utilize a pre-trained encoder model to perform a secondary retrieval on the multiple primary screening matching texts by determining a semantic matching probability between the text query requirement and each primary screening matching text, thereby determining a plurality of candidate matching texts;

[0039] The text matching module is used to determine the target matching text that matches the text query requirement by clustering the multiple candidate matching texts based on the semantic matching probability between the text query requirement and each candidate matching text using a pre-trained text classifier.

[0040] In a possible implementation, the preliminary search module is configured to use the keyword list to perform a preliminary search on the stored texts in the text library to determine a plurality of preliminarily screened matching texts related to the text query requirement. The preliminary search module is configured to:

[0041] Constructing an inverted index list of texts stored in the text library;

[0042] By matching the keyword list with the inverted index list, a preliminary search is performed on the stored texts in the text library to determine a plurality of pre-screened matching texts that reference any keyword included in the keyword list.

[0043] In one possible implementation, when the secondary retrieval module is used to use a pre-trained encoder model to determine the semantic matching probability between the text query requirement and each primary screening matching text, and to perform a secondary search on the multiple primary screening matching texts to determine multiple candidate matching texts, the secondary retrieval module is used to:

[0044] For each primary screening matching text, the primary screening matching text is concatenated with the keyword list to obtain a combined matching text;

[0045] Inputting the combined matching text into the BERT layer of the encoder model to determine a word representation vector of the combined matching text; wherein the word representation vector is used to represent the word semantics of each word in the combined matching text;

[0046] Performing average pooling processing on the word representation vectors to obtain a sentence representation vector of the combined matching text; wherein the sentence representation vector is used to represent the sentence semantics of the combined matching text;

[0047] Inputting the sentence representation vector into the linear layer of the encoder model to determine the semantic matching probability between the text query requirement and the pre-screened matching text;

[0048] Based on the semantic matching probability between the text query requirement and each of the pre-screened matching texts, a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts that meet preset conditions.

[0049] In one possible implementation, when the secondary retrieval module is used to perform a secondary retrieval on the multiple primary screening matching texts based on the semantic matching probability between the text query requirement and each primary screening matching text, and to determine multiple candidate matching texts that meet preset conditions, the secondary retrieval module is used to:

[0050] Based on the semantic matching probability between the text query requirement and each pre-screened matching text, the multiple pre-screened matching texts are sorted, and the pre-screened matching text at the front preset position is determined as the candidate matching text that meets the preset conditions; or

[0051] The pre-screened matching texts whose semantic matching probability is greater than a preset matching threshold are determined as candidate matching texts that meet the preset conditions.

[0052] In a possible implementation, when the keyword extraction module is used to pre-process the text query requirement and extract a keyword list from the text query requirement, the keyword extraction module is used to:

[0053] Performing word segmentation processing on the text query requirement, dividing the text query requirement into multiple text word segments;

[0054] performing stop word removal processing on the multiple text segmentations, and determining multiple candidate segmentations from the multiple text segmentations;

[0055] Using the tf / idf algorithm, extract at least one keyword from the multiple candidate word segmentations;

[0056] The at least one keyword is combined to obtain the keyword list.

[0057] In one possible implementation, when the text matching module is configured to determine a target matching text that matches the text query requirement by clustering the multiple candidate matching texts based on the semantic matching probability between the text query requirement and each candidate matching text using a pre-trained text classifier, the text matching module is configured to:

[0058] For each candidate matching text, based on the semantic matching probability between the text query requirement and the candidate matching text, a pre-trained text classifier is used to cluster the candidate matching text to determine the category to which the candidate matching text belongs and the probability of belonging to the category;

[0059] Based on the category and category belonging probability of each candidate matching text, a target matching text that matches the text query requirement is determined, and the target matching text is fed back to the user.

[0060] In a possible implementation, when the text matching module is used to determine a target matching text that matches the text query requirement based on the category and category probability of each candidate matching text, the text matching module is used to:

[0061] Based on the category belonging probability of each candidate matching text, sorting the multiple categories to which the multiple candidate matching texts belong, and determining the target category at the first priority;

[0062] The candidate matching texts belonging to the target category are determined as target matching texts fed back to the user.

[0063] An embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the text retrieval method as described above are performed.

[0064] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the above-mentioned text retrieval method are executed.

[0065] The text retrieval method, device, electronic device and readable storage medium provided in the embodiment of the present application obtain the user's text query requirements; pre-process the text query requirements and extract a keyword list from the text query requirements; use the keyword list to perform a preliminary search on the stored texts in the text library to determine multiple primary screening matching texts related to the text query requirements; use a pre-trained encoder model to determine the semantic matching probability between the text query requirements and each primary screening matching text, perform a secondary search on the multiple primary screening matching texts to determine multiple candidate matching texts; based on the semantic matching probability between the text query requirements and each candidate matching text, use a pre-trained text classifier to cluster the multiple candidate matching texts to determine the target matching text that matches the text query requirements. In this way, the precision of the retrieval results can be improved by means of the semantic information of the text query requirements and the candidate matching texts, and thus the accuracy of the retrieval results can be improved.

[0066] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0068] Figure 1 A flowchart of a text retrieval method provided in an embodiment of the present application;

[0069] Figure 2 A schematic diagram of the structure of an encoder model provided in an embodiment of the present application;

[0070] Figure 3 A schematic diagram of a text matching process provided in an embodiment of the present application;

[0071] Figure 4 A schematic diagram of the structure of a text search device provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for which protection is claimed, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work falls within the scope of protection of the present application.

[0074] Research has found that, at present, keyword comparison is usually adopted, and the text containing the search keywords entered by the user is pushed to the user as the search result. However, considering that the semantics represented by the same words are different in different language environments, simply using the text containing the search keywords as the search result is undoubtedly not accurate enough. Therefore, the search results finally presented still have the problem of low accuracy.

[0075] Based on this, an embodiment of the present application provides a text retrieval method, which, on the basis of multiple searches of the stored texts in the text library, combines the semantic matching probability between the text query requirements and the candidate matching texts, and searches the candidate matching texts again to retrieve the target matching text that best matches the text query requirements. In turn, the retrieval precision can be improved and the accuracy of the retrieval results can be improved.

[0076] See also Figure 1 , Figure 1 This is a flowchart of a text retrieval method provided in an embodiment of the present application. Figure 1 As shown in , the text retrieval method provided in the embodiment of the present application includes:

[0077] S101: Obtain the user's text query requirements.

[0078] S102: Preprocess the text query requirement and extract a keyword list from the text query requirement.

[0079] S103: Using the keyword list, perform a preliminary search on the stored texts in the text library to determine a plurality of preliminarily screened matching texts related to the text query requirement.

[0080] S104: Using a pre-trained encoder model, by determining the semantic matching probability between the text query requirement and each pre-screened matching text, a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts.

[0081] S105 , based on the semantic matching probability between the text query requirement and each candidate matching text, using a pre-trained text classifier, clustering the multiple candidate matching texts to determine a target matching text that matches the text query requirement.

[0082] The text retrieval method provided in the embodiment of the present application first uses a keyword list extracted from a text query requirement to perform multiple searches on the stored texts in the text library to quickly reduce the search space and determine multiple candidate matching texts that are highly likely to be related to the text query requirement; secondly, based on the secondary search results, combined with the semantic matching probability between the text query requirement and the candidate matching texts, a text classifier is used to cluster the multiple candidate matching texts obtained from the secondary search to determine the target matching text that matches the text query requirement. In this way, the precision of the retrieval results can be improved by means of the semantic information of the text query requirement and the candidate matching texts, and further, the accuracy of the retrieval results can be improved.

[0083] In step S101, when a user has a query requirement, he or she may input a text query requirement through the terminal device used. At this time, the text query requirement input by the user may be obtained to retrieve target matching texts matching the text query requirement in the text library.

[0084] Here, considering that the content in the text query requirements is too complicated and there are a large number of word segmentations, stop words and other contents that do not need to be paid attention to, therefore, in order to reduce the noise words and other contents in the text query requirements, before extracting the keywords in the text query requirements, the text query requirements can be preprocessed through preprocessing, thereby achieving the purpose of reducing the amount of data processing involved in the keyword extraction process.

[0085] In step S102 , the acquired text query requirement is pre-processed to simplify the text query requirement and extract a keyword list that can express the semantics contained in the text query requirement from the text query requirement.

[0086] In one embodiment, step S102 includes:

[0087] S1021: Perform word segmentation processing on the text query requirement, and split the text query requirement into multiple text word segments.

[0088] S1022: Perform stop word removal processing on the multiple text segmentations, and determine multiple candidate segmentations from the multiple text segmentations.

[0089] In this step, the multiple text segmentations obtained by splitting are cleaned to remove redundant segmentations, stop words and other noise content that will not affect the matching results, so as to extract multiple candidate segmentations from the multiple text segmentations that can express the semantics contained in the text query requirements.

[0090] S1023: Using the TF / IDF algorithm, extract at least one keyword from the multiple candidate word segmentations.

[0091] In this step, a keyword extraction algorithm tf / idf algorithm is used to extract multiple candidate segmentations based on the importance of each candidate segmentation, so as to extract at least one keyword with a higher importance from the multiple candidate segmentations.

[0092] S1024: Combine the at least one keyword to obtain the keyword list.

[0093] Here, considering that there are a large number of stored texts in the text library, in order to quickly narrow the search scope and reduce the amount of data processing in the subsequent semantic matching process, a "rough sorting" method can be used to perform a preliminary search on the stored texts in the text library to quickly filter out a large amount of irrelevant text and narrow the search scope.

[0094] In step S103, since the extracted keyword list can represent the semantics contained in the text query requirement, the keyword list can be used as a reference to perform a preliminary search on the stored text in the text library to quickly reduce the search space and narrow down the huge text collection to a small number of texts that may be related to the text query requirement, that is, to determine multiple preliminary matching texts related to the text query requirement.

[0095] Specifically, an "inverted index" of stored text can be established to perform a preliminary search of the stored text in the text library to quickly narrow the search scope. The "inverted index" can actually be considered a data structure that can map the words involved in the stored text to the stored text containing the word. When a user submits a text query, the constructed "inverted index" will be searched for stored text containing keywords, and the stored text containing keywords will be preliminarily sorted according to some predefined rules or algorithms to obtain pre-screened matching text. In this way, preliminary text retrieval can be achieved through rules such as word frequency, text relevance score, or other rules of the inverted index.

[0096] In one embodiment, step S103 includes:

[0097] S1031: Construct an inverted index list of texts stored in the text library.

[0098] In this step, an inverted index list of all stored texts in the text library is constructed using the words that appear in the stored texts. In this way, when performing a preliminary search, the constructed "index", i.e., the inverted index list, can be used to achieve rapid retrieval of the stored texts.

[0099] S1032: Perform a preliminary search on the stored texts in the text library by matching the keyword list with the inverted index list, and determine a plurality of preliminarily screened matching texts that reference any keyword included in the keyword list.

[0100] In this step, by matching the keyword list with the inverted index list, a preliminary search is performed on the stored texts in the text library with reference to each keyword extracted from the text query requirement, and the stored texts that reference any keyword included in the keyword list are mapped to the stored texts, and the stored texts that reference any keyword are determined as preliminarily screened matching texts that may be relevant to the text query requirement.

[0101] Specifically, for each keyword included in the keyword list, the keyword is matched with the inverted index list to determine whether there is a word matching the keyword in the inverted index list; if there is a word matching the keyword in the inverted index list, the word matching the keyword can be used to quickly map out the stored text that references the keyword, and the stored text that references the keyword is determined as a preliminary matching text that may be related to the text query requirement.

[0102] Here, considering that the same word may express different semantics in different scenarios, while different words may express the same semantics, the search results will be relatively one-sided and the accuracy of the search results will be low if only the keywords are matched. Therefore, in order to improve the accuracy of the search results, based on the preliminary search, the semantic features are used to perform a secondary search on the multiple preliminary screened matching texts obtained from the preliminary search.

[0103] In step S104, in order to further reduce the number of text comparisons involved in the semantic matching process, based on the preliminary search, combined with the keyword list extracted from the text query requirements, using a pre-trained encoder model, by determining the semantic matching probability between the text query requirements and each preliminary screening matching text, a secondary search is performed on the multiple preliminary screening matching texts obtained from the preliminary search to determine multiple candidate matching texts that are more relevant to the text query requirements.

[0104] Among them, the encoder model is trained based on sample matching text, sample query requirements and the semantic matching probability between the sample matching text and the sample query requirements; specifically, the sample matching text and the sample query requirements are used as input features of the encoder model, and the semantic matching probability between the sample matching text and the sample query requirements is used as the output feature of the encoder model. The pre-constructed encoder model is trained to obtain a trained encoder model.

[0105] In one embodiment, step S104 includes:

[0106] S1041. For each primary screening matching text, the primary screening matching text is concatenated with the keyword list to obtain a combined matching text.

[0107] In this step, for each primary matching text screened out by the preliminary search, the primary matching text is spliced ​​with the keyword list representing the text query requirement to obtain a combined spliced ​​text including the primary matching text and the text query requirement.

[0108] S1042. Input the combined matching text into the BERT layer of the encoder model to determine the word representation vector of the combined matching text.

[0109] In this step, the spliced ​​combined matching text is input into the BERT layer in the encoder model to encode the combined matching text through the BERT layer to obtain the word representation vector of the combined matching text, thereby enabling sufficient attention interaction between the initial screening matching text and the text query requirements.

[0110] The word representation vector is used to represent the semantics of each word in the combined matching text; that is, the word representation vector can represent the semantics contained in the preliminary matching text and the text query requirement at the word level.

[0111] The BERT layer in the encoder model is trained based on the combined sample text and the sample word representation vectors of the combined sample text. Specifically, the combined sample text is used as the input feature of the BERT layer of the encoder model, and the sample word representation vectors of the combined sample text are used as the output feature of the BERT layer of the encoder model. The pre-built BERT model is trained to obtain the trained BERT layer.

[0112] The combined sample text is obtained by concatenating the sample matching text and the sample query requirement.

[0113] S1043: Perform average pooling processing on the word representation vectors to obtain a sentence representation vector of the combined matching text.

[0114] In this step, considering that the features represented by the word representation vectors are relatively scattered and weakly correlated, in order to enhance the correlation between the features represented by the vectors, the word representation vectors are average pooled to obtain the sentence representation vectors of the combined matching text.

[0115] The sentence representation vector is used to represent the sentence semantics of the combined matching text; that is, the sentence representation vector can represent the semantics contained in the preliminary screening matching text and the text query requirement at the sentence level.

[0116] S1044: Input the sentence representation vector into the linear layer of the encoder model to determine the semantic matching probability between the text query requirement and the preliminarily screened matching text.

[0117] In this step, the sentence representation vector obtained after average pooling is input into the linear layer of the encoder model, and the semantic matching probability between the text query requirement and the initially screened matching text is determined through the thread layer of the encoder model.

[0118] The linear layer in the encoder model is trained based on the sample sentence representation vector of the combined sample text and the semantic matching probability between the sample matching text and the sample query requirement; specifically, the sample sentence representation vector of the combined sample text is used as the input feature of the linear layer of the encoder model, and the semantic matching probability between the sample matching text and the sample query requirement is used as the output feature of the linear layer of the encoder model, and the pre-constructed linear layer is trained to obtain a trained linear layer.

[0119] S1045. Based on the semantic matching probability between the text query requirement and each of the pre-screened matching texts, perform a secondary search on the multiple pre-screened matching texts to determine multiple candidate matching texts that meet preset conditions.

[0120] In this step, with the help of the semantic matching probability between the text query requirement and the primary screening matching text, a secondary search is performed on the multiple primary screening matching texts obtained in the initial search to further narrow the scope of the search results and determine multiple candidate matching texts that meet the preset conditions from the multiple primary screening matching texts.

[0121] Here, candidate matching texts with a higher probability of semantic matching with the text query requirement may be selected by sorting; or, candidate matching texts matching the text query requirement may be selected by setting a matching threshold.

[0122] In one embodiment, step S1045 includes: sorting the multiple preliminary screening matching texts based on the semantic matching probability between the text query requirement and each preliminary screening matching text, and determining the preliminary screening matching text located at the front preset position as the candidate matching text that meets the preset conditions.

[0123] In this step, the multiple preliminary matching texts obtained by the preliminary retrieval are sorted in descending order according to the semantic matching probability between the text query requirement and each preliminary matching text, and the preliminary matching text located in a preset position before the sorting result is determined as a candidate matching text that meets the preset conditions and is more relevant to the text query requirement.

[0124] In another embodiment, step S1045 includes: determining the pre-screened matching texts with semantic matching probabilities greater than a preset matching threshold as candidate matching texts that meet preset conditions.

[0125] In this step, according to the semantic matching probability between the text query requirement and each preliminary screening matching text, the preliminary screening matching text with a semantic matching probability greater than a preset matching threshold is determined as a candidate matching text that meets the preset conditions and is more relevant to the text query requirement.

[0126] Here, considering that the number of candidate matching texts obtained after the secondary retrieval is large, on this basis, in order to further select the target matching text to be fed back to the user, the candidate matching texts are clustered with the help of a pre-trained text classifier to determine the target matching text suitable for feedback to the user.

[0127] See also Figure 2 , Figure 2 This is a schematic diagram of the structure of an encoder model provided in an embodiment of the present application. Figure 2 As shown, the combined matching text obtained by concatenating the initial screening matching text and the keyword list is input into the encoder model, and the combined matching text is encoded through the BERT layer of the encoder model to obtain the word representation vector of the combined matching text at the word level; the sentence representation vector of the combined matching text is obtained by average pooling each word representation vector; the sentence representation vector is input into the linear layer of the encoder model, and the semantic matching probability between the text query requirement and the initial screening matching text is determined through the linear layer of the encoder model; then, according to the semantic matching probability, secondary retrieval can be implemented to determine multiple candidate matching texts.

[0128] In step S105, based on the semantic matching probability between the text query requirement and each candidate matching text, a pre-trained text classifier is used to cluster multiple candidate matching texts to determine the category to which each candidate matching text belongs, so as to select a target matching text that matches the text query requirement from the multiple candidate matching texts.

[0129] In one embodiment, step S105 includes:

[0130] S1051. For each candidate matching text, based on the semantic matching probability between the text query requirement and the candidate matching text, a pre-trained text classifier is used to cluster the candidate matching text to determine the category to which the candidate matching text belongs and the probability of belonging to the category.

[0131] In this step, for each candidate matching text obtained after the secondary retrieval, the semantic matching probability between the candidate matching text and the text query requirement is input into a pre-trained text classifier, so that the candidate matching text is clustered by the text classifier to determine the category to which the candidate matching text belongs and the category belonging probability of the candidate matching text belonging to the category; wherein the category is a plurality of categories pre-divided for all texts, and when clustering, the category to which the candidate matching text belongs can be determined by determining the category belonging probability between the candidate matching text and each category.

[0132] S1052: Based on the category and category probability of each candidate matching text, determine a target matching text that matches the text query requirement, and feed the target matching text back to the user.

[0133] In this step, the category to which each candidate matching text belongs and the probability of belonging to the category are combined to determine the target matching text that is suitable for feedback to the user and matches the user's text query needs, and the target matching text is fed back to the user; in this way, the target matching text that better meets the user's query needs can be pushed to the user, so as to avoid the user repeatedly performing multiple query operations to obtain the text he wants when he fails to obtain the query results accurately, thereby increasing the number of user queries and the response frequency of the terminal device.

[0134] In one embodiment, step S1052 includes:

[0135] S10521. Sort the categories to which the multiple candidate matching texts belong based on the category belonging probability of each candidate matching text, and determine the target category that ranks first.

[0136] In this step, after determining the category to which each candidate matching text belongs and the probability of belonging to the category, a target category suitable for recommendation to the user can be determined based on the category probability of each candidate matching text, so that the candidate matching text belonging to the target category can be recommended to the user as the target matching text; specifically, based on the category probability of each candidate matching text, the category probability of each of the multiple categories to which multiple candidate matching texts belong can be determined, and the category probability of each category can be used to sort the multiple categories to which multiple candidate matching texts belong, and determine the target category that is ranked first.

[0137] Here, when there are multiple candidate matching texts under the category, the category probability of the category can be determined by summing or weighted summing.

[0138] S10522: Determine the candidate matching text belonging to the target category as the target matching text fed back to the user.

[0139] In this step, candidate matching texts belonging to the target category are determined as target matching texts that match the text query requirement and are to be fed back to the user.

[0140] See also Figure 3 , Figure 3 This is a diagram of a text matching process provided by an embodiment of the present application. Figure 3 As shown, step 301, establish an inverted index list of texts stored in the text library; step 302, obtain the user's text query requirements; step 303, perform word segmentation processing on the text query requirements to obtain multiple text segmentations; step 304, remove stop words from the multiple text segmentations to obtain multiple candidate segmentations; step 305, use the TF / IDF algorithm to extract keywords, extract at least one keyword from the multiple candidate segmentations, and obtain a keyword list; step 306, use the keyword list and the inverted index list to perform a preliminary search on the stored texts, and screen out multiple preliminary screening matching texts from the stored texts in the text library; step 307, calculate the semantic matching probability between the text query requirement and each preliminary screening matching text, and screen out multiple candidate matching texts based on the semantic matching probability; step 308, classify the multiple candidate matching texts through a text classifier; step 309, determine the target matching text that matches the text query requirement.

[0141] The text retrieval method provided by the embodiment of the present application obtains the user's text query requirements; pre-processes the text query requirements and extracts a keyword list from the text query requirements; uses the keyword list to perform a preliminary search on the stored texts in the text library to determine multiple primary screening matching texts related to the text query requirements; uses a pre-trained encoder model to determine the semantic matching probability between the text query requirements and each primary screening matching text, performs a secondary search on the multiple primary screening matching texts to determine multiple candidate matching texts; based on the semantic matching probability between the text query requirements and each candidate matching text, uses a pre-trained text classifier to cluster the multiple candidate matching texts to determine the target matching text that matches the text query requirements. In this way, the precision of the retrieval results can be improved by means of the semantic information of the text query requirements and the candidate matching texts, and thus the accuracy of the retrieval results can be improved.

[0142] See also Figure 4 , Figure 4 This is a structural diagram of a text search device provided in an embodiment of the present application. Figure 4 As shown in , the text retrieval device 400 includes:

[0143] The demand acquisition module 410 is used to obtain the user's text query demand;

[0144] A keyword extraction module 420 is used to pre-process the text query requirement and extract a keyword list from the text query requirement;

[0145] A preliminary search module 430 is configured to perform a preliminary search on the stored texts in the text library using the keyword list to determine a plurality of preliminarily screened matching texts relevant to the text query requirement;

[0146] A secondary search module 440 is configured to use a pre-trained encoder model to determine a semantic match probability between the text query requirement and each of the primary screening matching texts, and perform a secondary search on the multiple primary screening matching texts to determine multiple candidate matching texts;

[0147] The text matching module 450 is used to determine the target matching text that matches the text query requirement by clustering the multiple candidate matching texts based on the semantic matching probability between the text query requirement and each candidate matching text using a pre-trained text classifier.

[0148] Furthermore, the preliminary search module 430 is used to perform a preliminary search on the stored texts in the text library using the keyword list to determine a plurality of preliminarily screened matching texts related to the text query requirement. The preliminary search module 430 is used to:

[0149] Constructing an inverted index list of texts stored in the text library;

[0150] By matching the keyword list with the inverted index list, a preliminary search is performed on the stored texts in the text library to determine a plurality of pre-screened matching texts that reference any keyword included in the keyword list.

[0151] Furthermore, when the secondary retrieval module 440 is used to perform a secondary retrieval on the multiple primary screening matching texts by using the pre-trained encoder model to determine the semantic matching probability between the text query requirement and each primary screening matching text, and to determine multiple candidate matching texts, the secondary retrieval module 440 is used to:

[0152] For each primary screening matching text, the primary screening matching text is concatenated with the keyword list to obtain a combined matching text;

[0153] Inputting the combined matching text into the BERT layer of the encoder model to determine a word representation vector of the combined matching text; wherein the word representation vector is used to represent the word semantics of each word in the combined matching text;

[0154] Performing average pooling processing on the word representation vectors to obtain a sentence representation vector of the combined matching text; wherein the sentence representation vector is used to represent the sentence semantics of the combined matching text;

[0155] Inputting the sentence representation vector into the linear layer of the encoder model to determine the semantic matching probability between the text query requirement and the pre-screened matching text;

[0156] Based on the semantic matching probability between the text query requirement and each of the pre-screened matching texts, a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts that meet preset conditions.

[0157] Furthermore, when the secondary search module 440 is used to perform a secondary search on the multiple primary screening matching texts based on the semantic matching probability between the text query requirement and each primary screening matching text, and to determine multiple candidate matching texts that meet the preset conditions, the secondary search module 440 is used to:

[0158] Based on the semantic matching probability between the text query requirement and each pre-screened matching text, the multiple pre-screened matching texts are sorted, and the pre-screened matching text at the front preset position is determined as the candidate matching text that meets the preset conditions; or

[0159] The pre-screened matching texts whose semantic matching probability is greater than a preset matching threshold are determined as candidate matching texts that meet the preset conditions.

[0160] Furthermore, when the keyword extraction module 420 is used to pre-process the text query requirement and extract a keyword list from the text query requirement, the keyword extraction module 420 is used to:

[0161] Performing word segmentation processing on the text query requirement, dividing the text query requirement into multiple text word segments;

[0162] performing stop word removal processing on the multiple text segmentations, and determining multiple candidate segmentations from the multiple text segmentations;

[0163] Using the tf / idf algorithm, extract at least one keyword from the multiple candidate word segmentations;

[0164] The at least one keyword is combined to obtain the keyword list.

[0165] Furthermore, when the text matching module 450 is used to determine a target matching text that matches the text query requirement by clustering the multiple candidate matching texts based on the semantic matching probability between the text query requirement and each candidate matching text, using a pre-trained text classifier, the text matching module 450 is used to:

[0166] For each candidate matching text, based on the semantic matching probability between the text query requirement and the candidate matching text, a pre-trained text classifier is used to cluster the candidate matching text to determine the category to which the candidate matching text belongs and the probability of belonging to the category;

[0167] Based on the category and category belonging probability of each candidate matching text, a target matching text that matches the text query requirement is determined, and the target matching text is fed back to the user.

[0168] Furthermore, when the text matching module 450 is used to determine a target matching text that matches the text query requirement based on the category and category probability of each candidate matching text, the text matching module 450 is used to:

[0169] Based on the category belonging probability of each candidate matching text, sorting the multiple categories to which the multiple candidate matching texts belong, and determining the target category at the first priority;

[0170] The candidate matching texts belonging to the target category are determined as target matching texts fed back to the user.

[0171] The text retrieval device provided by the embodiment of the present application obtains the user's text query requirements; pre-processes the text query requirements and extracts a keyword list from the text query requirements; uses the keyword list to perform a preliminary search on the stored texts in the text library to determine multiple primary screening matching texts related to the text query requirements; uses a pre-trained encoder model to determine the semantic matching probability between the text query requirements and each primary screening matching text, performs a secondary search on the multiple primary screening matching texts to determine multiple candidate matching texts; based on the semantic matching probability between the text query requirements and each candidate matching text, uses a pre-trained text classifier to cluster the multiple candidate matching texts to determine the target matching text that matches the text query requirements. In this way, the precision of the retrieval results can be improved by means of the semantic information of the text query requirements and the candidate matching texts, and thus the accuracy of the retrieval results can be improved.

[0172] See also Figure 5 , Figure 5This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in FIG, the electronic device 500 includes a processor 510, a memory 520 and a bus 530.

[0173] The memory 520 stores machine-readable instructions executable by the processor 510. When the electronic device 500 is running, the processor 510 communicates with the memory 520 via the bus 530. When the machine-readable instructions are executed by the processor 510, the above-mentioned Figure 1 The steps of the text retrieval method in the illustrated method embodiment and the specific implementation thereof can be found in the method embodiment, which will not be described in detail here.

[0174] The embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the text retrieval method in the illustrated method embodiment and the specific implementation thereof can be found in the method embodiment, which will not be described in detail here.

[0175] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. There may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interface, indirect coupling or communication connection of devices or units, which may be electrical, mechanical or other forms.

[0177] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0178] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0179] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0180] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The scope of protection of the present application is not limited thereto. Although the present application has been described in detail with reference to the above-mentioned embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-mentioned embodiments within the technical scope disclosed in the present application, or perform equivalent replacements for some of the technical features thereof. These modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A text retrieval method, characterized in that: The text retrieval method comprises: Obtain the user's text query requirements; Preprocessing the text query demand and extracting a keyword list from the text query demand; Using the keyword list, a preliminary search is performed on the stored texts in the text library to determine a plurality of preliminarily screened matching texts relevant to the text query requirement; Using a pre-trained encoder model, by determining the semantic matching probability between the text query requirement and each of the pre-screened matching texts, a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts; Based on the semantic matching probability between the text query requirement and each candidate matching text, a pre-trained text classifier is used to cluster the multiple candidate matching texts to determine a target matching text that matches the text query requirement; The method of determining a target matching text that matches the text query requirement by clustering the plurality of candidate matching texts based on the semantic matching probability between the text query requirement and each candidate matching text using a pre-trained text classifier includes: For each candidate matching text, based on the semantic matching probability between the text query requirement and the candidate matching text, a pre-trained text classifier is used to cluster the candidate matching text to determine the category to which the candidate matching text belongs and the probability of belonging to the category; Based on the category and category probability of each candidate matching text, determining a target matching text that matches the text query requirement, and feeding back the target matching text to the user; The step of determining a target matching text that matches the text query requirement based on the category and category probability of each candidate matching text includes: Based on the category belonging probability of each candidate matching text, sorting the multiple categories to which the multiple candidate matching texts belong, and determining the target category at the first priority; The candidate matching texts belonging to the target category are determined as target matching texts fed back to the user.

2. The text retrieval method according to claim 1, characterized in that The keyword list is used to perform a preliminary search on the stored texts in the text library to determine a plurality of preliminarily screened matching texts related to the text query requirement, including: Constructing an inverted index list of texts stored in the text library; By matching the keyword list with the inverted index list, a preliminary search is performed on the stored texts in the text library to determine a plurality of pre-screened matching texts that reference any keyword included in the keyword list.

3. The text retrieval method according to claim 1, wherein: The pre-trained encoder model is used to determine the semantic matching probability between the text query requirement and each pre-screened matching text, and then a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts, including: For each primary screening matching text, the primary screening matching text is concatenated with the keyword list to obtain a combined matching text; Inputting the combined matching text into the BERT layer of the encoder model to determine a word representation vector of the combined matching text; wherein the word representation vector is used to represent the word semantics of each word in the combined matching text; Performing average pooling processing on the word representation vectors to obtain a sentence representation vector of the combined matching text; wherein the sentence representation vector is used to represent the sentence semantics of the combined matching text; Inputting the sentence representation vector into the linear layer of the encoder model to determine the semantic matching probability between the text query requirement and the pre-screened matching text; Based on the semantic matching probability between the text query requirement and each of the pre-screened matching texts, a secondary search is performed on the multiple pre-screened matching texts to determine multiple candidate matching texts that meet preset conditions.

4. The text retrieval method according to claim 3, characterized in that The method of performing a secondary search on the multiple initially screened matching texts based on the semantic matching probability between the text query requirement and each initially screened matching text to determine multiple candidate matching texts that meet preset conditions includes: Based on the semantic matching probability between the text query requirement and each pre-screened matching text, the multiple pre-screened matching texts are sorted, and the pre-screened matching text at the front preset position is determined as the candidate matching text that meets the preset conditions; or The pre-screened matching texts whose semantic matching probability is greater than a preset matching threshold are determined as candidate matching texts that meet the preset conditions.

5. The text retrieval method according to claim 1, wherein: The preprocessing of the text query requirement to extract a keyword list from the text query requirement includes: Performing word segmentation processing on the text query requirement, dividing the text query requirement into multiple text word segments; performing stop word removal processing on the multiple text segmentations, and determining multiple candidate segmentations from the multiple text segmentations; Using the tf / idf algorithm, extract at least one keyword from the multiple candidate word segmentations; The at least one keyword is combined to obtain the keyword list.

6. A text retrieval device, characterized in that: The text retrieval device comprises: Demand acquisition module, used to obtain user text query requirements; A keyword extraction module, configured to pre-process the text query requirements and extract a keyword list from the text query requirements; A preliminary search module, configured to perform a preliminary search of the stored texts in the text library using the keyword list to determine a plurality of preliminarily screened matching texts relevant to the text query requirement; A secondary retrieval module is configured to utilize a pre-trained encoder model to perform a secondary retrieval on the multiple primary screening matching texts by determining a semantic matching probability between the text query requirement and each primary screening matching text, thereby determining a plurality of candidate matching texts; A text matching module is configured to determine a target matching text that matches the text query requirement by clustering the multiple candidate matching texts based on the semantic matching probability between the text query requirement and each candidate matching text, using a pre-trained text classifier; When the text matching module is used to determine a target matching text that matches the text query requirement by clustering the multiple candidate matching texts based on the semantic matching probability between the text query requirement and each candidate matching text, using a pre-trained text classifier, the text matching module is used to: For each candidate matching text, based on the semantic matching probability between the text query requirement and the candidate matching text, a pre-trained text classifier is used to cluster the candidate matching text to determine the category to which the candidate matching text belongs and the probability of belonging to the category; Based on the category and category probability of each candidate matching text, determining a target matching text that matches the text query requirement, and feeding back the target matching text to the user; When the text matching module is used to determine a target matching text that matches the text query requirement based on the category and category probability of each candidate matching text, the text matching module is used to: Based on the category belonging probability of each candidate matching text, sorting the multiple categories to which the multiple candidate matching texts belong, and determining the target category at the first priority; The candidate matching texts belonging to the target category are determined as target matching texts fed back to the user.

7. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate via the bus, and the machine-readable instructions are executed by the processor to execute the steps of the text retrieval method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the text retrieval method according to any one of claims 1 to 5 are executed.

Images