Text retrieval method, device and equipment, readable storage medium and program product

Abstract

The invention provides a text retrieval method, device and equipment, a readable storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the steps that a retrieval statement input by a user is input into a pre-trained first language model, and the retrieval intention complexity of the user is output; according to the complexity of the retrieval intention, a target text corresponding to the retrieval statement is subjected to self-adaptive partitioning processing, a plurality of text blocks are obtained, and the higher the complexity of the retrieval intention is, the smaller the number of clauses in the text blocks is; for each text block, respectively calculating the similarity between the text block and the retrieval statement to obtain a similarity score; and according to the complexity of the retrieval intention and the similarity score of the text block, generating a retrieval result corresponding to the retrieval statement. By adopting the embodiment of the invention, not only can the coverage range of the retrieval result be ensured, but also the accuracy and correlation of the retrieval result can be ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly to a text retrieval method, device, equipment, readable storage medium and program product. BACKGROUND

[0002] In the retrieval and intelligent question-answering scenarios, the system needs to obtain the corresponding results in the text according to the user's question sentence and feed back to the user. In the case of long text (chapter-level text such as reports, papers, books), how to balance the retrieval efficiency and retrieval accuracy is a dilemma.

[0003] At present, the retrieval scheme in the prior art is to divide the long text into multiple text blocks, calculate the similarity between each text block and the user's retrieval question sentence, and feed back the text block with high similarity to the user as the retrieval result.

[0004] However, the prior art only considers the similarity and does not consider other factors, resulting in poor retrieval efficiency and retrieval accuracy. SUMMARY

[0005] The purpose of the present application is to provide a text retrieval method, device, equipment, readable storage medium and program product, which solves the problem of poor accuracy of retrieval results.

[0006] In order to solve the above technical problems, the present application provides a text retrieval method, comprising the following steps:

[0007] inputting the retrieval sentence input by the user into a pre-trained first language model to output the retrieval intention complexity of the user;

[0008] According to the retrieval intention complexity, the target text corresponding to the retrieval sentence is adaptively divided into multiple text blocks;

[0009] For each text block, the similarity between the text block and the retrieval sentence is calculated respectively to obtain a similarity score;

[0010] According to the retrieval intention complexity and the similarity score corresponding to the text block, the retrieval result corresponding to the retrieval sentence is generated.

[0011] Optionally, the adaptive block processing of the target text corresponding to the retrieval sentence according to the retrieval intention complexity to obtain multiple text blocks comprises:

[0012] Identify the sentence punctuation marks in the target text corresponding to the retrieval sentence;

[0013] cutting the target text into multiple sentences by matching the sentence punctuation and the space or line break after the sentence punctuation by using a regular expression;

[0014] According to the search intent complexity, the number of sentences in each text block is determined, and adaptive blocking processing is performed on the target text to obtain multiple text blocks. When the complexity of the search intent is high, each text block contains fewer sentences. Correspondingly, when the complexity of the search intent is low, each text block contains more sentences.

[0015] Optionally, for each text block, the similarity between the text block and the search sentence is calculated to obtain a similarity score, including:

[0016] The text block is input into a pre-trained second language model to obtain a semantic information vector of the text block;

[0017] The search sentence is input into the second language model to obtain a semantic information vector of the search sentence;

[0018] For each text block, the similarity between the semantic information vector of the text block and the semantic information vector of the search sentence is calculated using cosine similarity to obtain a similarity score.

[0019] Optionally, the search result corresponding to the search sentence is generated according to the search intent complexity and the similarity score corresponding to the text block, including:

[0020] Based on the search intent complexity, a first preset number of text blocks are selected in order of the similarity score from high to low to determine target text blocks. When the complexity of the search intent is high, the first preset number of selected text blocks is large. Correspondingly, when the complexity of the search intent is low, the first preset number of selected text blocks is small.

[0021] When the context information of the target text block is complete, the context information integrity score of the target text block and the keyword matching score between the target text block and the search sentence are obtained;

[0022] According to the similarity score, the context information integrity score, and the keyword matching score, the target text block is scored to obtain a target score of each target text block. When the target score is high, the similarity between the target text block and the search sentence is high. Correspondingly, when the target score is low, the similarity between the target text block and the search sentence is low.

[0023] Selecting, in order from high to low of the target scores, a second preset number of the target text blocks, and determining the retrieval results corresponding to the retrieval sentence.

[0024] Optionally, the method further comprises the following steps:

[0025] In the case that the context information of the target text block is incomplete, expanding the context information of the target text block until a preset condition is met, and determining that the context information of the expanded target text block is complete;

[0026] The preset condition comprises at least one of the following:

[0027] The context information of the expanded target text block is complete;

[0028] The cumulative number of expanded sentences of the target text block meets a third preset number;

[0029] The number of sentences in the expanded target text block increases by a first preset percentage compared to the number of sentences in the target text block before expansion;

[0030] The target text block has been expanded to the boundary of the target text.

[0031] Optionally, the method further comprises the following steps:

[0032] Extracting keywords from the retrieval sentence and the target text block to obtain the keywords of the retrieval sentence and the keywords of the target text block;

[0033] In the case that the keywords of the retrieval sentence and the keywords of the target text block meet a matching condition, and the pronoun in the target text block has a referent in the target text block, it is determined that the context information of the target text block is complete;

[0034] In the case that the keywords of the retrieval sentence and the keywords of the target text block do not meet the matching condition, or the pronoun in the target text block has no referent in the target text block, it is determined that the context information of the target text block is incomplete.

[0035] The embodiment of the application further provides a text retrieval device, comprising:

[0036] A first analysis module is configured to input a retrieval sentence input by a user into a pre-trained first language model, and output retrieval intention complexity of the user;

[0037] A first processing module is configured to perform adaptive block processing on a target text corresponding to the retrieval sentence according to the retrieval intention complexity, and obtain a plurality of text blocks.

[0038] The first calculation module is configured to calculate the similarity between each text block and the search statement to obtain a similarity score.

[0039] The first generation module is configured to generate a search result corresponding to the search statement according to the search intention complexity and the similarity score corresponding to the text block.

[0040] The embodiment of the present application also provides a network device, which comprises a processor, a memory and a program stored in the memory and executable on the processor, and the program implements the text search method according to any one of the above when executed by the processor.

[0041] The embodiment of the present application also provides a readable storage medium, which comprises a program stored in the readable storage medium, and the program implements the steps of the text search method according to any one of the above when executed by a processor.

[0042] The embodiment of the present application also provides a computer program product, which comprises computer instructions, and the computer instructions implement the steps of the text search method according to any one of the above when executed by a processor.

[0043] The above technical solution of the present application has at least one of the following advantages:

[0044] In the above scheme, the complexity of the user's search intention is considered in the search process. First, the search statement input by the user is input into a pre-trained first language model, and the complexity of the user's search intention is output. Then, according to the complexity of the search intention, the target text corresponding to the search statement is adaptively segmented to obtain a plurality of text blocks. That is, the granularity of the segmentation of the target text is dynamically adjusted based on the complexity of the user's search intention. Finally, according to the complexity of the search intention and the similarity score between the text block and the search statement, the search result corresponding to the search statement is generated. Based on the above technical features, the problem of low search efficiency and search accuracy caused by the single factor considered in the prior art is solved. The complexity of the user's search intention is additionally considered in the present scheme, which can not only ensure the coverage of the search result, but also improve the search efficiency and search accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The flowchart of the text search method of the embodiment of the present application;

[0046] Figure 2 The structural diagram of the text search device of the embodiment of the present application.

[0047] Explanation of reference signs:

[0048] 201: first analysis module, 202: first processing module, 203: first calculation module, 204: first generation module. DETAILED DESCRIPTION

[0049] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention and not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0050] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects, and are not used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of this application can be implemented in an order other than that illustrated or described herein, and that the objects distinguished by "first," "second," and the like are generally of the same type, and do not limit the number of objects; for example, the first object can be one or more. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects connected are in an "or" relationship.

[0051] like Figure 1 As shown, an embodiment of the present invention provides a text retrieval method, comprising the following steps:

[0052] Step S101: inputting a search sentence input by a user into a pre-trained first language model, and outputting the complexity of the user's search intention;

[0053] In step S101, the search sentence input by the user is input into the pre-trained first language model using natural language processing technology to determine the complexity of the search intent. The complexity of the search intent in the embodiment of the present invention includes complex, medium, and simple, but is not limited to this. Specifically:

[0054] Input the search sentence into the pre-trained first language model and obtain the classification label corresponding to the complexity of the user's search intent output by the first language model. The first language model can be a Robustly Optimized Bidirectional Encoder Representations from Transformers Pretraining Approach (RoBERTa) model, and the class labels include complex, medium, and simple. The following describes the training process of the first language model:

[0055] 1) Obtain training samples, and add classification labels to the training samples.

[0056] Collect training samples, i.e., a large number of search sentences, and add classification labels corresponding to search intent complexity to each search sentence. The classification labels are specifically complex, medium, and simple. The search sentences can be obtained from public data sets or user logs.

[0057] 2) Train the first language model using the training samples and their labeled classification labels.

[0058] Input the training samples in the previous step into the first language model to obtain the recognition results output by the first language model. Adjust the model parameters of the first language model according to the recognition results and the classification labels of the training samples to achieve pre-training of the first language model.

[0059] Step S102, according to the search intent complexity, the target text corresponding to the search sentence is adaptively processed, and a plurality of text blocks are obtained.

[0060] In step S102, the number of sentences included in each text block is determined according to the search intent complexity of the user, and the target text is processed based on the number.

[0061] Step S103, for each text block, the similarity between the text block and the search sentence is calculated to obtain a similarity score.

[0062] In step S103, for each segmented text block, the text similarity algorithm (such as cosine similarity, Jaccard similarity, etc.) is used to match the user's search sentence, and the similarity score S sim (Q,T i ) of each text block relative to the search sentence is calculated.

[0063] For example, using the cosine similarity algorithm:

[0064] S sim (Q,T i )=CosineSimilarity(Embedding(Q);Embedding(T i ));

[0065] Where Embedding(Q) is the vector representation of the search sentence, and Embedding(T i ) is the vector representation of the text block.

[0066] Step S104, according to the search intent complexity and the similarity score corresponding to the text block, a search result corresponding to the search sentence is generated.

[0067] In the embodiment of the present application, first, the pre-trained first language model is used to analyze the user input search sentence to obtain the search intention complexity of the user; then, the granularity of the segmentation of the target text is dynamically adjusted according to the complexity of the user search intention, and the target text is further processed in blocks to obtain a plurality of text blocks, wherein the higher the search intention complexity, the smaller the segmentation granularity, and the fewer the number of sentences in the text block, so as to more accurately locate the information; the higher the search intention complexity, the greater the segmentation granularity, and the more the number of sentences in the text block, so as to improve the processing efficiency. Based on the above segmentation strategy, the search accuracy and search efficiency can be considered, and finally, the search result corresponding to the search sentence is generated according to the search intention complexity and the similarity score between the text block and the search sentence, which can ensure the coverage of the search result and the accuracy and relevance of the search result.

[0068] Optionally, the adaptive block processing of the target text corresponding to the search sentence according to the search intention complexity comprises:

[0069] Identifying the sentence punctuation marks in the target text corresponding to the search sentence;

[0070] Segmenting the target text into a plurality of sentences by matching the sentence punctuation marks and the spaces or line breaks after the sentence punctuation marks through a regular expression.

[0071] According to the search intention complexity, the number of sentences in each text block is determined, and the target text is adaptively processed in blocks to obtain a plurality of text blocks, wherein the higher the complexity of the search intention, the fewer the number of sentences contained in each text block, and correspondingly, the lower the complexity of the search intention, the more the number of sentences contained in each text block.

[0072] In the embodiment of the present application, the division mode of the text block in step S102 is described:

[0073] First, the sentence punctuation marks in the target text are identified, wherein the common sentence punctuation marks include period (.), question mark (?), exclamation mark (!) and the like. These punctuation marks usually mark the end of a sentence and are an important basis for dividing the target text.

[0074] Second, the target text is segmented based on a regular expression.

[0075] The target text is segmented into individual sentences, that is, sentences, by matching the sentence punctuation marks and the spaces or line breaks after the sentence punctuation marks through a regular expression. In this way, the target text can be accurately divided into individual sentence units.

[0076] Thirdly, the number of sentences in each text block is determined according to the complexity of the search intention, and adaptive blocking processing is performed on the target text to obtain a plurality of text blocks. If the complexity of the search intention is high, the number of sentences contained in each text block is small, and correspondingly, if the complexity of the search intention is low, the number of sentences contained in each text block is large. Here is an example:

[0077] 1) If the complexity of the search intention is complex, the text is blocked according to 2 sentences as a group.

[0078] The search question with complex intention usually involves multi-level information needs, and more fine-grained text blocks can ensure high-precision information retrieval. Reducing the number of sentences contained in the text block can improve the accuracy of information matching and avoid dilution of key information by redundant content.

[0079] 2) If the complexity of the search intention is medium, the text is blocked according to 4 sentences as a group.

[0080] The search question with medium difficulty needs to analyze multiple related information, so a balanced blocking strategy is adopted. 4 sentences as a group can provide relatively compact text blocks while ensuring sufficient context information, which is suitable for medium complexity requirements.

[0081] 3) If the complexity of the search intention is simple, the text is blocked according to 6 sentences as a group.

[0082] The search question with simple difficulty usually involves only a single or small number of information points, and larger blocking can provide more context to ensure that relevant information is not missed in simple queries.

[0083] Optionally, for each text block, the similarity between the text block and the search sentence is calculated to obtain a similarity score, including:

[0084] The text block is input into a pre-trained second language model to obtain a semantic information vector of the text block;

[0085] The search sentence is input into the second language model to obtain a semantic information vector of the search sentence;

[0086] For each text block, the similarity between the semantic information vector of the text block and the semantic information vector of the search sentence is calculated using cosine similarity to obtain a similarity score.

[0087] In the embodiment of the application, the method for obtaining the similarity score of the text block and the search sentence in step S103 is specifically described:

[0088] In a first step, a pre-trained second language model is selected as an embedding model, such as a Bidirectional Encoder Representations from Transformers (BERT) model and a Generative Pre-trained Transformer (GPT) model, which can capture context information and generate a context embedding representation of a sentence.

[0089] In a second step, each text block is converted into a vector representation based on the embedding model selected in the first step, as follows:

[0090] 1) Input text block: input the text block into the selected embedding model.

[0091] 2) Extract embedding: obtain a semantic information vector of the text block through the embedding model, which represents the semantic information of the entire input text block.

[0092] In a third step, the similarity between the text block and the search sentence is determined based on the semantic information vector of the text block and the semantic information vector of the search sentence, as follows:

[0093] 1) Obtain the semantic information vector of the search sentence through the selected embedding model.

[0094] 2) Calculate the similarity score between the text block and the search sentence using a text similarity algorithm.

[0095] In the embodiment of the present application, the Cosine Similarity algorithm is taken as an example, and the Cosine Similarity formula is as follows:

[0096]

[0097] Wherein, A represents the semantic information vector of the search sentence, and B represents the semantic information vector of the text block.

[0098] Optionally, the generating the search result corresponding to the search sentence according to the search intent complexity and the similarity score corresponding to the text block comprises:

[0099] According to the search intent complexity, the first preset number of text blocks are selected in descending order of the similarity score to determine the target text block, wherein the search intent complexity is high, the first preset number of selected text blocks is large, and correspondingly, the search intent complexity is low, the first preset number of selected text blocks is small.

[0100] In the case that the context information of the target text block is complete, the context information integrity score of the target text block and the keyword matching score between the target text block and the search sentence are obtained;

[0101] The target text block is scored according to the similarity score, the context information integrity score and the keyword matching score, and a target score of each target text block is obtained, wherein the target score is high, corresponding to high similarity between the target text block and the search sentence, and correspondingly, the target score is low, corresponding to low similarity between the target text block and the search sentence;

[0102] The second preset number of target text blocks are selected in the order from high to low according to the target score, and the search result corresponding to the search sentence is determined.

[0103] In the embodiment of the application, first, the number of target text blocks required is determined according to the complexity of the search intention. Specifically, the higher the complexity of the search intention, the more the number of target text blocks, and correspondingly, the lower the complexity of the search intention, the less the number of target text blocks. Here, examples are given as follows:

[0104] a) For a search sentence with simple complexity of search intention, the text blocks with similarity scores in the top 3 are determined as target text blocks.

[0105] b) For a search sentence with medium complexity of search intention, the text blocks with similarity scores in the top 6 are determined as target text blocks.

[0106] c) For a search sentence with complex complexity of search intention, the text blocks with similarity scores in the top 9 are determined as target text blocks.

[0107] Based on the complexity of the search intention, the number of sentences contained in the text block and the number of target text blocks selected are determined, which ensures that the search result can cover the breadth of relevant information and also ensures the accuracy and relevance of the information. Through flexible blocking strategy and target text block determination strategy, the needs of search sentences with different complexities are met, and the overall efficiency and effect of the search system are improved.

[0108] Then, in order to ensure the integrity of the search result, the integrity of the target text block determined in the previous step needs to be judged. In the case that the target text block is complete, the context information integrity score of the target text block and the keyword matching score between the target text block and the search sentence are obtained, and the specific formula is as follows:

[0109]

[0110] Wherein, S key (Q,T i) is the keyword matching score between the target text block and the search query, Keywords(Q) represents the set of keywords extracted from the search query, Keywords(T i ) represents the set of keywords extracted from the target text block.

[0111] S context (T i )=SemanticCoherence(T i );

[0112] where S context (T i ) is the context information integrity score of the target text block, and the SemanticCoherence function is calculated by analyzing the semantic coherence of the target text block and its context, for example, based on the sentence vector similarity of the pre-trained language model to calculate the context information integrity score.

[0113] Secondly, according to the similarity score S sim (Q,T I ), the context information integrity score S context (T i ) and the keyword matching score S key (Q,T i ), the target text block is scored to obtain the target score S final of each target text block, and the specific formula is as follows:

[0114] S final (T i )=αS sim (Q,T i )+βS key (Q,T i )+γS context (T i );

[0115] where α, β, γ are weight parameters that adjust the influence of each scoring item, which need to be adjusted according to actual application. The specific adjustment strategy can refer to:

[0116] a) According to different application scenarios and search targets, some scoring items may need higher weights.

[0117] User's emphasis on similarity: In some scenarios, users may be more concerned about the direct similarity between the search results and the query. At this time, the weight of α can be increased to make S sim (Q,T i ) occupy a larger proportion in the final score.

[0118] Information integrity requirement: If the application scenario emphasizes the information integrity of the retrieval result (such as legal documents, medical research), the weight of γ should be increased to ensure that the context integrity has a greater impact on the score.

[0119] Key content matching: If the matching of keywords in the user query is the core requirement (such as e-commerce search, question and answer system), the weight of β can be increased to prioritize the matching degree of key content.

[0120] b) Initial setting and adjustment process of weight.

[0121] Initial setting of weight according to experience or domain knowledge, and then fine-tuning:

[0122] Initial empirical value: In the absence of sufficient data support, the initial weight value can be set by relying on the opinions of domain experts. For example, assume that in technical document retrieval, α = 0.5, β = 0.3, γ = 0.2, to emphasize similarity and key content matching.

[0123] Adjustment process: Gradually adjust the weight, such as adjusting 10%-20% each time, and observe the impact of the final score on the result sorting.

[0124] Finally, according to the order of the target score from high to low, the second preset number of target text blocks are selected as the retrieval results corresponding to the retrieval statement, ensuring that the user obtains not only accurate retrieval results, but also necessary context information.

[0125] Optionally, the method further comprises the following steps:

[0126] In the case where the context information of the target text block is incomplete, expand the context information of the target text block until the preset condition is met, and determine the context information of the expanded target text block to be complete;

[0127] Wherein, the preset condition includes at least one of the following:

[0128] The context information of the expanded target text block is complete;

[0129] The cumulative expansion of the target text block satisfies the third preset number;

[0130] The number of sentences in the expanded target text block increases by a first preset percentage compared to the number of sentences in the target text block before expansion;

[0131] The target text block has been expanded to the boundary of the target text.

[0132] In the embodiment of the present application, if the target text block lacks complete context information, context expansion is performed on the target text block. Specifically, the context window technology is used to expand the target text block by 1-3 sentences forward or backward to supplement the key context information, thereby forming a new and larger target text block, and ensuring the completeness of the context of the target text block.

[0133] In the context expansion process, an iterative expansion combined with termination conditions is adopted.

[0134] In each expansion, 1 sentence is added forward or backward, and immediately after the expansion, it is determined whether the context information of the target text block is complete, that is, whether the missing important keywords in the target text block are covered and the pronoun reference is clear (by checking whether there is an entity name matching the pronoun in the expanded text block), if both conditions are met, it is determined that the context information of the target text block is complete.

[0135] If the completeness condition is still not met, the expansion continues until any of the following termination conditions is met:

[0136] The cumulative number of expanded sentences reaches a third preset number;

[0137] The total number of sentences in the target text block exceeds the first preset percentage of the original segmentation strategy (for example, the first preset percentage is 50%, and when the total number of sentences in the original target text block is 2, the maximum expansion is 3 sentences);

[0138] The target text block has been expanded to the start or end boundary of the target text.

[0139] After the termination condition is met, it is determined that the context information of the expanded target text block is complete, and the target text block is scored to obtain a target score, and the expansion process is not repeated to avoid infinite recursion.

[0140] Optionally, the method further comprises the following steps:

[0141] Keywords are extracted from the search sentence and the target text block to obtain keywords of the search sentence and keywords of the target text block;

[0142] In the case that the keywords of the search sentence and the keywords of the target text block satisfy the matching condition, and the pronoun in the target text block has a reference object in the target text block, it is determined that the context information of the target text block is complete;

[0143] In the case that the keywords of the search sentence and the keywords of the target text block do not satisfy the matching condition, or the pronoun in the target text block has no reference object in the target text block, it is determined that the context information of the target text block is not complete.

[0144] In the embodiment of the present application, by detecting the matching degree of the keywords in the user's search sentence and the keywords of the target text block, it is judged whether the target text block contains complete context information, i.e. whether the target text block has integrity, and the specific operation is as follows:

[0145] Firstly, the corresponding keywords are extracted in the user's search sentence and the target text block by using the Jieba segmentation library combined with the TF-IDF (Term Frequency-Inverse Document Frequency) algorithm.

[0146] For the search sentence, firstly, the Jieba segmentation library is used to perform segmentation processing on the search sentence; then, the TF-IDF value of each segmentation in the search sentence is calculated; finally, based on the TF-IDF value of each segmentation in the search sentence, the keywords in the search sentence are determined.

[0147] For the target text block, firstly, the Jieba segmentation library is used to perform segmentation processing on the target text block; then, the TF-IDF value of each segmentation in the target text block is calculated; finally, based on the TF-IDF value of each segmentation in the target text block, the keywords in the target text block are determined.

[0148] Among them, the Jieba segmentation tool can realize the calculation of Term Frequency (TF) and Inverse Document Frequency (IDF) without manual calculation of these values. And the Jieba segmentation tool will automatically get specific probability values based on the statistical results of common corpus, and sort according to the TF-IDF value of each word. In the extraction of keywords, the Jieba segmentation tool returns the top N words with the highest TF-IDF value by default. In the embodiment of the present application, preferably, all the keyword results of the search sentence are retained, and the top 20% of the keyword results of the target text block are retained, but the present application does not limit this.

[0149] Secondly, whether the target text block has integrity is judged according to whether the keywords in the search sentence and the keywords of the target text block meet the matching conditions and whether the pronouns in the target text block have definite referents. In the case that the matching conditions are met and the pronouns (such as "he", "they", etc.) in the target text block have referents in the target text block, it is determined that the context information of the target text block is complete. In the case that the matching conditions are not met or the pronouns (such as "he", "they", etc.) in the target text block have referents in the target text block, it is determined that the context information of the target text block is incomplete. Among them, the matching conditions include that the missing proportion of the keywords in the target text block relative to the important keywords in the search sentence is less than a first threshold, wherein the important keywords are determined by the TF-IDF value, and the specific operation is as follows:

[0150] In the retrieval sentence corresponding to the keyword, the TF-IDF value of the keyword in the top 50% of all keywords (i.e., the top 50% of the importance keywords) is determined as an important keyword. Among them, the keyword belongs to a noun, a verb or a professional term (filtered by part-of-speech tagging, for example, using the posseg module of jieba for part-of-speech analysis).

[0151] The determination of whether the important keywords in the retrieval sentence and the keywords of the target text block satisfy the matching condition is as follows:

[0152] The missing proportion of the important keywords of the retrieval sentence that are missing in the target text block is obtained, and the formula is as follows:

[0153]

[0154] Among them, K(Q) is an important keyword in the retrieval sentence, K(D) is an important keyword missing in the target text block relative to the retrieval sentence, and in the case where the missing proportion is less than the first threshold, it is determined that the two satisfy the matching condition, otherwise, it does not satisfy the matching condition.

[0155]

[0156] As shown in Figure 2 the embodiment of the present application also provides a text retrieval device, comprising:

[0157] The first analysis module 201 is used for inputting the retrieval sentence input by the user into the pre-trained first language model, and outputting the retrieval intention complexity of the user;

[0158] The first processing module 202 is used for performing adaptive block processing on the target text corresponding to the retrieval sentence according to the retrieval intention complexity, and obtaining a plurality of text blocks;

[0159] The first calculation module 203 is used for calculating the similarity between each text block and the retrieval sentence respectively, and obtaining a similarity score;

[0160] The first generation module 204 is used for generating a retrieval result corresponding to the retrieval sentence according to the retrieval intention complexity and the similarity score corresponding to the text block.

[0161] Optionally, the first processing module 202 comprises:

[0162] The first identification unit is used for identifying the sentence punctuation in the target text corresponding to the retrieval sentence;

[0163] The first processing unit is configured to divide the target text into multiple sentences by matching the punctuation mark and the space or line break after the punctuation mark according to a regular expression.

[0164] The second processing unit is configured to perform adaptive block processing on the target text according to the search intent complexity to obtain multiple text blocks, wherein the search intent complexity is high, and each text block contains a small number of sentences, and correspondingly, the search intent complexity is low, and each text block contains a large number of sentences.

[0165] Optionally, the first calculation module 203 comprises:

[0166] The third processing unit is configured to input the text block into a pre-trained second language model to obtain a semantic information vector of the text block.

[0167] The fourth processing unit is configured to input the search sentence into the second language model to obtain a semantic information vector of the search sentence.

[0168] The first calculation unit is configured to calculate the similarity between the semantic information vector of the text block and the semantic information vector of the search sentence by using cosine similarity for each text block to obtain a similarity score.

[0169] Optionally, the first generation module 204 comprises:

[0170] The first selection unit is configured to select a first preset number of text blocks in a descending order of the similarity score based on the search intent complexity, and determine the selected text blocks as target text blocks, wherein the first preset number of selected text blocks is large when the search intent complexity is high, and correspondingly, the first preset number of selected text blocks is small when the search intent complexity is low.

[0171] The first acquisition unit is configured to acquire a context information integrity score of the target text block and a keyword matching score between the target text block and the search sentence when the context information of the target text block is complete.

[0172] The first scoring unit is configured to score the target text block according to the similarity score, the context information integrity score and the keyword matching score to obtain a target score of each target text block, wherein the target score is high when the similarity between the target text block and the search sentence is high, and correspondingly, the target score is low when the similarity between the target text block and the search sentence is low.

[0173] The first determining unit is configured to select a second preset number of the target text blocks in a descending order of the target scores, and determine the search results corresponding to the search statement.

[0174] Optionally, the first generating module 204 further includes:

[0175] The first expanding unit is configured to expand the context information of the target text block until a preset condition is met in a case where the context information of the target text block is incomplete, and determine that the context information of the target text block after expansion is complete.

[0176] The preset condition includes at least one of the following:

[0177] The context information of the target text block after expansion is complete.

[0178] The number of expanded sentences of the target text block meets a third preset number.

[0179] The number of sentences in the target text block after expansion increases by a first preset percentage compared to the number of sentences in the target text block before expansion.

[0180] The target text block has been expanded to the boundary of the target text.

[0181] Optionally, the apparatus further includes:

[0182] The first extracting module is configured to extract keywords from the search statement and the target text block, and obtain the keywords of the search statement and the keywords of the target text block.

[0183] The first determining module is configured to determine that the context information of the target text block is complete in a case where the keywords of the search statement and the keywords of the target text block meet a matching condition, and pronouns in the target text block have referents in the target text block.

[0184] In a case where the keywords of the search statement and the keywords of the target text block do not meet the matching condition, or the pronouns in the target text block do not have referents in the target text block, it is determined that the context information of the target text block is incomplete.

[0185] It should be noted that the embodiments of the apparatus correspond to the embodiments of the above method, and all implementation manners in the embodiments of the above method are applicable to the embodiments of the apparatus, and can achieve the same technical effects.

[0186] The embodiment of the present application also provides a network device, comprising a processor, a memory and a program stored in the memory and executable on the processor, the program implements the text retrieval method according to any one of the above when executed by the processor, and achieves the same technical effects, and details are not repeated here to avoid repetition.

[0187] The embodiment of the present application also provides a readable storage medium, comprising a program stored in the readable storage medium, the program implements the steps of the text retrieval method according to any one of the above when executed by a processor, and achieves the same technical effects, and details are not repeated here to avoid repetition. The computer readable storage medium is, for example, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk.

[0188] The embodiment of the present application also provides a computer program product, comprising computer instructions, the computer instructions implement the steps of the text retrieval method according to any one of the above when executed by a processor, and achieve the same technical effects, and details are not repeated here to avoid repetition.

[0189] It should be noted that, in this document, the relationship terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0190] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A text retrieval method, characterized in that: The following steps are involved: Inputting a search sentence input by a user into a pre-trained first language model, and outputting the complexity of the user's search intention; According to the complexity of the search intent, the target text corresponding to the search statement is adaptively divided into blocks to obtain multiple text blocks; For each of the text blocks, respectively calculating the similarity between the text block and the search statement to obtain a similarity score; A retrieval result corresponding to the retrieval statement is generated according to the retrieval intention complexity and the similarity score corresponding to the text block.

2. The text retrieval method according to claim 1, characterized in that The adaptively segmenting the target text corresponding to the search statement according to the complexity of the search intent to obtain multiple text blocks includes: Identifying clause punctuation marks in the target text corresponding to the search statement; Matching the sentence punctuation marks and the space characters or line breaks following the sentence punctuation marks by regular expressions, thereby dividing the target text into multiple sentences; The number of sentences in each text block is determined according to the complexity of the retrieval intention, and the target text is adaptively segmented to obtain multiple text blocks. Among them, if the complexity of the retrieval intention is high, the number of sentences contained in each text block is small, and correspondingly, if the complexity of the retrieval intention is low, the number of sentences contained in each text block is large.

3. The text retrieval method according to claim 1, wherein: For each of the text blocks, respectively calculating the similarity between the text block and the search statement to obtain a similarity score includes: Inputting the text block into a pre-trained second language model to obtain a semantic information vector of the text block; Inputting the search sentence into the second language model to obtain a semantic information vector of the search sentence; For each of the text blocks, the similarity between the semantic information vector of the text block and the semantic information vector of the search sentence is calculated using cosine similarity to obtain a similarity score.

4. The text retrieval method according to claim 1, wherein: Generating a search result corresponding to the search statement according to the search intent complexity and the similarity score corresponding to the text block includes: Based on the complexity of the search intent, a first preset number of the text blocks are selected in descending order of the similarity scores and determined as target text blocks, wherein the higher the complexity of the search intent, the more the first preset number of the text blocks are selected, and correspondingly, the lower the complexity of the search intent, the fewer the first preset number of the text blocks are selected; When the context information of the target text block is complete, obtaining a context information completeness score of the target text block and a keyword matching score between the target text block and the search statement; Scoring the target text block according to the similarity score, the context information completeness score, and the keyword matching score to obtain a target score for each target text block, wherein a high target score corresponds to a high similarity between the target text block and the search statement, and correspondingly, a low target score corresponds to a low similarity between the target text block and the search statement; A second preset number of the target text blocks are selected in descending order of the target scores and determined as the search results corresponding to the search statement.

5. The text retrieval method according to claim 4, characterized in that: The method further comprises the following steps: In the case where the context information of the target text block is incomplete, expanding the context information of the target text block until a preset condition is satisfied, and determining that the context information of the target text block after expansion is complete; The preset conditions include at least one of the following: The context information of the expanded target text block is complete; The cumulative number of expanded clauses in the target text block satisfies a third preset number; The number of sentences in the target text block after expansion is increased by a first preset percentage compared to the number of sentences in the target text block before expansion; The target text block has been extended to the boundary of the target text.

6. The text retrieval method according to any one of claims 4 or 5, characterized in that: The method further comprises the following steps: Performing keyword extraction on the search statement and the target text block to obtain keywords of the search statement and the target text block; If the keywords of the search statement and the keywords of the target text block meet a matching condition and the pronouns in the target text block have referents in the target text block, determining that the context information of the target text block is complete; If the keywords of the search statement and the keywords of the target text block do not meet the matching condition, or the pronouns in the target text block do not refer to an object in the target text block, it is determined that the context information of the target text block is incomplete.

7. A text retrieval device, characterized in that: include: A first analysis module is configured to input a search sentence input by a user into a pre-trained first language model and output the complexity of the user's search intention; A first processing module is configured to adaptively divide the target text corresponding to the search statement into blocks according to the complexity of the search intent to obtain a plurality of text blocks; A first calculation module is configured to calculate the similarity between each text block and the search statement to obtain a similarity score; The first generating module is used to generate a search result corresponding to the search statement according to the complexity of the search intention and the similarity score corresponding to the text block.

8. A network device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program implements the text retrieval method according to any one of claims 1 to 6 when executed by the processor.

9. A readable storage medium, characterized in that: include: The readable storage medium stores a program, and when the program is executed by a processor, the steps of the text retrieval method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the text retrieval method according to any one of claims 1 to 6.

Images