Semantic search method, device and equipment, terminal equipment and storage medium

Abstract

The invention discloses a semantic search method, device and equipment, terminal equipment and a storage medium. The method comprises the following steps: determining a complexity level of a to-be-retrieved question; determining the length of a context window of each statement in a knowledge document according to the complexity level of the to-be-retrieved question; the statements are divided according to sentence punctuation marks; determining a target number of preceding sentences and following sentences corresponding to each sentence according to the length of the context window; and performing semantic search on the to-be-retrieved question according to the target number of the preceding sentences and the following sentences to obtain a retrieval result corresponding to the to-be-retrieved question. According to the semantic search method provided by the embodiment of the invention, the complexity of the to-be-retrieved question is graded, and the length of the context window of each statement in the knowledge document corresponding to different complexity grades can be determined, so that a retrieval result with relatively high accuracy is obtained, and the problem of relatively low accuracy of a semantic search result in the prior art is solved.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and specifically to a semantic search method, apparatus, device, terminal device and storage medium. Background Art

[0002] Semantic search technology aims to understand the user's query intent and provide results relevant to that intent, rather than simply matching keywords. It relies on natural language processing (NLP) and artificial intelligence (AI) technologies to gain a deeper understanding of user queries. Unlike traditional keyword search, semantic search considers factors such as query context, synonyms, and sentiment to provide more accurate and targeted search results.

[0003] In existing technologies, open-source application development frameworks are typically used to load reference knowledge documents in a specified format. These documents are then segmented into multiple text blocks of specified sizes, or regular expression matching is used to segment the documents into multiple text blocks based on paragraphs. This fails to guarantee semantic similarity and integrity between the text blocks, resulting in poor search results accuracy. Summary of the Invention

[0004] At least one embodiment of the present application provides a semantic search method, apparatus, device, terminal device, and storage medium for solving the problem of poor accuracy of semantic search results in the prior art.

[0005] In order to solve the above technical problems, this application is implemented as follows:

[0006] In a first aspect, an embodiment of the present application provides a semantic search method, comprising:

[0007] Determine the complexity level of the query to be retrieved;

[0008] Determining the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; dividing the sentence according to the clause punctuation marks;

[0009] Determining a target number of preceding and following sentences corresponding to each sentence according to the length of the context window;

[0010] A semantic search is performed on the question to be retrieved according to the target number of preceding sentences and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

[0011] Optionally, performing a semantic search on the question to be retrieved based on a target number of preceding and following sentences to obtain a search result corresponding to the question to be retrieved includes:

[0012] Determining a first similarity between each sentence in the knowledge document and a corresponding preceding sentence and a following sentence;

[0013] Segmenting the knowledge document according to the first similarity corresponding to each sentence in the knowledge document to obtain a plurality of semantic text blocks;

[0014] A search result corresponding to the question to be retrieved is determined according to the second similarity between each semantic text block and the question to be retrieved.

[0015] Optionally, determining a first similarity between each sentence in the knowledge document and a corresponding preceding sentence, and a following sentence, includes:

[0016] Input each of the sentences into the model to obtain an embedding feature corresponding to each of the sentences;

[0017] Determine the cosine similarity between the first embedded feature and the second embedded feature of each of the sentences as the first similarity;

[0018] The first embedded feature is an embedded feature corresponding to the preceding sentence of the sentence, and the second embedded feature is an embedded feature corresponding to the following sentence of the sentence.

[0019] Optionally, the knowledge document is segmented according to the first similarity corresponding to each sentence in the knowledge document to obtain a plurality of semantic text blocks, including:

[0020] Determine a local minimum value of the first similarity, where the local minimum value is a first similarity that is smaller than a first value and smaller than a second value, the first value is a first similarity to a preceding sentence of the sentence, and the second value is a first similarity to a following sentence of the sentence;

[0021] Determining a position index according to the local minimum value;

[0022] The knowledge document is segmented at the position index to obtain a plurality of semantic text blocks.

[0023] Optionally, determining a position index according to the local minimum value includes:

[0024] determining a valid threshold value of the local minimum;

[0025] Determining the local minimum value that is less than the validity threshold value as a valid local minimum value;

[0026] Determine that the statement corresponding to the effective local minimum value is the position index.

[0027] Optionally, determining a valid threshold of the local minimum value includes:

[0028] Determining the number of first sentences in the knowledge document, where the first sentences are sentences in the context window corresponding to the current sentence and the first similarity is greater than or equal to the current sentence;

[0029] The percentage of the number of the first sentences to the total number of all sentences in the knowledge document is determined as the effective threshold.

[0030] Optionally, the knowledge document is segmented at the position index to obtain a plurality of semantic text blocks, including:

[0031] In the case where the position index is one, all texts before the position index are obtained as a semantic text, and all texts after the position index are obtained as a semantic text;

[0032] In the case where there are multiple position indexes, the text between every two position indexes is determined to be a semantic text, all text before the first position index is determined to be a semantic text, and all text after the last position index is determined to be a semantic text; the first position index and the last position index are determined according to their position order in the knowledge document.

[0033] Optionally, determining the complexity level of the question to be retrieved includes:

[0034] Performing text normalization processing on the question to be retrieved;

[0035] Inputting the processed question to be retrieved into a language model to obtain a complexity level of the question to be retrieved;

[0036] Wherein, the language model is a model obtained through training of historical retrieval questions;

[0037] The complexity levels include: simple, medium, and complex.

[0038] In a second aspect, an embodiment of the present application provides a semantic search device, comprising:

[0039] A first determination module is used to determine the complexity level of the question to be retrieved;

[0040] A second determination module is configured to determine the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; the sentences are divided according to clause punctuation marks;

[0041] a third determining module, configured to determine a target number of preceding and following sentences corresponding to each sentence according to the length of the context window;

[0042] The retrieval module is configured to perform a semantic search on the question to be retrieved based on a target number of preceding sentences and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

[0043] In a third aspect, an embodiment of the present application provides a semantic search device, including a processor, wherein:

[0044] The processor is configured to determine the complexity level of the question to be retrieved;

[0045] Determining the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; dividing the sentence according to the clause punctuation marks;

[0046] Determining a target number of preceding and following sentences corresponding to each sentence according to the length of the context window;

[0047] A semantic search is performed on the question to be retrieved according to the target number of preceding sentences and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

[0048] In a fourth aspect, an embodiment of the present application provides a terminal device comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the semantic search method as described in the first aspect.

[0049] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the program implements the steps of the semantic search method described above.

[0050] In a sixth aspect, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the semantic search method described above.

[0051] Compared with the prior art, the semantic search method, apparatus, device, terminal device and storage medium provided in the embodiments of the present application, by grading the complexity of the query to be retrieved and determining the length of the context window of each sentence in the knowledge document, can determine the target number of preceding and following sentences corresponding to each sentence in the knowledge document of the query to be retrieved during the retrieval process; thereby, a semantic search is performed on the query to be retrieved based on the target number of preceding and following sentences. The semantic search method of the embodiments of the present application, by grading the complexity of the query to be retrieved, can determine the length of the context window of each sentence in the knowledge document corresponding to different complexity levels, thereby obtaining a retrieval result with higher accuracy, solving the problem of poor accuracy of semantic search results in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present application. The same reference symbols are used throughout the drawings to represent the same components. In the drawings:

[0053] Figure 1 A schematic diagram of the steps of the semantic search method according to an embodiment of the present application;

[0054] Figure 2 This is a structural diagram of a semantic search device according to an embodiment of the present application;

[0055] Figure 3 This is a structural diagram of a semantic search device according to an embodiment of the present application;

[0056] Figure 4 This is a schematic diagram of the structure of the terminal device according to an embodiment of the present application. DETAILED DESCRIPTION

[0057] The following describes exemplary embodiments of the present application in more detail with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0058] The terms "first", "second" etc. in the specification and claims of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments of the present application described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "comprise" and "have" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units need not be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or that are intrinsic to these processes, methods, products or equipment. "And / or" in the specification and claims represents at least one of the connected objects.

[0059] The following description provides examples and does not limit the scope, applicability, or configuration set forth in the claims. Changes may be made to the function and arrangement of the elements discussed without departing from the spirit and scope of this disclosure. The various examples may appropriately omit, substitute, or add various procedures or components. For example, the described methods may be performed in an order different from that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

[0060] As described in the background technology, in the prior art, when segmenting reference knowledge documents, they are either segmented into multiple text blocks of specified sizes, or segmented into multiple text blocks according to paragraphs using regular matching. The semantic similarity and semantic integrity between the individual text blocks cannot be guaranteed, resulting in poor accuracy of the final search results. In order to solve at least one of the above problems, an embodiment of the present application provides a semantic search method that can reduce or avoid the occurrence of the above situation and improve the accuracy of semantic search.

[0061] like Figure 1 As shown, a semantic search method provided in an embodiment of the present application includes the following steps:

[0062] Step 101, determining the complexity level of the question to be retrieved;

[0063] Step 102: determining the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; the sentences are divided according to clause punctuation marks;

[0064] Step 103, determining a target number of preceding and following sentences corresponding to each sentence according to the length of the context window;

[0065] Step 104 : performing a semantic search on the question to be retrieved based on the target number of preceding sentences and following sentences to obtain a search result corresponding to the question to be retrieved.

[0066] It should be noted that natural language processing technology is used to perform semantic analysis on the user's search questions and automatically classify them into one of the three categories: complex, medium, and simple;

[0067] The higher the complexity level of the search question, the longer the context window of each sentence in the knowledge document.

[0068] Optionally, the knowledge document is a reference knowledge document in a knowledge base.

[0069] The semantic search method provided in the embodiment of the present application, by grading the complexity of the retrieval questions and determining the length of the context window of each sentence in the knowledge document, can determine the target number of previous sentences and following sentences corresponding to each sentence in the knowledge document of the question to be retrieved during the retrieval process; thereby performing a semantic search on the question to be retrieved based on the target number of the previous sentences and the following sentences. The semantic search method provided in the embodiment of the present application, by grading the complexity of the retrieval questions, can determine the length of the context window of each sentence in the knowledge document corresponding to different complexity levels, thereby obtaining a retrieval result with higher accuracy, solving the problem of poor accuracy of semantic search results in the prior art.

[0070] Optionally, determining the complexity level of the question to be retrieved includes:

[0071] Performing text normalization processing on the question to be retrieved;

[0072] Inputting the processed question to be retrieved into a language model to obtain a complexity level of the question to be retrieved;

[0073] Wherein, the language model is a model obtained through training of historical retrieval questions;

[0074] The complexity levels include: simple, medium, and complex.

[0075] Optionally, performing text normalization processing on the question to be retrieved includes:

[0076] The question to be searched is subjected to uniform processing of full / half-width symbols and uniform processing of upper and lower case letters.

[0077] In the embodiment of the present application, the complexity level needs to be marked and determined first, specifically:

[0078] Data collection: Collect a large number of historical user retrieval questions and their corresponding complexity labels, for example, through publicly available datasets in the field or from actual user logs.

[0079] Data cleaning: processing the historical user search queries, including irrelevant information, punctuation, and case conversion;

[0080] Complexity annotation: Use a large model (such as GPT) to annotate the complexity of questions.

[0081] Secondly, the complexity level of the query to be retrieved is classified based on the fine-tuned model. Specifically:

[0082] Model selection: Use a pre-trained language model (e.g. RoBERTa);

[0083] Fine-tuning training: taking the historical retrieval questions as input and the complexity classification labels (simple, medium, and complex) as output, the pre-trained language model is trained to obtain a classification model;

[0084] Automatic classification: inputting the question to be retrieved into the classification model to obtain the complexity level of the question to be retrieved.

[0085] It should be noted that the sentences in the knowledge document need to be determined first, specifically including:

[0086] Identifying sentence punctuation marks in the knowledge document, such as a period, question mark, exclamation mark, etc.;

[0087] Sentences are segmented using regular expressions, for example, regular expressions are used to match segmentation punctuation marks and the spaces or line breaks following them, so as to segment the long text in the knowledge document into multiple sentences.

[0088] Optionally, the length of the context window of each sentence in the knowledge document is determined according to the complexity level of the question to be retrieved; which can be understood as:

[0089] The higher the complexity level of the question to be retrieved, the longer the length of the context window.

[0090] Optionally, determining a target number of preceding and following statements corresponding to each statement according to the length of the context window includes:

[0091] The current sentence is used as the window center of the context window, and a first number of the preceding sentences and a second number of the following sentences are determined according to the target number.

[0092] In the embodiment of the present application, for each of the statements, the first number is equal to the second number;

[0093] For example, for the complex question to be retrieved: the Window_size of the context window is set to 7 (i.e., the target number of previous sentences and following sentences corresponding to each sentence is 7); wherein, with the current sentence as the window center of the context window, the first number of the previous sentences is 3, and the number of the following sentences is 3;

[0094] 2) For the medium-difficulty question to be retrieved: the context window's Window_size is set to 5 (i.e., the target number of preceding and following sentences for each sentence is 5); wherein, with the current sentence as the center of the context window, the first number of preceding sentences is 2, and the number of following sentences is 2;

[0095] 3) For the question to be retrieved of simple difficulty: the Window_size of the context window is set to 3 (i.e., the target number of previous sentences and following sentences corresponding to each sentence is 3); wherein, the current sentence is used as the window center of the context window, the first number of the previous sentences is 1, and the number of the following sentences is 1.

[0096] It should be noted that the above is an example of the Window_size of the context window. It is sufficient to ensure that the Window_size of the context window corresponding to the complex level of the to-be-retrieved question, the Window_size of the context window corresponding to the medium level of the to-be-retrieved question, and the Window_size of the context window corresponding to the simple level of the to-be-retrieved question are gradually reduced.

[0097] The semantic search method of the embodiment of the present application determines the length of the context window of each sentence in the knowledge document by determining the complexity level of the file to be retrieved, thereby enhancing the semantic integrity between the text blocks obtained after cutting the knowledge document.

[0098] Optionally, performing a semantic search on the question to be retrieved based on a target number of preceding and following sentences to obtain a search result corresponding to the question to be retrieved includes:

[0099] Determining a first similarity between each sentence in the knowledge document and a corresponding preceding sentence and a following sentence;

[0100] Segmenting the knowledge document according to the first similarity corresponding to each sentence in the knowledge document to obtain a plurality of semantic text blocks;

[0101] A search result corresponding to the question to be retrieved is determined according to the second similarity between each semantic text block and the question to be retrieved.

[0102] It should be noted that the first similarity is a sentence spacing score.

[0103] The semantic search method of the embodiment of the present application can segment the knowledge document by determining the first similarity to obtain the semantic text block with semantic integrity; and determine the search result by the second similarity between the semantic text block and the question to be retrieved.

[0104] Optionally, determining a first similarity between each sentence in the knowledge document and a corresponding preceding sentence, and a following sentence, includes:

[0105] Input each of the sentences into the model to obtain an embedding feature corresponding to each of the sentences;

[0106] Determine the cosine similarity between the first embedded feature and the second embedded feature of each of the sentences as the first similarity;

[0107] The first embedded feature is an embedded feature corresponding to the preceding sentence of the sentence, and the second embedded feature is an embedded feature corresponding to the following sentence of the sentence.

[0108] Optionally, the preceding statement is a statement before the statement with the statement as the center in the context window; the following statement is a statement after the statement with the statement as the center in the context window.

[0109] In the embodiment of the present application, determining the embedded features corresponding to each of the sentences includes:

[0110] Each of the sentences is transformed using a trained language model (e.g., a BERT series model) to obtain an embedded feature (high-dimensional vector representation) for each sentence, specifically including:

[0111] Input sentence: Input each sentence into the BERT model;

[0112] Extract embedding: Extract the vector of the [CLS] tag output by the model as the vector representation S_E of the sentence.

[0113] In the embodiment of the present application, for a long text, it includes the following sentences [..., S[i-2], S[i-1], S[i], S[i+1], S[i+2], ...];

[0114] Taking the current sentence S[i] as an example, assuming that the length of the context window is 5, the previous sentence of the current sentence S[i] is (S[i-2], S[i-1]), and the following sentence is (S[i+1], S[i+2]);

[0115] The vector corresponding to the above statement is (S_E[i-2], S_E[i-1]), and the vector corresponding to the following statement is (S_E[i+1], S_E[i+2]);

[0116] The first embedding feature of the previous sentence is calculated by directly summing and averaging each vector within the Window_size range of the previous sentence (S_E[i-2], S_E[i-1]); the second embedding feature of the following sentence is calculated by directly summing and averaging each vector within the Window_size range of the following sentence (S_E[i-2], S_E[i-1]);

[0117] Then the first embedding feature Front_Embedding = (S_E[i-1]+S_E[i-2]) / 2;

[0118] The second embedding feature Behind_Embedding=(S_E[i]+S_E[i+1]+S_E[i+2]) / 3;

[0119] Optionally, the cosine similarity between the first embedded feature and the second embedded feature of each of the sentences is calculated as the first similarity Interval_Score(S[i]) by the following formula:

[0120] Interval_Score(S[i])=sim_score(Front_Embedding, Behind_Embedding).

[0121] Optionally, the knowledge document is segmented according to the first similarity corresponding to each sentence in the knowledge document to obtain a plurality of semantic text blocks, including:

[0122] Determine a local minimum value of the first similarity, where the local minimum value is a first similarity that is smaller than a first value and smaller than a second value, the first value is a first similarity to a preceding sentence of the sentence, and the second value is a first similarity to a following sentence of the sentence;

[0123] Determining a position index according to the local minimum value;

[0124] The knowledge document is segmented at the position index to obtain a plurality of semantic text blocks.

[0125] Optionally, the knowledge document is segmented at the position index to obtain a plurality of semantic text blocks, including:

[0126] In the case where the position index is one, all texts before the position index are obtained as a semantic text, and all texts after the position index are obtained as a semantic text;

[0127] In the case where there are multiple position indexes, the text between every two position indexes is determined to be a semantic text, all text before the first position index is determined to be a semantic text, and all text after the last position index is determined to be a semantic text; the first position index and the last position index are determined according to their position order in the knowledge document.

[0128] In an embodiment of the present application, for a long text in the knowledge document, the position index is determined to be the 5th sentence and the 9th sentence, then the 1st sentence to the 5th sentence (all text before the first index position) is a text block, the 6th sentence to the 9th sentence (sentences between the two position indexes) is the second text block, and the 10th sentence to the last sentence of the long text (all sentences after the last position index) is the last text block.

[0129] Optionally, determining the local minimum of the first similarity includes:

[0130] If the first similarity of a target sentence is smaller than the first similarity corresponding to the previous sentence and smaller than the first similarity corresponding to the subsequent sentence, then the first similarity corresponding to the target sentence is a local minimum.

[0131] For each long text, there are one or more local minimum values.

[0132] Optionally, determining a valid threshold of the local minimum value includes:

[0133] Determining the number of first sentences in the knowledge document, where the first sentences are sentences in the context window corresponding to the current sentence and the first similarity is greater than or equal to the current sentence;

[0134] The percentage of the number of the first sentences to the total number of all sentences in the knowledge document is determined as the effective threshold.

[0135] Optionally, determining a position index according to the local minimum value includes:

[0136] determining a valid threshold value of the local minimum;

[0137] Determining the local minimum value that is less than the validity threshold value as a valid local minimum value;

[0138] Determine that the statement corresponding to the effective local minimum value is the position index.

[0139] Optionally, the effective threshold of the local minimum is determined by the following formula:

[0140]

[0141] Among them, P(x) represents the percentage (effective threshold); x represents the first similarity of the current statement; Number of scores greater than or aqual to x represents the statements in the context window corresponding to the current statement whose first similarity is greater than or equal to the current statement; Total number of scores represents all statements in the knowledge document.

[0142] In the embodiment of the present application, for a long text sentence_list=[S[i-3], S[i-2], S[i-1], S[i], S[i+1], S[i+2], S[i+3]] in the knowledge document, the first similarity of each sentence is: interval_score_list=[0.3, 0.1, 0.2, 0.4, 0.7, 0.6, 0.8];

[0143] Then the local minimum values ​​are 0.1 and 0.6;

[0144] For S[i-3](0.3), the local minimum values ​​of S[i](0.4), S[i+1](0.7), S[i+2](0.6), and S[i+2](0.8) are greater than S[i-3](0.3). Therefore, the percentile value of S[i-3] is 4 / 7 = 47.14%. The percentile values ​​of other sentences can be calculated similarly; when the percentile value of a sentence is ≥ 95%, the local minimum corresponding to that sentence is a valid local minimum.

[0145] In an embodiment of the present application, for a long text in the knowledge document, the 5th statement and the 9th statement are statements corresponding to the valid local minimum, then the 1st to 5th statements are a text block, the 6th to 9th statements are the second text block, and the 10th statement to the last statement of the long text is the last text block.

[0146] The semantic search method of the embodiment of the present application determines the effective threshold by percentile, and can adaptively select an effective segmentation standard based on the data distribution, avoiding the subjectivity and error caused by manually setting the effective threshold, and improving the accuracy and flexibility of text semantic segmentation.

[0147] Optionally, determining a search result corresponding to the question to be retrieved based on the second similarity between each semantic text block and the question to be retrieved includes:

[0148] performing vectorization processing on the question to be retrieved to obtain a target vector of the question to be retrieved;

[0149] Calculating the second similarity between the target vector and the vector corresponding to each text block in the knowledge document;

[0150] The text block with the second similarity greater than a preset threshold is the search result.

[0151] In an embodiment of the present application, the file to be retrieved is vectorized using a pre-trained model to obtain the target vector.

[0152] Optionally, the second similarity is cosine similarity.

[0153] The above describes various methods of the embodiments of the present application. The following further provides apparatuses for implementing the above methods.

[0154] like Figure 2 As shown, the embodiment of the present application further provides a semantic search device 200, comprising:

[0155] The first determination module 201 is used to determine the complexity level of the question to be retrieved;

[0156] The second determining module 202 is configured to determine the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; the sentences are divided according to clause punctuation marks;

[0157] A third determining module 203 is configured to determine a target number of preceding and following sentences corresponding to each sentence based on the length of the context window;

[0158] The retrieval module 204 is configured to perform a semantic search on the question to be retrieved based on a target number of the preceding and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

[0159] The semantic search device of the embodiment of the present application, by grading the complexity of the search questions and determining the length of the context window of each sentence in the knowledge document, can determine the target number of previous sentences and following sentences corresponding to each sentence in the knowledge document of the search question to be retrieved during the search process; thereby, a semantic search is performed on the search question to be retrieved based on the target number of the previous sentences and the following sentences. The semantic search method of the embodiment of the present application, by grading the complexity of the search questions to be retrieved, can determine the length of the context window of each sentence in the knowledge document corresponding to different complexity levels, thereby obtaining a search result with higher accuracy, solving the problem of poor accuracy of semantic search results in the prior art.

[0160] like Figure 3 As shown, the embodiment of the present application further provides a semantic search device 300, including: a processor 301;

[0161] The processor 301 is configured to determine the complexity level of the question to be retrieved;

[0162] Determining the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; dividing the sentence according to the clause punctuation marks;

[0163] Determining a target number of preceding and following sentences corresponding to each sentence according to the length of the context window;

[0164] A semantic search is performed on the question to be retrieved according to the target number of preceding sentences and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

[0165] The semantic search device provided in the embodiment of the present application, by grading the complexity of the retrieval questions and determining the length of the context window of each sentence in the knowledge document, can determine the target number of previous sentences and following sentences corresponding to each sentence in the knowledge document of the question to be retrieved during the retrieval process; thereby performing a semantic search on the question to be retrieved based on the target number of the previous sentences and the following sentences. The semantic search method of the embodiment of the present application, by grading the complexity of the question to be retrieved, can determine the length of the context window of each sentence in the knowledge document corresponding to different complexity levels, thereby obtaining a retrieval result with higher accuracy, solving the problem of poor accuracy of semantic search results in the prior art.

[0166] Please refer to Figure 4 , an embodiment of the present application also provides a terminal device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the computer program is executed by the processor 401, the computer program implements the various processes of the above-mentioned semantic search method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0167] The present application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-mentioned semantic search method embodiment and can achieve the same technical effect. To avoid repetition, the details are not described here. The computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0168] An embodiment of the present application also provides a computer program product, including computer instructions. When the computer instructions are executed by a processor, the various processes of the above-mentioned semantic search method embodiment are implemented and can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0169] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0170] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0171] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A semantic search method, characterized in that: include: Determine the complexity level of the query to be retrieved; Determining the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; The sentences are divided according to clause punctuation marks; Determining a target number of preceding and following sentences corresponding to each sentence according to the length of the context window; A semantic search is performed on the question to be retrieved according to the target number of preceding sentences and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

2. The semantic search method according to claim 1, characterized in that: Performing a semantic search on the question to be retrieved based on the target number of preceding and following sentences to obtain a search result corresponding to the question to be retrieved, including: Determining a first similarity between each sentence in the knowledge document and a corresponding preceding sentence and a following sentence; Segmenting the knowledge document according to the first similarity corresponding to each sentence in the knowledge document to obtain a plurality of semantic text blocks; A search result corresponding to the question to be retrieved is determined according to the second similarity between each semantic text block and the question to be retrieved.

3. The semantic search method according to claim 2, characterized in that: Determining a first similarity between each sentence in the knowledge document and a corresponding preceding sentence and a following sentence includes: Input each of the sentences into the model to obtain an embedding feature corresponding to each of the sentences; Determine the cosine similarity between the first embedded feature and the second embedded feature of each of the sentences as the first similarity; The first embedded feature is an embedded feature corresponding to the preceding sentence of the sentence, and the second embedded feature is an embedded feature corresponding to the following sentence of the sentence.

4. The semantic search method according to claim 2, characterized in that: Segment the knowledge document according to the first similarity corresponding to each sentence in the knowledge document to obtain a plurality of semantic text blocks, including: Determine a local minimum value of the first similarity, where the local minimum value is a first similarity that is smaller than a first value and smaller than a second value, the first value is a first similarity to a preceding sentence of the sentence, and the second value is a first similarity to a following sentence of the sentence; Determining a position index according to the local minimum value; The knowledge document is segmented at the position index to obtain a plurality of semantic text blocks.

5. The semantic search method according to claim 4, characterized in that: Determining a position index according to the local minimum value includes: determining a valid threshold value of the local minimum; Determining the local minimum value that is less than the validity threshold value as a valid local minimum value; Determine that the statement corresponding to the effective local minimum value is the position index.

6. The semantic search method according to claim 5, characterized in that: Determining an effective threshold of the local minimum includes: Determining the number of first sentences in the knowledge document, where the first sentences are sentences in the context window corresponding to the current sentence and the first similarity is greater than or equal to the current sentence; The percentage of the number of the first sentences to the total number of all sentences in the knowledge document is determined as the effective threshold.

7. The semantic search method according to claim 4, characterized in that: The knowledge document is segmented at the position index to obtain a plurality of semantic text blocks, including: In the case where the position index is one, all texts before the position index are obtained as a semantic text, and all texts after the position index are obtained as a semantic text; In the case where there are multiple position indexes, the text between every two position indexes is determined to be a semantic text, all text before the first position index is determined to be a semantic text, and all text after the last position index is determined to be a semantic text; the first position index and the last position index are determined according to their position order in the knowledge document.

8. The semantic search method according to claim 1, wherein: Determine the complexity level of the query to be searched, including: Performing text normalization processing on the question to be retrieved; Inputting the processed question to be retrieved into a language model to obtain a complexity level of the question to be retrieved; Wherein, the language model is a model obtained through training of historical retrieval questions; The complexity levels include: simple, medium, and complex.

9. A semantic search device, characterized in that: include: A first determination module is used to determine the complexity level of the question to be retrieved; A second determining module is configured to determine the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; The sentences are divided according to clause punctuation marks; a third determining module, configured to determine a target number of preceding and following sentences corresponding to each sentence according to the length of the context window; The retrieval module is configured to perform a semantic search on the question to be retrieved based on a target number of preceding sentences and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

10. A semantic search device, characterized in that: comprising a processor, wherein The processor is configured to determine the complexity level of the question to be retrieved; Determining the length of the context window of each sentence in the knowledge document according to the complexity level of the question to be retrieved; dividing the sentence according to the clause punctuation marks; Determining a target number of preceding and following sentences corresponding to each sentence according to the length of the context window; A semantic search is performed on the question to be retrieved according to the target number of preceding sentences and following sentences to obtain a retrieval result corresponding to the question to be retrieved.

11. A terminal device, characterized in that: include: A processor, a memory, and a program stored in the memory and executable on the processor, wherein the program implements the steps of the semantic search method according to any one of claims 1 to 8 when executed by the processor.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the semantic search method according to any one of claims 1 to 8.

13. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the semantic search method according to any one of claims 1 to 8.

Images