Long-text-based query retrieval method, device and electronic equipment

Abstract

The application discloses a kind of based on long text query retrieval method, device and electronic equipment, it is proposed with the semantic retrieval of word information and the reordering strategy with robustness, so as to give consideration to retrieval effect and efficiency.It is specifically to long text according to different granularity, and in the segmentation process, the multi-stage retrieval related to word information is carried out by combining query demand and segmented text granularity, then the reordering model modeled by the automatic pseudo-label mechanism in advance is used, the semantic correlation prediction of several candidates finally output by multi-stage retrieval and query demand is carried out, the retrieval result of fine sorting is obtained, so as to obtain the target retrieval text corresponding to query.The application can accurately retrieve the information required by user, and reduce the number of change query condition, and the rearrangement process of retrieval result relies on the pre-built model with robustness, effectively improves the retrieval generalization ability and retrieval effect for long text, so as to be able to greatly improve the use experience for search engine.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a query and retrieval method, apparatus, and electronic device based on long text. Background Technology

[0002] Currently, when processing long texts, especially for retrieval tasks involving chapter-level long texts (such as, but not limited to, news, work reports, academic papers, books, etc.), the core technical approach of search engines is to return target documents that are highly relevant to the query conditions from the long text (document) based on the input query text.

[0003] Currently, most mainstream search engines use the following search strategies:

[0004] (1) The keyword matching-based retrieval strategy obtains retrieval results by calculating the word overlap between the query and the document. However, the keyword-based retrieval method cannot solve the problem of words with consistent semantics but inconsistent expression (i.e., it is difficult to handle retrieval tasks where there is no word overlap).

[0005] (2) Semantic association-based retrieval strategies: In order to overcome the drawbacks of keyword matching and achieve effective retrieval, the industry generally believes that retrieval engines should understand the semantics of query and document. As a result, the following semantic retrieval algorithms have been developed: semantic retrieval based on language model, fast semantic retrieval based on semantic vector, and semantic retrieval based on contrastive learning.

[0006] This invention analyzes the practical application of the latter strategy, namely the semantic association-based retrieval strategy, in long text retrieval at the chapter level, and finds that it has at least the following drawbacks:

[0007] Current semantic retrieval strategies do not specifically model long text retrieval scenarios. On the one hand, although deep semantic retrieval methods can achieve better retrieval results than keyword matching, their retrieval efficiency is far lower than that of keyword matching methods. The main reason is that current semantic retrieval strategies cannot perform inverted indexing of the semantic vector of each document, making it difficult to apply to the retrieval needs of massive amounts of long texts. On the other hand, current semantic retrieval strategies do not consider the data noise problem that occurs in practical applications.

[0008] Specifically, datasets used for training and optimizing semantic retrieval models typically provide a subset of positive search examples from the query text (positive search examples refer to the target text to be retrieved for each input query text). However, datasets usually do not provide negative search examples for each query text (negative search examples refer to non-target text that is semantically unrelated to the input query text). This is because for the input query text, anything semantically unrelated can be considered a negative example, making it difficult to exhaustively list all such examples given the massive amount of long texts. Similarly, not all positive search examples can be provided, as there may be more than one target text that meets the criteria, and the positive examples are constantly being updated and changed. In addition, since datasets are constructed automatically, semi-automatically, or manually, errors are inevitable. This necessitates that semantic retrieval strategies possess generalization and robustness to avoid being influenced by noisy examples (noisy examples refer to text that should be considered positive search examples but is not actually labeled, or text that is labeled as positive search examples but is actually irrelevant) during the training of the semantic retrieval model. Summary of the Invention

[0009] In view of the above, the present invention aims to provide a query and retrieval method, apparatus and electronic device based on long text, so as to solve specific technical problems in querying and retrieving long text.

[0010] The technical solution adopted in this invention is as follows:

[0011] In a first aspect, the present invention provides a query and retrieval method based on long text, including:

[0012] Long texts are segmented at different granularities, and corresponding to the granularity of the segmented text, a phased retrieval is performed in conjunction with the query text; wherein, the phased retrieval adopts a retrieval strategy that integrates word and sequence semantic relevance;

[0013] According to the preset re-ranking model, the final output results of the staged retrieval are combined with the query text to form a semantic interaction, thereby obtaining the re-ranking result; wherein, the re-ranking model is modeled using an automatic pseudo-labeling mechanism.

[0014] The reordering results are used to obtain the target retrieval text corresponding to the query requirements.

[0015] In at least one possible implementation, the retrieval strategy that integrates word and sequence semantic relevance includes:

[0016] Perform word segmentation on both the query text and the segmented text;

[0017] Based on the word segmentation results, obtain the word-level vectors and sequence-level vectors of the query text and the segmented text respectively;

[0018] The first similarity between the query text and the segmented text is obtained based on the word-level vectors, and the initial retrieval score is determined by the first similarity; wherein, the first similarity represents the degree of overlap between words;

[0019] The second similarity between the query text and the segmented text is obtained based on the sequence-level vector, and the second similarity is fused with the initial retrieval score to obtain the target retrieval score.

[0020] In at least one possible implementation, obtaining the first similarity between the query text and the segmented text based on the word-level vector, and determining the initial retrieval score based on the first similarity, includes:

[0021] Calculate the preset weight values ​​for all words in the segmented text;

[0022] Using the preset weight values ​​of each word, the first similarity values ​​of the first similarity vectors of multiple words with the highest first similarity are weighted and summed to obtain the initial retrieval score.

[0023] In at least one possible implementation, the automatic pseudo-labeling mechanism includes:

[0024] First, several binary classification semantic retrieval models with different random seeds are jointly trained using real labels given in advance for each input data.

[0025] After the training process reaches a certain level, pseudo-labels are obtained based on the predicted probability distribution of several binary classification semantic retrieval models for the same input data; wherein, the candidate retrieval content representing the final output result is concatenated with the query text as the input data.

[0026] In at least one possible implementation, obtaining the pseudo-labels for subsequent training includes:

[0027] Obtain several predicted probability distribution vectors of several binary classification semantic retrieval models based on the same input data;

[0028] Calculate the average vector of several predicted probability distribution vectors, and use the average vector as a pseudo-label;

[0029] If the deviation between the pseudo-label and the real label exceeds a predetermined standard, then calculate the first loss value between the prediction result and the real label, and calculate the second loss value between the pseudo-label and a single binary classification semantic retrieval model.

[0030] The subsequent training is completed by combining the first loss value and the second loss value.

[0031] In at least one possible implementation, the phased retrieval includes:

[0032] The long text is divided into multiple paragraphs, and a first search is performed on the paragraphs to obtain several candidate paragraphs;

[0033] Each candidate paragraph is divided into multiple sentences, and a second search is performed on the sentences to obtain several candidate sentences, which are then used as the final output of the phased search.

[0034] In at least one possible implementation, obtaining the target retrieval text corresponding to the query requirement using the reordering result includes: placing the semantically most relevant final output result first, and determining the target retrieval text based on the hierarchical correspondence between the long text and the final output result.

[0035] Secondly, the present invention provides a query and retrieval device based on long text, comprising:

[0036] The phased retrieval module is used to segment long texts at different granularities, and corresponding to the granularity of the segmented text, it performs phased retrieval in conjunction with the query text; wherein, the phased retrieval adopts a retrieval strategy that integrates word and sequence semantic relevance;

[0037] The reordering module is used to combine several final output results of the staged retrieval with the query text to form a semantic interaction based on a preset reordering model, thereby obtaining the reordering result; wherein, the reordering model is modeled using an automatic pseudo-labeling mechanism.

[0038] The search result output module is used to obtain the target search text corresponding to the query requirements using the reordering results.

[0039] In at least one possible implementation, the staged retrieval module includes:

[0040] The word segmentation unit is used to segment the query text and the segmented text into words respectively;

[0041] Semantic representation unit, used to obtain word-level vectors and sequence-level vectors of the query text and the segmented text respectively based on the word segmentation results;

[0042] The retrieval score acquisition unit is used to obtain the first similarity between the query text and the segmented text based on the word-level vector, and to determine the initial retrieval score based on the first similarity; wherein, the first similarity characterizes the degree of overlap between words;

[0043] The similarity fusion unit is used to obtain the second similarity between the query text and the segmented text based on the sequence-level vector, and to fuse the second similarity with the initial retrieval score to obtain the target retrieval score.

[0044] In at least one possible implementation, the retrieval score acquisition unit includes:

[0045] The word weighting component is used to calculate the preset weight values ​​of all words in the segmented text.

[0046] The initial retrieval score calculation component is used to use the preset weight values ​​of each word to perform a weighted sum of the first similarity values ​​of multiple word-level vectors with the highest first similarity to obtain the initial retrieval score.

[0047] In at least one possible implementation, the reordering module includes:

[0048] The first training unit is used to jointly train several binary semantic retrieval models with different random seeds by using real labels given in advance for each input data.

[0049] The second training unit is used to obtain pseudo-labels for subsequent training based on the predicted probability distribution of several binary classification semantic retrieval models for the same input data after the training process has reached a certain level; wherein, the candidate retrieval content representing the final output result is concatenated with the query text as the input data.

[0050] In at least one possible implementation, the second training unit includes:

[0051] The component for obtaining the predicted probability distribution vector is used to obtain several predicted probability distribution vectors of several binary classification semantic retrieval models based on the same input data.

[0052] A pseudo-label calculation component is used to calculate the average vector of several predicted probability distribution vectors and use the average vector as a pseudo-label.

[0053] A dual loss calculation component is used to calculate a first loss value between the prediction result and the real label, and a second loss value between the pseudo label and a single binary classification semantic retrieval model, if the deviation between the pseudo label and the real label exceeds a predetermined standard.

[0054] The post-training component is used to combine the first loss value and the second loss value to complete subsequent training.

[0055] In at least one possible implementation, the staged retrieval module specifically includes:

[0056] The segmented retrieval submodule is used to divide a long text into multiple segments and perform a first retrieval on the segments to obtain several candidate segments;

[0057] The sentence segmentation retrieval submodule is used to segment each candidate paragraph into multiple sentences, perform a second retrieval on the sentences to obtain several candidate sentences, and use the candidate sentences as the final output result of the phased retrieval.

[0058] In at least one of the possible implementations, the retrieval result output module is specifically used to: prioritize the semantically most relevant final output result and determine the target retrieval text based on the hierarchical correspondence between the long text and the target final output result.

[0059] Thirdly, the present invention provides an electronic device, comprising:

[0060] One or more processors, a memory, and one or more computer programs, the memory being a non-volatile storage medium, wherein the one or more computer programs are stored in the memory, the one or more computer programs including instructions that, when executed by the device, cause the device to perform the method as described in the first aspect or any possible implementation thereof.

[0061] The main concept of this invention lies in proposing a semantic retrieval strategy related to word information and a robust re-ranking strategy to achieve a long-text semantic retrieval scheme that balances retrieval effectiveness and efficiency. Specifically, the long text is segmented according to different granularities from large to small. During the segmentation process, multi-stage retrieval related to word information is performed based on the query requirements and the different granularities of the segmented text. Then, a re-ranking model pre-modeled using an automatic pseudo-labeling mechanism is used to predict the semantic relevance between the candidate results from the multi-stage retrieval and the query requirements, resulting in a finely ranked retrieval result, thereby obtaining the target retrieval text for the corresponding query. This invention can accurately retrieve the information needed by the user and reduce the number of times the user changes their query conditions. The re-ranking process of the retrieval results relies on a robust pre-built model, effectively improving the generalization ability and retrieval effect for long texts, thus significantly improving the user experience of the search engine.

[0062] Furthermore, in order to avoid the influence of multiple highly overlapping words on semantic association during the retrieval process, it is proposed to use the preset weights of each word to weight multiple overlapping results, thereby reducing the role of irrelevant words in the semantic retrieval stage and highlighting the role of keywords with strong semantic relevance.

[0063] Furthermore, by initializing multiple pre-trained semantic retrieval models and calculating pseudo-labels for each training data in the dataset at the beginning of training, and then using these pseudo-labels to replace the given real labels for subsequent training, the negative impact of the original incorrect labels in the dataset on the training of the reordered models can be mitigated. Attached Figure Description

[0064] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described below with reference to the accompanying drawings, wherein:

[0065] Figure 1 A flowchart illustrating an embodiment of the long-text-based query and retrieval method provided by the present invention;

[0066] Figure 2 This is a schematic diagram of the data flow architecture of a preferred embodiment of the long text-based query and retrieval method provided by the present invention;

[0067] Figure 3 A schematic diagram of the modeling process for a preferred embodiment of the reordering model provided by the present invention;

[0068] Figure 4 A schematic diagram illustrating an embodiment of the long-text-based query and retrieval device provided by the present invention;

[0069] Figure 5 A schematic diagram of an embodiment of the electronic device provided by the present invention. Detailed Implementation

[0070] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0071] To address the shortcomings of current methods for querying and retrieving long texts at the chapter level, this invention proposes at least one embodiment of a query and retrieval method based on long texts, such as... Figure 1 As shown, it can specifically include:

[0072] Step S1: Segment the long text according to different granularities, and perform phased retrieval in conjunction with the query text according to the granularity of the segmented text; wherein, the phased retrieval adopts a retrieval strategy that integrates word and sequence semantic relevance;

[0073] Step S2: Based on the preset re-ranking model, combine the several final output results of the phased retrieval with the query text to form a semantic interaction and obtain the re-ranking result; wherein, the re-ranking model is modeled using an automatic pseudo-labeling mechanism.

[0074] Step S3: Use the reordering results to obtain the target retrieval text corresponding to the query requirements.

[0075] Specifically, in the aforementioned embodiments, the main characteristics of long texts at the chapter level are: content semantically related to the query is likely to appear in the middle or even later part of the long text. However, since current semantic retrieval algorithms cannot directly encode long texts at the chapter level to obtain semantic representations, and if all long texts are directly split into sentences, the size of the text database to be retrieved will increase exponentially. In particular, most of the sentences will not have a semantic connection with the query, which is obviously not convenient for subsequent semantic retrieval.

[0076] Based on this, the present invention proposes a phased retrieval mode, and the illustrative idea can be combined with... Figure 2 The process involves first dividing the long text into several relatively short paragraphs, and then searching these paragraphs in the first retrieval stage. Next, the candidate paragraphs obtained in the first retrieval stage are further divided into sentences, and the second retrieval stage searches these sentences. Finally, the target paragraph containing the target sentence and the target long text are returned.

[0077] This method can efficiently locate which specific sentence in a long text is semantically related to the input query. In practical applications, the target sentence can be highlighted directly, making it convenient for users to obtain the information they need.

[0078] Regarding the aforementioned granularity of segmentation, it can refer to segmentation into paragraphs or sentences. The processing flow for segmentation and sentence segmentation can be referenced as follows:

[0079] 1) Set the length threshold of window w to β, and divide the long text d into n sentences. Add the sentences to window w in sequence until the sum of the text lengths in window w is exactly less than β (adding another sentence will make the sum of the text lengths in the window greater than β). Let all the text in window w at this time be paragraph p1;

[0080] 2) Continue adding the remaining sentences to a new window w. Once the length threshold β is reached, record it as a new paragraph p. j The window is continuously filled until the sentence list is processed, and the returned paragraph list {p1, ..., p} is filled. j , ...} is the paragraph division of a long text;

[0081] 3) Based on the above paragraph division, for each paragraph p j Sentence segmentation is performed to obtain the segmentation results in each window.

[0082] Regarding the aforementioned multi-stage retrieval of text segmented at different granularities, the following illustrative implementation method can be referenced:

[0083] based on Figure 2When performing semantically relevant paragraph retrieval, the user inputs a query, and the paragraph-level semantic retrieval model retrieves n semantically relevant candidate paragraphs from the long text database. Then, these candidate paragraphs are segmented into sentences, and the sentence-level semantic retrieval model retrieves k semantically most relevant candidate sentences from these n candidate paragraphs. These k candidate sentences are the final output of the aforementioned staged retrieval (while the n candidate paragraphs can be regarded as the intermediate output of the staged retrieval).

[0084] Furthermore, regarding the application of the aforementioned paragraph-level or sentence-level semantic retrieval models in the training process, please refer to the following:

[0085] After splitting massive amounts of long text into multiple paragraphs, semantic similarity matching can be used to compare the query with each paragraph one by one to find the key paragraph most relevant to the query. This key paragraph is then used as a paragraph-level positive sample dataset (other paragraphs can be paragraph-level negative sample datasets) to construct a smaller retrieval sample library for training and optimizing the paragraph-level semantic retrieval model. Next, the top-ranking sample data of the paragraph retrieval results can be used to further utilize the sentence segmentation function in NLTK to split the positive and negative paragraph sample data into several single sentences, constructing an even smaller retrieval sample library, obtaining sentence-level positive and negative samples for training and optimizing the sentence-level semantic retrieval model.

[0086] The aforementioned paragraph-level and sentence-level semantic retrieval models are merely illustrative. Different semantic retrieval models can correspond to different modeling granularities (i.e., segmentation granularities). However, the structure of semantic retrieval models can be considered consistent, and therefore will be introduced uniformly below.

[0087] To combine the performance advantages of semantic retrieval with the efficiency advantages of keyword matching, the following example of a retrieval strategy that integrates word and sequence semantic relevance is provided:

[0088] Step S11: Perform word segmentation on the query text and the segmented text respectively (the segmented text mentioned here can refer to paragraphs segmented from the original long text and sentences segmented from the paragraphs at different stages of the aforementioned text segmentation).

[0089] Step S12: Based on the word segmentation results, obtain the word-level vector and sequence-level vector of the query text and the segmented text respectively (for the segmented text, the sequence-level vector mentioned here can correspond to the long sentence-level vector representation in paragraphs and the short sentence-level vector representation in sentences at different stages of the aforementioned text segmentation).

[0090] Step S13: Obtain the first similarity between the query text and the segmented text based on the word-level vector, and determine the initial retrieval score based on the first similarity; the first similarity represents the degree of overlap between words;

[0091] Step S14: Obtain the second similarity between the query text and the segmented text based on the word-level vector, and fuse the second similarity with the initial retrieval score to obtain the target retrieval score.

[0092] In practice, you can first use a word-granular BERT model, such as WoBERT or wordBERT, to segment the text in the entire long text library. Then, refer to the idea of ​​keyword matching similar to BM25 (but not the keyword matching method), and based on the segmentation results, select a subset of texts that overlap with the query from the massive text library.

[0093] Then, the segmented query text and the segmented text are independently input into a word-level pre-trained BERT model to obtain their respective sequence-level vector representations (CLS vectors) and word-level vector representations (token vectors). The formula for calculating the word-level vector is as follows:

[0094]

[0095]

[0096] Here, LM represents a word-level pre-trained BERT model, and W... token It is a trainable weight matrix that maps and transforms the output of the language model; The vector representation of the i-th word in the query. This represents the vector representation of the j-th word in the document.

[0097] Using the aforementioned word-level vectors, the overlapping words q in the query and segmented text are analyzed within the text subset. i ∈q∩d, calculate the retrieval score (initial retrieval score) at the level of overlapping words. To elaborate further, we can first analyze each word q in the current query... i Match all identical words d in the subset j =q i Then, using the word-level vectors provided by the pre-trained BERT model at the word granularity, d is calculated respectively. j and q i The dot product similarity (first similarity) is calculated, and all d values ​​are extracted. j The similarity score of the token-level word vector with the highest similarity is used as the aforementioned retrieval score.

[0098] It is important to emphasize that in real-world scenarios, the query and segmented text may contain multiple overlapping words. However, the importance of these overlapping words in relation to the current query requirements may not be consistent. Therefore, in some preferred embodiments of this invention, the step of obtaining the first similarity between the query text and the segmented text based on the word-level vector, and determining the initial retrieval score based on the first similarity, includes:

[0099] Step S131: Calculate the preset weight values ​​of all words in the segmented text;

[0100] Step S132: Using the preset weight values ​​of each word, the first similarity values ​​of the multiple word-level vectors with the highest first similarity are weighted and summed to obtain the initial retrieval score.

[0101] In practice, after initial word segmentation, the TF-IDF weight value of each segmented word is calculated based on the original text (query text, segmented text). Then, the semantic relevance of multiple overlapping words (those with the highest first similarity) is weighted and summed by combining the TF-IDF weight values ​​of each word. The specific calculation formula is as follows:

[0102]

[0103] Continuing from the previous point, it should be noted that the initial retrieval score does not take into account the semantic relationships between non-overlapping words. Therefore, it may also face word mismatch problems similar to keyword matching. To address this, this invention utilizes [CLS] to integrate sentence-level representations (sequence-level vectors):

[0104]

[0105]

[0106] Through sequence-level vectors and The second similarity between words (which can be calculated as dot product similarity, etc.) provides higher-level semantic matching information, thereby alleviating the problem of mismatch between non-overlapping words. Finally, the target retrieval score of the semantic retrieval strategy that integrates word and sequence semantic similarity is obtained. The following is an illustrative comprehensive scoring function:

[0107]

[0108] In other words, the aforementioned preliminary search score reflects the semantic similarity between the same words (literal overlap with the query requirements and high relevance), while the target search score represents the semantic similarity supplement to the preliminary search score. Specifically, it is the multidimensional semantic relevance reflected by the overall semantic similarity of the entire sequence of the two input texts (query text and segmented text) at the semantic level, based on the preliminary search score.

[0109] After obtaining the semantic retrieval results of the query through the aforementioned steps, since the query does not semantically interact with the final output of the staged retrieval (as in some embodiments, the sentence-level retrieval results), this invention proposes to semantically rearrange the final output of the staged retrieval. Specifically, this involves concatenating the query text with the final output to achieve semantic interaction.

[0110] As mentioned earlier, noisy samples can significantly impact the performance of semantic retrieval processing. In particular, models with stronger learning capabilities are more likely to fit noisy data and are thus more susceptible to the impact. The reordering mechanism described in this invention concatenates the query (which can consist of n words) with several retrieval results (the retrieved results are taken at the sentence level, with each sentence consisting of m words) to obtain semantic interaction, thereby learning the relevance between the query and the retrieval results. In practice, this can be achieved by concatenating the query text represented by the [CLS] vector with the retrieval statement represented by the [SEP] vector. This modeling method has extremely strong learning capabilities, therefore requiring specific design for the training of the aforementioned reordering strategy to protect the model from the influence of noisy data, thereby improving the robustness of the reordering strategy. Accordingly, this invention proposes using an automatic pseudo-label mechanism to improve the reordering model's ability to model noisy data and enhance its robustness. The core idea is to first use pre-given real labels for each input data to jointly train several binary classification semantic retrieval models (model 1...model M) with different random seeds; after the training process reaches a certain level, pseudo-labels for subsequent training are obtained based on the prediction logit of several binary classification semantic retrieval models for the same input data; wherein, the candidate retrieval content representing the final output result is concatenated with the query text as the input data.

[0111] Specific combination Figure 3As shown, this invention preferably uses a BERT-based binary classification pre-trained language model to reorder the final output results of the staged retrieval (the following uses several sentences output by a sentence-level semantic retrieval model as examples), that is, to let the binary classification model learn whether the retrieved sentences are truly semantically relevant sentences. Then, using different random seeds, M binary classification models with the same structure but different initial parameters are initialized. In the first 10% of the training phase (which can be set as needed), the loss is calculated using the given true labels in the dataset, and gradient backpropagation is then performed to optimize the model parameters of the M binary classification models. When the number of training steps exceeds 10%, when training for each input data, the predicted logit vectors of the M binary classification models on each input data are first obtained, and the average vector of the predicted logit vectors is taken as the pseudo-label. If the pseudo-label differs from the given true label in the dataset by more than a predetermined standard, then when calculating the loss, in addition to calculating the loss between the predicted result and the given true label in the dataset, the loss is also calculated. true It is also necessary to calculate the loss of the pseudo-labels and a single binary classification model. add Value, take loss true With loss add The gradient of each model parameter is calculated by weighted average of the values, and then gradient backpropagation is performed to optimize the model parameters.

[0112] For example, if "Cicadas generally eat the sap of crops and plants" is mistakenly labeled as a negative example of the query "What do cicadas usually eat?", that is, the dataset is given with the true label [1,0], where 1 represents 100% semantic relevance and 0 represents no semantic relevance. During the training of the re-ranking model proposed in this invention, it can be assumed that three binary classification models are randomly initialized. The probability distributions (predicted logit) of the prediction results of these three binary classification models for the input data "[CLS] What do cicadas usually eat? [SEP] Cicadas generally eat the sap of crops and plants" are [0.2, 0.8], [0.11, 0.89], and [0.35, 0.65], respectively. Then the pseudo label is [0.22, 0.78], which is significantly different from the true label given in the dataset. This also indicates that there is a problem with the given label of this data. Therefore, for the subsequent 90% of the training, the pseudo label is used for parameter optimization (that is, no longer relying on the given label in the dataset, but correcting and replacing the given label in the dataset with the new label determined at the beginning of training), thereby establishing a robust re-ranking model, which can alleviate the impact of noisy data on the retrieval results.

[0113] Finally, it can be added that, for the aforementioned step S3, obtaining the target retrieval text corresponding to the query requirements using the reordering results, specifically, it can mean placing the semantically most relevant target final output result before (other final output results), and determining the target retrieval text based on the hierarchical correspondence between the long text and the target final output result. On this basis, the target final output results related to the query can also be highlighted.

[0114] In summary, the main concept of this invention lies in proposing a semantic retrieval strategy related to word information and a robust re-ranking strategy to achieve a long-text semantic retrieval scheme that balances retrieval effectiveness and efficiency. Specifically, the long text is segmented according to different granularities from large to small. During the segmentation process, multi-stage retrieval related to word information is performed based on the query requirements and the different granularities of the segmented text. Then, a re-ranking model pre-modeled by an automatic pseudo-labeling mechanism is used to predict the semantic relevance between the candidate results from the multi-stage retrieval and the query requirements, resulting in a finely ranked retrieval result, thereby obtaining the target retrieval text for the corresponding query. This invention can accurately retrieve the information needed by the user and reduce the number of times the user changes their query conditions. The re-ranking process of the retrieval results relies on a robust pre-built model, effectively improving the generalization ability and retrieval effect for long texts, thus significantly improving the user experience of the search engine.

[0115] Corresponding to the above embodiments and preferred solutions, the present invention also provides an embodiment of a query and retrieval device based on long text, such as... Figure 4 As shown, it may specifically include the following components:

[0116] The phased retrieval module 1 is used to segment long texts at different granularities, and to perform phased retrieval in conjunction with the query text according to the granularity of the segmented text; wherein, the phased retrieval adopts a retrieval strategy that integrates word and sequence semantic relevance;

[0117] The reordering module 2 is used to combine several final output results of the staged retrieval with the query text to form a semantic interaction according to a preset reordering model, so as to obtain the reordering result; wherein, the reordering model adopts an automatic pseudo-labeling mechanism for modeling.

[0118] The retrieval result output module 3 is used to obtain the target retrieval text corresponding to the query requirements using the reordering results.

[0119] In at least one possible implementation, the staged retrieval module includes:

[0120] The word segmentation unit is used to segment the query text and the segmented text into words respectively;

[0121] Semantic representation unit, used to obtain word-level vectors and sequence-level vectors of the query text and the segmented text respectively based on the word segmentation results;

[0122] The retrieval score acquisition unit is used to obtain the first similarity between the query text and the segmented text based on the word-level vector, and to determine the initial retrieval score based on the first similarity; wherein, the first similarity characterizes the degree of overlap between words;

[0123] The similarity fusion unit is used to obtain the second similarity between the query text and the segmented text based on the sequence-level vector, and to fuse the second similarity with the initial retrieval score to obtain the target retrieval score.

[0124] In at least one possible implementation, the retrieval score acquisition unit includes:

[0125] The word weighting component is used to calculate the preset weight values ​​of all words in the segmented text.

[0126] The initial retrieval score calculation component is used to use the preset weight values ​​of each word to perform a weighted sum of the first similarity values ​​of multiple word-level vectors with the highest first similarity to obtain the initial retrieval score.

[0127] In at least one possible implementation, the reordering module includes:

[0128] The first training unit is used to jointly train several binary semantic retrieval models with different random seeds by using real labels given in advance for each input data.

[0129] The second training unit is used to obtain pseudo-labels for subsequent training based on the predicted probability distribution of several binary classification semantic retrieval models for the same input data after the training process has reached a certain level; wherein, the candidate retrieval content representing the final output result is concatenated with the query text as the input data.

[0130] In at least one possible implementation, the second training unit includes:

[0131] The component for obtaining the predicted probability distribution vector is used to obtain several predicted probability distribution vectors of several binary classification semantic retrieval models based on the same input data.

[0132] A pseudo-label calculation component is used to calculate the average vector of several predicted probability distribution vectors and use the average vector as a pseudo-label.

[0133] A dual loss calculation component is used to calculate a first loss value between the prediction result and the real label, and a second loss value between the pseudo label and a single binary classification semantic retrieval model, if the deviation between the pseudo label and the real label exceeds a predetermined standard.

[0134] The post-training component is used to combine the first loss value and the second loss value to complete subsequent training.

[0135] In at least one possible implementation, the staged retrieval module specifically includes:

[0136] The segmented retrieval submodule is used to divide a long text into multiple segments and perform a first retrieval on the segments to obtain several candidate segments;

[0137] The sentence segmentation retrieval submodule is used to segment each candidate paragraph into multiple sentences, perform a second retrieval on the sentences to obtain several candidate sentences, and use the candidate sentences as the final output result of the phased retrieval.

[0138] In at least one of the possible implementations, the retrieval result output module is specifically used to: prioritize the semantically most relevant final output result and determine the target retrieval text based on the hierarchical correspondence between the long text and the target final output result.

[0139] The above should be understood Figure 4 The division of components in the long-text-based query and retrieval device shown is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or physically separated. These components can be implemented entirely in software via processing element calls; entirely in hardware; or partially in software via processing element calls and partially in hardware. For example, a particular module can be a separate processing element or integrated into a chip within an electronic device. The implementation of other components is similar. Furthermore, these components can be fully or partially integrated together or implemented independently. During implementation, each step of the above method or each of the above components can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0140] For example, these components can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more Digital Signal Processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, these components can be integrated together to form a System-On-a-Chip (SOC).

[0141] Based on the above embodiments and preferred solutions, those skilled in the art will understand that, in actual operation, the technical concept involved in this invention can be applied to various implementation methods. The following embodiments are used as illustrative examples:

[0142] (1) An electronic device. The device may specifically include: one or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the steps / functions of the foregoing embodiments or equivalent embodiments.

[0143] Specifically, the electronic device can be a computer-related electronic device, such as, but not limited to, various interactive terminals and electronic products, mobile terminals, etc.

[0144] Figure 5 The schematic diagram illustrates an embodiment of the electronic device provided by the present invention. Specifically, the electronic device 900 includes a processor 910 and a memory 930. The processor 910 and the memory 930 can communicate with each other via an internal connection to transmit control and / or data signals. The memory 930 stores computer programs, and the processor 910 retrieves and runs the computer programs from the memory 930. The processor 910 and the memory 930 can be combined into a single processing device, but more commonly they are independent components. The processor 910 executes the program code stored in the memory 930 to achieve the aforementioned functions. In specific implementations, the memory 930 can be integrated into the processor 910, or it can be independent of the processor 910.

[0145] In addition, to further enhance the functionality of the electronic device 900, the device 900 may also include one or more of the following: an input unit 960, a display unit 970, an audio circuit 980, a camera 990, and a sensor 901. The audio circuit may also include a speaker 982, a microphone 984, etc. The display unit 970 may include a display screen.

[0146] Furthermore, the aforementioned device 900 may also include a power supply 950 for providing electrical energy to various devices or circuits in the device 900.

[0147] It should be understood that the operation and / or function of the various components in the device 900 can be specifically referred to in the foregoing descriptions of the methods, systems, and other embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

[0148] It should be understood that Figure 5 The processor 910 in the illustrated electronic device 900 can be a system-on-a-chip (SoC). The processor 910 may include a central processing unit (CPU) and may further include other types of processors, such as a graphics processing unit (GPU), which will be described in detail below.

[0149] In summary, the various processors or processing units inside the processor 910 can work together to implement the previous method flow, and the corresponding software programs of each processor or processing unit can be stored in the memory 930.

[0150] (2) A computer data storage medium storing a computer program or the aforementioned device, which, when executed, causes a computer to perform the steps / functions of the foregoing embodiments or equivalent embodiments.

[0151] In several embodiments provided by this invention, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer data storage medium. Based on this understanding, certain technical solutions of this invention, or the parts that contribute to the prior art, or parts of such technical solutions, can be embodied in the form of software products as described below.

[0152] It should be noted in particular that the storage medium may refer to a server or a similar computer device, specifically, that is, the aforementioned computer program or the aforementioned device is stored in the storage device of the server or similar computer device.

[0153] (3) A computer program product (which may include the above-mentioned device), which, when run on a terminal device, causes the terminal device to execute the long text-based query and retrieval method of the foregoing embodiments or equivalent implementations.

[0154] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the above implementation methods can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the above-mentioned computer program products may include, but are not limited to, APPs.

[0155] Continuing from the previous text, the aforementioned device / terminal can be a computer device, and the hardware structure of this computer device can specifically include: at least one processor, at least one communication interface, at least one memory, and at least one communication bus; the processor, communication interface, and memory can all communicate with each other through the communication bus. The processor may be a central processing unit (CPU), DSP, microcontroller, or digital signal processor, and may also include a GPU, an embedded neural network processing unit (NPU), and an image signal processor (ISP). The processor may also include a specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention. Furthermore, the processor may have the function of operating one or more software programs, which can be stored in a storage medium such as memory. The aforementioned memory / storage medium may include: non-volatile memory, such as a non-removable disk, USB flash drive, portable hard drive, optical disc, etc., as well as read-only memory (ROM), random access memory (RAM), etc.

[0156] In this embodiment of the invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0157] Those skilled in the art will recognize that the modules, units, and method steps described in the embodiments disclosed in this specification can be implemented using electronic hardware, computer software, and a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention.

[0158] Furthermore, the modules and units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed in multiple places, such as nodes in a system network. Specifically, some or all of the modules and units can be selected to achieve the purpose of the above-described embodiments, depending on actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0159] The above description of the structure, features, and effects of the present invention is based on the embodiments shown in the figures. However, the above are only preferred embodiments of the present invention. It should be noted that the technical features involved in the above embodiments and their preferred methods can be reasonably combined and matched by those skilled in the art to form a variety of equivalent solutions without departing from or changing the design concept and technical effects of the present invention. Therefore, the present invention is not limited to the scope of implementation shown in the figures. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments, that do not exceed the spirit covered by the specification and figures, should be within the protection scope of the present invention.

Claims

1. A query and retrieval method based on long text, characterized in that, include: The long text is segmented at different granularities, and corresponding to the granularity of the segmented text, a phased retrieval is performed in conjunction with the query text. This includes: segmenting the long text into multiple paragraphs and performing a first retrieval on each paragraph to obtain several candidate paragraphs; segmenting each candidate paragraph into multiple sentences and performing a second retrieval on each sentence to obtain several candidate sentences, with the candidate sentences serving as the final output of the phased retrieval; wherein the phased retrieval employs a retrieval strategy that integrates word and sequence semantic relevance, including: segmenting the query text and the segmented text into words respectively; based on the word segmentation results, obtaining word-level vectors and sequence-level vectors for both the query text and the segmented text respectively; obtaining a first similarity between the query text and the segmented text based on the word-level vectors, and determining an initial retrieval score based on the first similarity representing the degree of overlap between words; obtaining a second similarity between the query text and the segmented text based on the sequence-level vectors, and fusing the second similarity with the initial retrieval score to obtain a target retrieval score; According to the preset re-ranking model, the final output results of the staged retrieval are combined with the query text to form a semantic interaction and obtain the re-ranking result; wherein, the re-ranking model adopts an automatic pseudo-label mechanism for modeling, including: initializing multiple pre-trained semantic retrieval models and calculating pseudo-labels for each training data in the dataset at the beginning of training, and replacing the given real labels with pseudo-labels for subsequent training. The reordering results are used to obtain the target retrieval text corresponding to the query requirements.

2. The query and retrieval method based on long text according to claim 1, characterized in that, The step of obtaining the first similarity between the query text and the segmented text based on the word-level vectors, and determining the initial retrieval score based on the first similarity, includes: Calculate the preset weight values ​​for all words in the segmented text; Using the preset weight values ​​of each word, the first similarity values ​​of the first similarity vectors of multiple words with the highest first similarity are weighted and summed to obtain the initial retrieval score.

3. The query and retrieval method based on long text according to claim 1, characterized in that, The automatic pseudo-labeling mechanism includes: First, several binary classification semantic retrieval models with different random seeds are jointly trained using real labels given in advance for each input data. After the training process reaches a certain level, pseudo-labels are obtained based on the predicted probability distribution of several binary classification semantic retrieval models for the same input data; wherein, the candidate retrieval content representing the final output result is concatenated with the query text as the input data.

4. The query and retrieval method based on long text according to claim 3, characterized in that, The methods for obtaining the pseudo-labels for subsequent training include: Obtain several predicted probability distribution vectors of several binary classification semantic retrieval models based on the same input data; Calculate the average vector of several predicted probability distribution vectors, and use the average vector as a pseudo-label; If the deviation between the pseudo-label and the real label exceeds a predetermined standard, then calculate the first loss value between the prediction result and the real label, and calculate the second loss value between the pseudo-label and a single binary classification semantic retrieval model. The subsequent training is completed by combining the first loss value and the second loss value.

5. The query and retrieval method based on long text according to any one of claims 1 to 4, characterized in that, The step of obtaining the target retrieval text corresponding to the query requirements using the reordering results includes: placing the semantically most relevant final output result at the top, and determining the target retrieval text based on the hierarchical correspondence between the long text and the final output result.

6. A query and retrieval device based on long text, characterized in that, include: The phased retrieval module is used to segment long texts at different granularities. Corresponding to the granularity of the segmented text, it performs phased retrieval in conjunction with the query text. This includes: segmenting the long text into multiple paragraphs and performing a first retrieval on each paragraph to obtain several candidate paragraphs; segmenting each candidate paragraph into multiple sentences and performing a second retrieval on each sentence to obtain several candidate sentences, with the candidate sentences serving as the final output of the phased retrieval. The phased retrieval employs a retrieval strategy that integrates word and sequence semantic relevance, including: segmenting the query text and the segmented text into words respectively; based on the word segmentation results, obtaining word-level vectors and sequence-level vectors for both the query text and the segmented text respectively; obtaining a first similarity between the query text and the segmented text based on the word-level vectors, and determining an initial retrieval score based on the first similarity representing the degree of overlap between words; obtaining a second similarity between the query text and the segmented text based on the sequence-level vectors, and fusing the second similarity with the initial retrieval score to obtain a target retrieval score. The re-ranking module is used to combine several final output results of the staged retrieval with the query text to form a semantic interaction according to a preset re-ranking model, and obtain the re-ranking result; wherein, the re-ranking model adopts an automatic pseudo-label mechanism for modeling, including: initializing multiple pre-trained semantic retrieval models and calculating pseudo-labels for each training data in the dataset at the beginning of training, and replacing the given real labels with pseudo-labels for subsequent training. The search result output module is used to obtain the target search text corresponding to the query requirements using the reordering results.

7. An electronic device, characterized in that, include: One or more processors, a memory, and one or more computer programs, wherein the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the electronic device, cause the electronic device to perform the long text-based query retrieval method according to any one of claims 1 to 5.

8. A computer data storage medium, characterized in that, The computer data storage medium stores a computer program, which, when run on the computer, causes the computer to execute the long text-based query and retrieval method according to any one of claims 1 to 5.

Images