Model explainability text processing method based on vector retrieval and storage medium

Abstract

The present application relates to the technical field of natural language processing, and particularly relates to a text processing method based on vector retrieval and a storage medium, the text processing method based on vector retrieval provided by the present application comprises the following steps: obtaining a to-be-retrieved text and a vector retrieval model, obtaining an input word vector and a text vector corresponding to the to-be-retrieved text through the vector retrieval model; determining a target similarity of the input word vector and the text vector; performing normalization processing on the target similarity to obtain a normalized similarity; determining a target inverse document frequency of all words in the to-be-retrieved text from an inverse document frequency library constructed, and determining rare words in the to-be-retrieved text according to the target inverse document frequency of all words in the to-be-retrieved text; obtaining a word weight of the rare words, and comparing the word weight of the rare words with the normalized similarity, so as to accurately judge the information retention and loss degree of important words in the to-be-retrieved text in the corresponding vector according to the comparison result, so as to have better interpretability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a model explainability text processing method based on vector retrieval and a storage medium. BACKGROUND

[0002] The vector retrieval system is a core component of a large language model (LLM) for retrieval augmented generation (RAG). Existing text vector retrieval models often use vector dot product as text similarity to retrieve documents. The vector calculation process is to input text, tokenize, and input the retrieval model to calculate a vector to represent the entire text. Existing retrieval models usually use aggregation methods to calculate the vector of the entire text. Common aggregation methods include classification token (CLS) aggregation, average pooling, and max pooling. CLS aggregation adds a CLS token to the front of all input texts when inputting, and uses the output vector of the CLS token as the vector of the text. Average pooling takes the average of the output vectors of all words as the vector of the text. Max pooling takes the maximum value of each dimension of the output vectors of all words as the vector of the text.

[0003] Although the existing vector retrieval model is trained on a large amount of corpus, it lacks explainability, and researchers have difficulty understanding which information of the input text is contained in the output vector and which information of the input text is lost from the output vector. SUMMARY

[0004] In order to solve the problem of insufficient output explainability of the existing vector retrieval model, and the difficulty for researchers to understand which information of the input text is contained in the output vector and which information of the input text is lost from the output vector, the present application provides a model explainability text processing method based on vector retrieval and a storage medium.

[0005] The technical problem solving scheme of the present application provides a model explainable text processing method based on vector retrieval, comprising the following steps: obtaining a to-be-retrieved text and a vector retrieval model, obtaining an input word vector and a text vector corresponding to the to-be-retrieved text through the vector retrieval model; the vector retrieval model is obtained by contrast training of a constructed positive sample pair and a negative sample pair on a bert model; the positive sample pair comprises a preset query request and a document related to the preset query request, and the negative sample pair comprises the preset query request and a document unrelated to the preset query request; determining a target similarity of the input word vector and the text vector; normalizing the target similarity to obtain a normalized similarity; the normalization processing converts the target similarity into a percentage representing importance;

[0006] The normalized similarity is:

[0007]

[0008] Wherein, t is a target word in the to-be-retrieved text; f is the to-be-retrieved text; s is any one word in the to-be-retrieved text; vocab is a smoothed inverse document frequency word library; score(f,t) is the target similarity of the target word vector and the text vector; score(f,s) is the target similarity of any one word in the to-be-retrieved text and the text vector;

[0009] An inverse document frequency word library is constructed, the target inverse document frequency of all words in the to-be-retrieved text is determined from the inverse document frequency word library, and the rare word in the to-be-retrieved text is determined according to the target inverse document frequency of all words in the to-be-retrieved text; the word weight of the rare word is obtained, the word weight of the rare word is compared with the normalized similarity, and a comparison result is obtained; the inverse document frequency in the inverse document frequency word library is determined according to the total number of documents in a preset document library and the number of documents in which each word appears; the inverse document frequency represents the universality of each word in the preset query request;

[0010] The word weight of the rare word is:

[0011] ;

[0012] Wherein, h is a rare word in the to-be-retrieved text f; w f is the word weight of the rare word h; idf_smooth h is the smoothed inverse document frequency of the rare word h in the to-be-retrieved text f; s3 is any one word in the to-be-retrieved text f; idf_smooth s3 is the smoothed inverse document frequency of the word s3 in the query; query is a preset query request.

[0013] Preferably, the positive sample pair and the negative sample pair are obtained by: obtaining a preset query request and a preset document library, the preset document library comprising a plurality of documents; and constructing the positive sample pair and the negative sample pair based on the preset query request and the documents.

[0014] Preferably, before the constructing the positive sample pair and the negative sample pair based on the preset query request and the documents, the method further comprises: obtaining an inverse document frequency of each word in the preset query request and all the documents in the preset document library; and performing smoothing processing on the inverse document frequency of each word to obtain a smoothed inverse document frequency of each word.

[0015] Preferably, the method further comprises: obtaining a BERT model; and performing contrastive training on the BERT model based on the positive sample pair and the negative sample pair to obtain a vector retrieval model, specifically comprising: obtaining the BERT model; inputting the preset query request and the documents in the positive sample pair and the negative sample pair into the BERT model respectively to obtain a vector of each word in the preset query request and the documents in the positive sample pair and the negative sample pair respectively; determining a weight of each word in the positive sample pair and the negative sample pair in the documents and the preset query request respectively according to the inverse document frequency of each word in the preset query request; determining a training text vector of the preset query request and the documents in the positive sample pair and the negative sample pair respectively based on the vector and the weight corresponding to each word; determining a similarity between the preset query request and the documents in the positive sample pair and the negative sample pair respectively according to the training text vector of the preset query request and the documents; and performing contrastive training on the BERT model based on the similarity between the preset query request and the documents in the positive sample pair and the negative sample pair to obtain the vector retrieval model.

[0016] Preferably, after the smoothing processing on the inverse document frequency of each word to obtain the smoothed inverse document frequency of each word, the method further comprises: summarizing the inverse document frequency of all the words corresponding to the preset query request and the inverse document frequency of all the words in all the documents in the preset document library to form an inverse document frequency word library; and summarizing the smoothed inverse document frequency of all the words corresponding to the preset query request and the smoothed inverse document frequency of all the words in all the documents in the preset document library to form a smoothed inverse document frequency word library.

[0017] Preferably, the vector retrieval model is used to obtain an input word vector and a text vector corresponding to the to-be-retrieved text, specifically comprising: using the vector retrieval model to obtain the input word vector of each word in the to-be-retrieved text; obtaining a target smoothed inverse document frequency of each word in the to-be-retrieved text from the smoothed inverse document frequency word library; determining a target weight of each word in the to-be-retrieved text based on the target smoothed inverse document frequency; and determining a text vector of the to-be-retrieved text based on the input word vector and the target weight of each word in the to-be-retrieved text.

[0018] Preferably, the target similarity of the input word vector and the text vector is determined, specifically comprising: determining the target similarity of the input word vector and the text vector of each word in the to-be-retrieved text according to the vector dot product of the input word vector and the text vector.

[0019] The application also provides a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the model explainability text processing method based on vector retrieval according to any one of the preceding claims.

[0020] Compared with the prior art, the model explainability text processing method based on vector retrieval and the storage medium have the following beneficial effects:

[0021] 1. In the model explainability text processing method based on vector retrieval, the input word vector and the text vector of the to-be-retrieved text are obtained through the vector retrieval model; the target similarity of the input word vector and the text vector is determined; then the target similarity is normalized to obtain a normalized similarity; and the rare words are determined according to the inverse document frequency corresponding to all the words in the to-be-retrieved text, so as to compare the word weight of the rare words with the normalized similarity and obtain a comparison result. In the retrieval scene, the more rare the words are, the greater the effect of matching is, and the document with a high matching degree of rare words is generally more relevant than the document with a low matching degree of rare words. Therefore, by determining the similarity of the input word vector and the corresponding text vector of each word in the to-be-retrieved text and comparing the normalized similarity with the weight of the rare words in the to-be-retrieved text, the information retention and loss degree of each word in the to-be-retrieved text in the corresponding vector can be accurately judged according to the comparison result, so as to have better explainability.

[0022] 2. In the vector retrieval model, the preset query request and the preset document library including a plurality of documents are obtained; and the positive sample pairs and the negative sample pairs of the preset query request and the documents are constructed; after the BERT model is obtained, the BERT model is compared and trained based on the positive sample pairs and the negative sample pairs, so as to obtain the vector retrieval model, so as to improve the representation ability of the vector retrieval model through the comparison and training of the positive sample pairs and the negative sample pairs, so that the vector retrieval model can better distinguish the relevance of the retrieval text and the to-be-retrieved content, and thus the accuracy of the retrieval matching when the vector retrieval is performed is enhanced.

[0023] 3. In the embodiment, the inverse document frequency and the smoothed inverse document frequency of each word in the preset query request and the preset document library are determined respectively, so as to measure the universality of each word in the preset query request and the universality of each word in the document through the inverse document frequency; and the discrimination degree when the inverse document frequency is relatively high is reduced through the smoothed inverse document frequency.

[0024] 4、In the embodiment of the present application, the preset query request and the document in the positive sample and the negative sample are respectively input into the bert model to obtain the vector of each word in the preset query request and the document in the positive sample pair and the negative sample pair; then the weight of each word in the positive sample pair and the negative sample pair in the document and the preset query request is respectively determined according to the inverse document frequency of each word in the preset query request; then the training text vector of the preset query request and the document in the positive sample pair and the negative sample pair is respectively determined based on the vector and the weight corresponding to each word; then the similarity of the preset query request and the document in the positive sample pair and the negative sample pair is respectively determined according to the training text vector of the preset query request and the document; finally, the similarity of the preset query request and the document in the positive sample pair and the negative sample pair is used to compare and train the bert model, and the vector retrieval model is obtained. In the aggregation stage, each vector in the preset query request and the document is weighted and aggregated according to the inverse document frequency of the rare word to obtain the weighted rare word in the vector of the preset query request and the document, so that the bert model can retain more information of the rare word in the aggregation stage, can more accurately represent the semantics of the text, and can realize more accurate retrieval.

[0025] 5、In the embodiment of the present application, the inverse document frequency of all words corresponding to the preset query request and the inverse document frequency of all words in all documents in the preset document library are summarized to form an inverse document frequency word library; and the smoothed inverse document frequency of all words corresponding to the preset query request and the smoothed inverse document frequency of all words in all documents in the preset document library are summarized to form a smoothed inverse document frequency word library. In this way, the inverse document frequency and the smoothed inverse document frequency of all words during training are summarized and saved, so that after the model training is completed, the corresponding inverse document frequency and smoothed inverse document frequency can be directly called during user query, the conversion process is simplified, the speed of obtaining the inverse document frequency and the smoothed inverse document frequency is improved, and the user can use more conveniently and quickly.

[0026] 6、In the embodiment of the present application, the target similarity of the vector of each word in the to-be-retrieved text and the text vector is determined according to the vector dot product of the input word vector and the text vector, so that the semantic relationship in natural language can be better understood according to the target similarity.

[0027] 7、The present application also provides a computer readable storage medium, which has the same beneficial effects as the above-mentioned model explainability text processing method based on vector retrieval, and will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0029] Figure 1 is a step flow chart of the model explainability text processing method based on vector retrieval provided by the first embodiment of the present application.

[0030] Figure 2 is a specific step flow chart of step S130 of the model explainability text processing method based on vector retrieval provided by the first embodiment of the present application.

[0031] Figure 3 is a specific step flow chart of steps S140 to S170 of the inappropriate speech prediction method provided by the first embodiment of the present application. DETAILED DESCRIPTION

[0032] In order to make the technical solutions in the embodiments of the present application clearer, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0033] The terms "vertical", "horizontal", "left", "right", "up", "down", "top left", "top right", "bottom left", "bottom right" and similar expressions used herein are only for illustrative purposes.

[0034] Although the existing vector retrieval model has been trained on a large amount of corpus, it lacks explainability, and researchers are difficult to understand from the output vector which information of the input text is contained in the output vector and which information of the input text is lost. A direct idea is that the keyword information in the input text should be preserved as much as possible, and preserving more keyword information during training can improve the retrieval quality, and checking the similarity between the vector and the keyword vector after training can improve the explainability of the model. There are some works in the field of vector retrieval that improve the performance and explainability of the model through keywords, different works are based on different observations, and different keyword extraction methods are proposed.

[0035] The first method is to consider that a set of words with higher language model likelihood of a document is more representative of the keywords of the document. Two sets of words are sampled through the language model of the document, and the model needs to predict the preference relationship between the two sets of words, that is, to determine which set of words is more representative of the document. The representative word prediction task is combined with the Masked Language Model (MLM) objective to jointly train the Transformer model, improving the effect of the retrieval model. Although representative word sets are generated through the document language model, there is no mechanism to explain which words in the word set contribute most to the retrieval results.

[0036] The second method is to consider that keywords will have higher attention weights in the BERT model. The difference between the attention of [CLS] (classification token) and random words is used to sample keywords, and the MLM pre-training is added to improve the effect of the retrieval model. However, the self-attention weight of the BERT model does not directly reflect the representativeness or importance of the words, and cannot provide intuitive explanations about how these words are associated with the semantics of the document.

[0037] The third method is to consider that words highly related to the context are more critical in model training. These words usually have an important influence on the overall semantics and context of the paragraph, so masking these words during the pre-training process of the masked autoencoder can bring greater challenges to the decoder, thereby prompting the encoder to learn more powerful text representation capabilities. However, this method is based on statistical information to calculate keywords, and cannot directly explain why some words are more important for the retrieval task.

[0038] The fourth method is to design a very efficient bag-of-words pre-training method, which significantly enhances the representation ability of dense vectors through text word information compression-prediction restoration, while providing high interpretability. However, the bag-of-words considers that all words have the same weight, so although the trained vector contains all the input text information, it does not highlight the role of the key words. Although a lot of information is provided in terms of interpretability, it still cannot explain which words in the text are more critical.

[0039] Based on the above, some existing methods improve the performance and interpretability of vector retrieval models through keywords. However, there are still deficiencies in their representation ability and interpretability. Therefore, the embodiments of the present application provide a model interpretability text processing method based on vector retrieval, which accurately judges the information retention and loss degree of important words in the corresponding vector in the text to be retrieved by weighting rare words, so as to improve the representation ability of the vector retrieval model and enhance the interpretability.

[0040] Please refer to Figure 1The first embodiment of the present application provides a model explainable text processing method based on vector retrieval, comprising the following steps:

[0041] S10: obtaining a to-be-retrieved text and a vector retrieval model, and obtaining an input word vector and a text vector corresponding to the to-be-retrieved text through the vector retrieval model;

[0042] Specifically, the to-be-retrieved text can be a query request or a query document input by a user in the vector retrieval model. The query request includes but is not limited to a query word, a query sentence or a paragraph of text. The query document can be a document pre-stored in a preset document library. The preset document library can be a general document library or a document library for a specific field, and the present embodiment does not limit this. When a user queries, the document in the preset document library can be directly called as a query document for retrieval. The query document can also be a document directly input by the user for retrieval.

[0043] More specifically, the input word vector can be an embedding vector of each independent word in the to-be-retrieved text. The text vector is a semantic vector corresponding to the to-be-retrieved text. After the to-be-retrieved text is input into the vector retrieval model, the input word vector corresponding to the to-be-retrieved text can be obtained. And the text vector of the to-be-retrieved text can also be obtained through a weighted aggregation manner according to the input word vector corresponding to the to-be-retrieved text.

[0044] S20: determining a target similarity of the input word vector and the text vector;

[0045] Specifically, the similarity of the input word vector and the text vector of each word in the to-be-retrieved text can be determined, so that the correlation degree of the word with the overall semantics of the to-be-retrieved text can be determined through the similarity. And by determining the similarity of all words in the to-be-retrieved text, the core vocabulary with the largest contribution to the text semantics can be identified according to the size of the similarity. The core vocabulary can be a keyword in the present embodiment, and the keyword can include a rare word. The similarity of the input word vector and the text vector corresponding to the core vocabulary can be the target similarity.

[0046] Optionally, the user can also select any one word in the to-be-retrieved text as a target word, and the similarity of the input word vector and the text vector of the target word can also be the target similarity.

[0047] S30: normalizing the target similarity to obtain a normalized similarity;

[0048] Specifically, by normalizing the target similarity, all similarities corresponding to all words can be mapped to a fixed interval, and the contribution degree of the key word is amplified through normalization, so that the difference between the similarities of different words is significantly highlighted, which is more conducive to distinguishing the core vocabulary.

[0049] S40: constructing an inverse document frequency library, determining target inverse document frequencies of all words in the text to be searched from the inverse document frequency library, determining rare words in the text to be searched according to the target inverse document frequencies of all words in the text to be searched; obtaining the word weight of the rare word, comparing the word weight of the rare word with the normalized similarity, and obtaining a comparison result.

[0050] Specifically, in the retrieval scenario, the more rare a word is, the greater the role it plays in matching, and a document with a high degree of matching of rare words is generally more relevant than a document with a low degree of matching of rare words. The higher the inverse document frequency of a word is, the lower the frequency of the word in all documents, and the higher the degree of differentiation. Therefore, the rare words in the text to be searched can be accurately determined by the high and low of the inverse document frequencies of all words in the text to be searched, thereby improving the accuracy of the identification and determination of rare words. For example, in a document library about plants, "de" is not rare, "needle leaf" is relatively rare, and "iron wire fern" is even rarer.

[0051] More specifically, after obtaining the text to be searched and the target inverse document frequencies of all corresponding words, the rare words can be determined according to the ranking of the target inverse document frequencies, that is, selecting words with larger target inverse document frequencies as rare words. After the rare words are determined, the word weight corresponding to the rare word can be determined from the target weight of all words in the text to be searched, so as to compare the word weight of the rare word with the normalized similarity and obtain a comparison result.

[0052] More specifically, the comparison result can be the gap between the word weight of the rare word and the normalized similarity. The smaller the gap between the word weight of the rare word and the normalized similarity, that is, the closer the two are, the more reasonable the information retained by the key word is; the larger the gap between the word weight of the rare word and the normalized similarity, the more information the word loses if the target normalized similarity is low, and the more information the word retains if the target normalized similarity is high, and accordingly, more information of other words is lost.

[0053] It can be understood that by determining the normalized similarity between the input word vector of each word in the text to be searched and the text vector of the text to be searched, and comparing the normalized similarity with the weight of the rare word in the text to be searched, the information retention and loss of each word in the corresponding vector in the text to be searched can be accurately determined according to the comparison result, so as to have better interpretability.

[0054] Optionally, if the target similarity is the target similarity corresponding to the core vocabulary, the final comparison is a comparison of the normalized similarity corresponding to the core vocabulary including the rare word and the word weight of the rare word, which realizes accurate judgment of the information retention and loss degree of the important word in the corresponding vector in the to-be-retrieved text according to the comparison result, so as to have better interpretability. And the comparison of the normalized similarity corresponding to the core vocabulary and the word weight of the rare word avoids the comparison process of general words or irrelevant words, so as to reduce the number of comparison rounds and improve the comparison efficiency.

[0055] Further, the word weight of the rare word is:

[0056] ;

[0057] Wherein, h is a rare word in the to-be-retrieved text f; w f is the word weight of the rare word h; idf_smooth h is the smooth inverse document frequency of the rare word h in the to-be-retrieved text f; s3 is any word in the to-be-retrieved text f; idf_smooth s3 is the smooth inverse document frequency of the word s3 in the query.

[0058] It can be understood that when determining the word weight of the rare word, each word in the to-be-retrieved text needs to be traversed, so as to combine the global information of the to-be-retrieved text, so as to more accurately determine the importance of the rare word.

[0059] Further, the vector retrieval model is obtained by:

[0060] S110: obtaining a preset query request and a preset document library, the preset document library including a plurality of documents;

[0061] Specifically, the preset query request can be provided by the data set itself, or generated, or called from the log by user input during use. In this embodiment, the preset query request can be several, so as to construct several different positive sample pairs and negative sample pairs.

[0062] S120: constructing positive sample pairs and negative sample pairs based on the preset query request and the document;

[0063] Specifically, the positive sample pair includes the preset query request and the document related to the preset query request; the negative sample pair includes the preset query request and the document unrelated to the preset query request. In this embodiment, the preset query request can be several, and the corresponding document related to the preset query request and the document unrelated to the preset query request can be determined according to each preset query request, so as to construct several different positive sample pairs and negative sample pairs.

[0064] S130: Obtain a BERT model, and perform contrastive training on the BERT model based on the positive sample and the negative sample to obtain a vector retrieval model.

[0065] Specifically, the contrastive training is to perform contrastive training on the BERT model by using the positive sample and the negative sample respectively, so as to improve the classification ability of the BERT model in distinguishing the documents that are "similar" and "not similar" to the preset retrieval request. Thus, the vector representation learned by the BERT model can more accurately capture semantic similarity. Queries and documents that are semantically similar have close vectors, while semantically unrelated vectors are far apart.

[0066] It can be understood that, by performing contrastive training on the BERT model by using the positive sample pair and the negative sample pair, the representation ability of the vector retrieval model can be improved, so that the vector retrieval model can better distinguish the relevance of the retrieval text and the content to be retrieved, and thus the accuracy of retrieval matching when performing vector retrieval is enhanced.

[0067] Further, before step S120, the method further comprises:

[0068] S115: Obtain the inverse document frequency of each word in the preset document library and the preset query request;

[0069] Specifically, the distribution of each word in the entire preset document library can be counted, and the total number of documents in the preset document library and the number of documents in which each word appears are determined. The total number of documents is denoted as |D|, and the number of documents in which word j appears is denoted as |Dj|. The inverse document frequency idf of each word is calculated, and the idf of word j is:

[0070]

[0071] S116: Smooth the inverse document frequency of each word to obtain the smoothed inverse document frequency of each word.

[0072] Specifically, the idf value of each word is subjected to log saturation smoothing, and the smoothed inverse document frequency of word j is:

[0073]

[0074] By determining the inverse document frequency and the smoothed inverse document frequency of each word in the document and the preset query request respectively, the universality of each word in the preset query request and the universality of each word in the document are measured by the inverse document frequency; and the discrimination degree when the inverse document frequency is high is reduced by the smoothed inverse document frequency.

[0075] It should be noted that if it is a rare word and a non-rare word, the importance is greatly different. But if they are all rare words, the difference between rare and more rare is not very great. Therefore, by smoothing the inverse document frequency of each word, the inverse document frequency of some rare words with too high inverse document frequency can be reduced, and some rare words with relatively low inverse document frequency can be avoided.

[0076] Further, after step S116, the method further comprises:

[0077] S117: The inverse document frequency of all words corresponding to the preset query request and the inverse document frequency of all words in all documents in the preset document library are summarized to form an inverse document frequency word library;

[0078] Specifically, in the embodiment, the inverse document frequency word library can be constructed by step S117.

[0079] S118: The smoothed inverse document frequency of all words corresponding to the preset query request and the smoothed inverse document frequency of all words in all documents in the preset document library are summarized to form a smoothed inverse document frequency word library.

[0080] Specifically, in the training process of the bert model, the inverse document frequency of all words can be summarized to form an inverse document frequency word library, and the smoothed inverse document frequency of all words can be summarized to form a smoothed inverse document frequency word library, and they can be stored correspondingly, so that after the model training is completed, when the user uses the vector retrieval model to retrieve, the user inputs the text to be retrieved, the corresponding inverse document frequency and smoothed inverse document frequency can be directly called, the conversion process is simplified, the speed of obtaining the inverse document frequency and the smoothed inverse document frequency is improved, and the user uses more conveniently and quickly.

[0081] Please refer to Figure 2 Further, step S130 specifically comprises:

[0082] S1301: Obtain the bert model, input the preset query request and the document in the positive sample and the negative sample into the bert model respectively, and obtain the vector of each word in the preset query request and the document in the positive sample pair and the negative sample pair respectively;

[0083] Specifically, after the preset query request and the document in the positive sample and the negative sample are input into the bert model, the last layer of the bert model can output the vector corresponding to each word. For example, the vector corresponding to the word t can be vt.

[0084] S1302: According to the inverse document frequency of each word in the preset query request, the weight of each word in the positive sample pair and the negative sample pair in the document and the preset query request is determined respectively;

[0085] Specifically, the preset query request can be represented as query, the document can be represented as doc, and the weight of each word in the query and the doc can be determined according to the idf_smooth of each word in the query and the doc. Taking the calculation of the weight of each word in the query as an example:

[0086]

[0087] t1 is a word in the query; w q is the weight of the word t1; idf_smooth t1 is the smooth inverse document frequency of the word t1 in the query; s1 is any word in the query; idf_smooth s1 is the smooth inverse document frequency of the word s1 in the query.

[0088] The weight of each word in the doc:

[0089]

[0090] t2 is a word in the doc; w d is the weight of the word t2; idf_smooth t2 is the smooth inverse document frequency of the word t2 in the doc; s2 is any word in the doc; idf_smooth s2 is the smooth inverse document frequency of the word s2 in the doc.

[0091] S1303: Determine the training text vector of the preset query request and the document in the positive sample pair and the negative sample pair respectively based on the vector corresponding to each word and the weight.

[0092] Specifically, the text vector of the query and the doc can be calculated by using the weighted aggregation method. Taking the calculation of the text vector of the query as an example:

[0093]

[0094] v q is the text vector of the query; w tq is the weight of the word t1; t1 represents a word in ; v t1 is the vector of the word t1.

[0095] The text vector of the doc:

[0096]

[0097] v d is the text vector of the doc; w td is the weight of the word t2; t2 represents a word in the doc; vt2 is a vector of words t2.

[0098] It should be noted that the embodiment mainly aims at the weighting of rare words, and thus v q and v d can also be obtained based on the weighting of rare words, that is, the word t1 can be a rare word in the query, and the word t2 can be a rare word in the document.

[0099] S1304: determining the similarity between the preset query request and the document in the positive sample pair and the negative sample pair respectively according to the training text vector of the preset query request and the document;

[0100] Specifically, by calculating the similarity between the preset query request and the document in the positive sample pair and the negative sample pair, the abstract and subjective "relevance" concept is converted into a specific and calculable numerical score. Thus, the user can more intuitively understand the relevance between the preset query request and the document.

[0101] The similarity between the preset query request and the document in the positive sample pair and the negative sample pair can be obtained by the following formula:

[0102]

[0103] S1305: performing contrastive training on the bert model through the similarity between the preset query request and the document in the positive sample pair and the negative sample pair, to obtain a vector retrieval model.

[0104] Specifically, by performing contrastive training on the bert model through the positive sample pair and the negative sample pair, the representation ability of the vector retrieval model obtained after training can be improved and the interpretability can be enhanced, so that the BERT model can capture the context, break through the keyword matching limitation, and also enable the semantically similar texts to be clustered in the vector space. In addition, the model discrimination ability can also be enhanced. And by performing contrastive training through the similarity obtained by weighting and aggregating the rare words, the vector retrieval model after training can focus on the core semantic rare words and retain more information of the rare words.

[0105] More specifically, the contrastive training can be completed through a loss function. Through the loss function, the vector retrieval model finally obtained can have better representation ability and can more accurately and quickly determine the relevant files in the retrieval process.

[0106] It can be understood that by weighting and aggregating each vector in the preset query request and the document according to the inverse document frequency of the rare words in the aggregation stage, the vector retrieval model can focus on the rare words in the aggregation stage, and can more accurately represent the semantics of the text, so as to achieve more accurate retrieval.

[0107] Please refer to Figure 3 Further, the input word vector and the text vector corresponding to the text to be retrieved are obtained through the vector retrieval model, and specifically comprising:

[0108] S140: Obtain the input word vector of each word in the text to be retrieved through the vector retrieval model;

[0109] Specifically, the input word vector is obtained in the same way as the word vector is obtained when the BERT model is trained. After inputting the text to be retrieved into the vector retrieval model, the vector retrieval model can input the input word vector corresponding to each word in the text to be retrieved.

[0110] S150: Obtain the target smoothed inverse document frequency of each word in the text to be retrieved from the smoothed inverse document frequency library;

[0111] Specifically, after obtaining the input word vector of the text to be retrieved, the corresponding smoothed inverse document frequency can be matched from the smoothed inverse document frequency library according to the input word vector of each word in the text to be retrieved. Thus, the calling of the smoothed inverse document frequency is realized, so that the user can use more quickly and efficiently.

[0112] S160: Determine the target weight of each word in the text to be retrieved based on the target smoothed inverse document frequency;

[0113] Specifically, the target weight of each word in the text to be retrieved can be obtained in the same way as the weight of each word in the query and doc is calculated when the BERT model is trained.

[0114] The target weight w f is:

[0115]

[0116] h is a word in the text to be retrieved f, which can be a rare word in the text to be retrieved; w f is the weight of the word h; idf_smooth h is the smoothed inverse document frequency of the word h in the text to be retrieved f; s3 is any word in the text to be retrieved f; idf_smooth s3 is the smoothed inverse document frequency of the word s3 in the query.

[0117] It should be noted that in step S40, the word weight of the rare word can be determined through step S160, that is, after obtaining the target weight of all words in the text to be retrieved, the rare word can be selected from all words in the text to be retrieved, and the target weight corresponding to the rare word is determined, thereby avoiding the process of determining the rare word weight again, thereby improving the processing efficiency of the text to be retrieved.

[0118] S170: determining a text vector of the text to be searched based on the input word vector of each word in the text to be searched and the target weight.

[0119] Specifically, the text vector vf of the text to be searched f is:

[0120] ;

[0121] v f is the text vector of the text to be searched f; w f is the target weight of the word h; h represents a word in the text to be searched f; v h is the vector of the word h.

[0122] It can be understood that in the embodiment, the input word vector, the target weight and the text vector are determined by the same method when the bert model is trained, so that the processing logic of the vector retrieval model obtained after training for the text to be searched is consistent with that during training, thereby making the vector retrieval model better preserve the information of rare words and more accurately represent the semantics of the text, thereby achieving more accurate retrieval.

[0123] Further, step S20 specifically comprises:

[0124] S201: determining the vector of each word in the text to be searched and the target similarity of the text to be searched according to the vector dot product of the input word vector and the text vector.

[0125] Specifically, in the embodiment, the vector of each word in the text to be searched and the target similarity of the text to be searched can be determined by the vector dot product. The target similarity score(h, f) is:

[0126]

[0127] v h is the input word vector in the text to be searched f; v f is the text vector of the text to be searched f.

[0128] It can be understood that by determining the target similarity by the vector dot product, the semantic relevance can be converted into a numerical value, thereby facilitating sorting by the target similarity to quickly determine the documents related to the current word. And according to the target similarity, the user can more intuitively understand the semantic relationship in natural language.

[0129] Optionally, the comparison result can also be comparing the normalized similarity of the core vocabulary including the rare words in the text to be searched with the word weight of the corresponding rare words, thereby accurately judging the information retention and loss of each keyword according to the comparison result.

[0130] Optionally, the comparison result can also be that the normalized similarity of all words in the text to be searched is compared with the word weight of the rare word, so that the information retention and loss of each word in the text to be searched can be accurately judged according to the comparison result, and the comparison result can be more comprehensive.

[0131] Optionally, the rare word can also be determined by other ways, including but not limited to user-defined rare word.

[0132] Further, the normalized similarity is:

[0133]

[0134] Wherein, h is a target word in the text to be searched, which can be a rare word; f is the text to be searched; s is any word in the text to be searched; vocab is a smoothed inverse document frequency word library; score(f, h) is the target similarity of the target word vector and the text vector; score(f, s) is the target similarity of any word in the text to be searched and the text vector.

[0135] It can be understood that by normalizing the target similarity based on the rare word weight, the target similarity can be converted into a percentage representing importance, which is convenient for comparison with the target weight of the rare word to accurately judge the information retention and loss of each word in the text in the vector, and has better interpretability.

[0136] It should be noted that if the idf of several words in the text to be searched is low, the normalized similarity of these several words is usually small; if the normalized similarity of one of the words is relatively low, it means that the model does not learn the information of this word.

[0137] Optionally, when searching the text to be searched, the similarity calculation and comparison analysis of any word in the text to be searched and any document can be realized. That is, not only the text to be searched itself can be analyzed, but also the rare word in the text to be searched can be analyzed based on other documents. That is, the text to be searched can be analyzed based on the rare word and the documents in the preset document library. The document can be a related document of the rare word in the text to be searched, or a document selected by the user, so that the flexibility of the analysis of the text to be searched can be realized, and the user can understand the content he wants to understand.

[0138] Specifically, in the analysis based on the rare word in the text to be retrieved and other documents, the document vector in step S10 can be the preset document text vector. Specifically, based on the rare word in the text to be retrieved, the relevant document can also be determined; the target weight of each word in the text to be retrieved is calculated and determined; the text vector of the document is determined; the target similarity is determined based on the input word vector and the text vector of the document; the target similarity is normalized to obtain the normalized similarity; the word weight of the rare word is compared with the normalized similarity to obtain the comparison result. Thus, the information retention and loss degree of each word in the vector in the text to be retrieved is determined through the comparison result.

[0139] Specifically, the specific process of determining the relevant document based on the rare word can be that the vector retrieval model can initially determine at least one relevant document in the preset document library for the rare word; if there are more than or equal to 2 documents, the documents can be sorted according to the similarity of the rare word and the document, and the documents with high similarity in the front of the sorting are taken as the relevant documents of the rare word, so that the relevant documents are analyzed.

[0140] Specifically, the normalized similarity is:

[0141]

[0142] Wherein, t is a target word in the text to be retrieved, which can be any word in the text to be retrieved or a rare word; d is a document; s is any word in the document; vocab is a smoothing inverse document frequency word library; score(d,t) is the target similarity of the target word vector and the text vector; score(d,s) is the target similarity of any word in the text to be retrieved and the text vector.

[0143] For example, the smoothing inverse document frequency word library idf_smooth has 2W words, and vocab is the 2W words. d can be "xxxx sunny rainy xxxxxx", and t can be "weather", at which time the above formula can represent how much information about the weather is contained in the vector of the document d.

[0144] Optionally, when the user wants to analyze a word, the word in the text to be retrieved can be selected as the target word, which can be any word in the text to be retrieved or a rare word, so that the similarity analysis of any word and any document can be realized, thereby improving the user's use experience.

[0145] The second embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to realize the model explainable text processing method based on vector retrieval in any one of the first embodiment.

[0146] In the embodiments provided in the present application, it should be understood that "B corresponding to A" means that B is associated with A, and B can be determined according to A. However, it should also be understood that the determination of B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0147] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should also be understood by those skilled in the art that the embodiments described in the specification are optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0148] In various embodiments of the present application, it should be understood that the size of the sequence number of the above processes does not mean the inevitable sequence of execution, and the execution sequence of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0149] In the flowcharts and block diagrams in the drawings of the present application, the possible implementation architecture, function and operation of the system, method and computer program product according to various embodiments of the present application are illustrated. In this regard, each block in the flowchart or block diagram can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks can also occur in a different order from that indicated in the drawings. For example, two blocks indicated in succession can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, based on the functions involved. It should be particularly noted that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system performing the specified function or operation, or can be implemented by a combination of special-purpose hardware and computer instructions. The above has introduced in detail a model explainable text processing method based on vector retrieval and a storage medium according to the embodiments of the present application, the principles and implementation manners of the present application have been described in this paper, and the above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed; in view of the above, the content of the specification should not be understood as a limitation of the present application, any modification, equivalent replacement and improvement made within the principles of the present application should be included in the protection scope of the present application.

Claims

1. A text processing method for model interpretability based on vector retrieval, characterized in that: Includes the following steps: Obtain the text to be retrieved and the vector retrieval model, and obtain the input word vector and text vector corresponding to the text to be retrieved through the vector retrieval model; The vector retrieval model is obtained by comparing and training the BERT model with constructed positive sample pairs and negative sample pairs; the positive sample pairs include a preset query request and documents related to the preset query request, and the negative sample pairs include a preset query request and documents unrelated to the preset query request. Determine the target similarity between the input word vector and the text vector; The target similarity is normalized to obtain a normalized similarity; the normalization process converts the target similarity into a percentage representing importance. The normalized similarity is: Where t is the target word in the text to be retrieved; f is the text to be retrieved; s is any word in the text to be retrieved; vocab is a smoothed inverse document frequency vocabulary; score(f,t) is the target similarity between the target word vector and the text vector; score(f,s) is the target similarity between any word in the text to be retrieved and the text vector. An inverse document frequency (IVF) lexicon is constructed. The target IVF of all words in the text to be retrieved is determined from the IVF lexicon. Rare words in the text to be retrieved are determined based on the target IVF of all words in the text. The word weights of the rare words are obtained, and the word weights of the rare words are compared with normalized similarity to obtain the comparison results. The IVF lexicon is determined based on the total number of documents in a preset document library and the number of documents in which each word appears. The IVF lexicon represents the prevalence of each word in the preset query request. The word weights of the rare words are: ； Where h is a rare word in the text f to be retrieved; wf is the word weight of the rare word h; idf_smoothh is the smoothed inverse document frequency of the rare word h in the text f to be retrieved; s3 is any word in the text f to be retrieved; idf_smooths3 is the smoothed inverse document frequency of the word s3 in the text f to be retrieved; and query is the preset query request.

2. The text processing method for model interpretability based on vector retrieval as described in claim 1, characterized in that: The positive and negative sample pairs are obtained in the following way: Obtain a preset query request and a preset document library, wherein the preset document library includes multiple documents; Construct positive and negative sample pairs based on the preset query request and document.

3. The text processing method for model interpretability based on vector retrieval as described in claim 2, characterized in that: Before constructing the positive and negative sample pairs based on the preset query request and document, the method further includes: Obtain the inverse document frequency of all documents in the preset document library and each word in the preset query request; The inverse document frequency of each word is smoothed to obtain the smoothed inverse document frequency of each word.

4. The text processing method for model interpretability based on vector retrieval as described in claim 3, characterized in that: Obtain the BERT model, and perform comparative training on the BERT model based on the positive and negative samples to obtain a vector retrieval model, specifically including: To obtain the BERT model, input the preset query requests and documents from the positive and negative samples into the BERT model respectively, and obtain the vectors of each word in the preset query requests and documents in the positive and negative sample pairs respectively; Based on the inverse document frequency of each word in the preset query request, determine the weight of each word in the positive sample pair and the negative sample pair in the document and the preset query request, respectively. The preset query request and the training text vector of the document in the positive sample pair and the negative sample pair are determined based on the vector and weight corresponding to each word. The similarity between the preset query request and the document in the positive and negative sample pairs is determined based on the preset query request and the training text vector of the document, respectively. The BERT model is trained by comparing the similarity between the preset query requests in positive and negative sample pairs and the documents to obtain a vector retrieval model.

5. The text processing method for model interpretability based on vector retrieval as described in claim 3, characterized in that: After smoothing the inverse document frequency (IVF) of each word to obtain the smoothed IVF of each word, the method further includes: The inverse document frequency (IVF) of all words corresponding to the preset query request and the IVF of all words in all documents in the preset document library are combined to form an inverse document frequency (IVF) word library. The smooth inverse document frequency (SNV) of all words corresponding to the preset query request and the smooth inverse document frequency of all words in all documents in the preset document library are combined to form a smooth inverse document frequency (SNV) word library.

6. The text processing method for model interpretability based on vector retrieval as described in claim 5, characterized in that: The vector retrieval model is used to obtain the input word vector and text vector corresponding to the text to be retrieved, specifically including: The vector retrieval model is used to obtain the input word vector for each word in the text to be retrieved. Obtain the target smoothed inverse document frequency (SNV) of each word in the text to be retrieved from the smoothed inverse document frequency lexicon. The target weight of each word in the text to be retrieved is determined based on the target smoothed inverse document frequency. Based on the input word vector and target weight of each word in the text to be retrieved, the text vector of the text to be retrieved is determined.

7. The text processing method for model interpretability based on vector retrieval as described in claim 6, characterized in that: Determining the target similarity between the input word vector and the text vector specifically includes: The target similarity between the input word vector and the text vector of each word in the text to be retrieved is determined based on the vector dot product of the input word vector and the text vector.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the vector retrieval-based model interpretability text processing method as described in any one of claims 1-7.

Images