A document processing method, apparatus, device, and storage medium

Abstract

A document processing method, device and equipment and storage medium are disclosed, the method comprising: determining each document needing to be searched as a to-be-searched document; inputting each to-be-searched document into a word weight model to obtain word weights of each word in the corresponding to-be-searched document output by the word weight model; adding each to-be-searched document having obtained word weights of each word to a knowledge base, so as to realize searching of the to-be-searched document in the knowledge base based on the word weights of each word. The word weight model is trained in advance using a plurality of texts and word weights of each word in the corresponding texts. It can be seen that for a document needing to be searched, the application can obtain word weights effectively representing the importance of each word in the document by using a model, and then realize searching of the corresponding document based on the obtained word weights of each word when needed, thereby effectively improving the information search accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information processing, more particularly to a document processing method, device, equipment and storage medium. BACKGROUND

[0002] With the advent of the information explosion era, a large amount of information is mixed with a large amount of worthless and irrelevant information. Information retrieval is a basic and critical task for finding important and valuable information. Through information retrieval, information with low relevance to the retrieval problem can be filtered out, and the most useful information can be presented. Most current retrieval methods use word weight based on word frequency to achieve information retrieval, but the accuracy of information retrieval achieved in this way is low. SUMMARY

[0003] The purpose of the present application is to provide a document processing method, device, equipment and storage medium, which can effectively improve the accuracy of information retrieval.

[0004] To achieve the above purpose, the present application provides the following technical scheme:

[0005] A document processing method, comprising:

[0006] determining each document to be retrieved as a to-be-retrieved document;

[0007] inputting each to-be-retrieved document into a word weight model to obtain the word weight of each word in the corresponding to-be-retrieved document output by the word weight model; wherein the word weight model is obtained by pre-training using a plurality of texts and the word weight of each word in the corresponding text;

[0008] adding each to-be-retrieved document having obtained the word weight of each word contained therein to a knowledge base for implementing retrieval of the to-be-retrieved document in the knowledge base based on the word weight of each word contained therein.

[0009] Preferably, the word weight model is obtained by training using a plurality of texts and the word weight of each word in the corresponding text, comprising:

[0010] obtaining a data set containing a plurality of similar pairs; wherein a single similar pair contains a single question and the document title of a plurality of documents corresponding to the single question;

[0011] performing word segmentation processing on each similar pair contained in the data set, and determining the word weight of each word in each question contained in each similar pair based on the result of the word segmentation processing using a similar question statistical method;

[0012] using each question contained in each similar pair in the data set and the word weight of each word in each question as training corpus, and training the corresponding word weight model using the training corpus.

[0013] Preferably, a similarity question statistical method is used to determine the word weight of each word in each question contained in each similarity pair based on the results of word segmentation, including:

[0014] The word weights of each word in each question within each similarity pair are determined using the following formula:

[0015] ;

[0016] in, For the t-th word in question q, for The word weights are given by D, which is the set of document titles contained in the similarity pair to which question q belongs, and n and |D| are the number of document titles in set D.

[0017] Preferably, the corresponding word weight model is trained using the training corpus, including:

[0018] The DeepCT model is trained using the aforementioned training corpus to obtain the corresponding word weight model.

[0019] Preferably, before inputting each document to be retrieved into the word weight model, the process further includes:

[0020] The process involves uniformly encoding each document to be retrieved, removing invalid characters, and creating an index.

[0021] Preferably, before adding each document to be retrieved, for which the word weights of each included word have been obtained, to the knowledge base, the process further includes:

[0022] For each document to be retrieved, perform word frequency statistics and part-of-speech tagging, and add each document to the knowledge base with the word weights of each word already obtained, so that the document to be retrieved can be retrieved in the knowledge base based on word frequency statistics, part-of-speech tagging and word weights of each word.

[0023] Preferably, after adding all the documents to be retrieved, for which the word weights of each included word have been obtained, to the knowledge base, the process further includes:

[0024] The question to be searched is identified as the current question. Based on the word frequency statistics, part-of-speech tags, and word weights of each word in the document to be searched, the document to be searched is retrieved from the knowledge base through matching search, interactive search, or hybrid search.

[0025] A document processing apparatus, comprising:

[0026] The document identification module is used to: identify each document to be retrieved as a document to be retrieved;

[0027] The weight determination module is configured to input each of the to-be-retrieved documents into a word weight model to obtain word weights of each word in the corresponding to-be-retrieved document output by the word weight model, wherein the word weight model is obtained by pre-training using a plurality of texts and word weights of each word in the corresponding texts.

[0028] The document storage module is configured to add each of the to-be-retrieved documents for which the word weights of each word contained therein have been obtained into a knowledge base to implement retrieval of the to-be-retrieved documents in the knowledge base based on the word weights of each word contained therein.

[0029] An electronic device comprises:

[0030] A memory configured to store a computer program;

[0031] A processor configured to implement the steps of the document processing method according to any one of the preceding items when executing the computer program.

[0032] A computer readable storage medium having a computer program stored thereon, the computer program being configured to implement the steps of the document processing method according to any one of the preceding items when executed by a processor.

[0033] The present application provides a document processing method, device, equipment and storage medium, the method comprises: determining each document needing retrieval as a to-be-retrieved document; inputting each to-be-retrieved document into a word weight model to obtain word weights of each word in the corresponding to-be-retrieved document output by the word weight model; adding each to-be-retrieved document for which the word weights of each word contained therein have been obtained into a knowledge base to implement retrieval of the to-be-retrieved documents in the knowledge base based on the word weights of each word contained therein; wherein the word weight model is obtained by pre-training using a plurality of texts and word weights of each word in the corresponding texts. The present application pre-trains a word weight model using a plurality of texts and word weights representing the importance of each word in the corresponding texts, and then obtains word weights of each word in the document needing retrieval using the word weight model, and adds the document for which the word weights of each word contained therein have been obtained into a knowledge base to implement document retrieval in the knowledge base based on the word weights of each word contained therein subsequently. It can be seen that for the document needing retrieval, the present application can obtain word weights effectively representing the importance of each word in the document using the model, and then implement retrieval of the corresponding document based on the obtained word weights of each word when needed, which can effectively improve the information retrieval accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the accompanying drawings in the following description only aim at the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art without creative effort based on the provided drawings are within the protection scope of the present application.

[0035] Figure 1 A flowchart of a document processing method provided by an embodiment of the present application;

[0036] Figure 2 An example diagram of a similar pair in a document processing method provided by an embodiment of the present application;

[0037] Figure 3 An example diagram of a similar pair with a corresponding word weight calculated in a document processing method provided by an embodiment of the present application;

[0038] Figure 4 An implementation schematic diagram of a DeepCT model in a document processing method provided by an embodiment of the present application;

[0039] Figure 5 A specific implementation schematic diagram of a document processing method provided by an embodiment of the present application;

[0040] Figure 6 A structural schematic diagram of a document processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present application.

[0042] Please refer to Figure 1 which shows a document processing method provided by an embodiment of the present application, and can include:

[0043] S11: Determine that each document to be retrieved is a to-be-retrieved document.

[0044] The document to be retrieved is the document to be retrieved that needs to be added to the knowledge base. In addition, the embodiments of the present application can be applied to the financial field, and of course, can be applied to other fields according to actual needs, which are also within the protection scope of the present application.

[0045] S12: input each to-be-retrieved document into the word weight model respectively to obtain word weights of each word in the corresponding to-be-retrieved document output by the word weight model; wherein the word weight model is obtained by pre-training using a plurality of texts and word weights of each word in the corresponding texts.

[0046] In order to make the word weight set for each word in the document fully reflect the importance of the word in the corresponding document, the embodiment of the present application can pre-acquire a plurality of texts (which can be sentences, paragraphs, etc.) and word weights of each word contained in each text (the word weight is data information corresponding to the importance of the word in the text based on the importance of the word in the text), and then use these texts and the word weights of each word contained in the corresponding texts as training corpus to train the word weight model; and then when the word weight of each word in an arbitrary document is needed, the arbitrary document is input into the word weight model to obtain the word weight set for each word in the arbitrary document output by the word weight model.

[0047] S13: add each to-be-retrieved document whose contained word weight has been obtained into the knowledge base for implementing retrieval of the to-be-retrieved document in the knowledge base based on the contained word weight of each word.

[0048] After obtaining the word weight of each word contained in each document, the word weight of each word in the corresponding document can replace the word frequency weight of the corresponding word, and then when knowledge retrieval is performed on the knowledge base, the retrieval can be implemented based on the word weight of each word in each document; in addition, the implementation principle of the retrieval based on the word weight of each word in the document is consistent with that of the retrieval based on the word frequency weight of each word in the prior art, and thus will not be described in detail.

[0049] The present application pre-trains the word weight model using a plurality of texts and word weights representing the importance of each word in the corresponding texts, and then obtains the word weight of each word in a document to be retrieved using the word weight model, and adds the document whose contained word weight has been obtained into the knowledge base for implementing document retrieval in the knowledge base based on the contained word weight of each word in the subsequent. It can be seen that for a document to be retrieved, the present application can obtain the word weight which can effectively represent the importance of each word in the document using the model, and then implement retrieval of the corresponding document based on the obtained word weight of each word when needed, which can effectively improve the information retrieval accuracy.

[0050] The document processing method provided by the embodiment of the present application is trained using a plurality of texts and word weights of each word in the corresponding texts to obtain a word weight model, which can include:

[0051] obtain a data set containing a plurality of similar pairs; wherein a single similar pair contains a single question and document titles of a plurality of documents corresponding to the single question; obtain a data set containing a plurality of similar pairs; wherein a single similar pair contains a single question and document titles of a plurality of documents corresponding to the single question;

[0052] The similarity pairs in the dataset are segmented into words, and the word weights of each word in each question in each similarity pair are determined based on the results of the segmentation using a similarity question statistics method.

[0053] The training corpus is obtained by using the questions contained in each similar pair in the dataset and the word weights of each word in each question, and then using the training corpus to train the corresponding word weight model.

[0054] The word weights of each word in each question within each similarity pair are determined based on the results of word segmentation using a similarity question statistical method. This can include:

[0055] The word weights of each word in each question within each similarity pair are determined using the following formula:

[0056] ;

[0057] in, For the t-th word in question q, for The word weights are given by D, which is the set of document titles contained in the similarity pair to which question q belongs, and n and |D| are the number of document titles in set D.

[0058] The corresponding word weight model can be trained using training corpora, which may include:

[0059] The DeepCT (Deep Context Word Weighting Framework) model is trained using training corpus to obtain the corresponding word weighting model.

[0060] It should be noted that the DeepCT model can be used to implement the word weight model in this embodiment, thereby leveraging the advantages of the DeepCT model to effectively improve the accuracy of word weight determination. Furthermore, to ensure that the word weights learned by the word weight model effectively reflect their importance, the word weight model construction process in this embodiment may specifically include:

[0061] The open source question and answer data set is screened to construct a data set for model training; wherein the open source question and answer data set includes a plurality of samples, one sample includes one question and a plurality of corresponding documents, each document includes a title, the similarity of the title and the corresponding question is high, therefore, for a paragraph document, a question and title similarity pair can be constructed therefrom, a single similarity pair includes a single question and the title of each document corresponding to the single question, generally one question in a single similarity pair can correspond to 5-15 titles, and for a long document, the document and the corresponding title can be selected as a similarity pair, generally one document corresponds to 4-5 titles, at this time the document is equivalent to the question, in order to facilitate unified description, in the embodiment of the application, when the similarity pair includes a document and a corresponding title, the document included in the similarity pair is also referred to as a question, thereby obtaining a screened data set including a plurality of similarity pairs (an example diagram of the similarity pair is shown in FIG. 1). Figure 2 An example diagram of the similarity pair is shown in FIG. 1.

[0062] The screened data set is subjected to word segmentation processing, then the word weight of each word included in each question in the data set is counted according to the similar question, and then the question (i.e. the text) included in each similarity pair and the word weight of each word included therein are taken as training corpus (an example diagram of the similarity pair in which the word weight of the corresponding word is determined is shown in FIG. 2). Figure 3 An example diagram of the similarity pair in which the word weight of the corresponding word is determined is shown in FIG. 2.

[0063] .

[0064] The implementation of the DeepCT model can be as shown in FIG. 3, which includes two main parts: (1) generating context-aware word embedding through BERT; (2) predicting word weights through linear regression. Specifically, it can include: Figure 4

[0065] Context-aware word embedding generation: the most critical issue to estimate the importance of a word in a specific text is to generate features that can describe the relationship between a word and the context of the text; DeepCT uses the currently best-performing BERT model to extract the context features of the word;

[0066] Mapping to target weights: the context-aware word embedding is a feature vector containing the syntactic and semantic information of the word in a specific context, and DeepCT linearly transforms these feature vectors into the importance score of the word:

[0067] ;

[0068] wherein, is the context representation vector of the tth word in the text c, and w and b are the weight and bias of the linear transformation, respectively;

[0069] ​DeepCT trains a regression task for each word: given the true weight of each word in a document c, denoted as , …, , the goal is to minimize the mean squared error between the true label y and the predicted label :

[0070] ;

[0071] The range of the predicted word weights in the prediction process is one , but since the true labels from training are mostly between [0, 1], most of the predicted weights are actually also between [0, 1]. In the process of building the index, the query question and the weighting method of the document accept any non-negative weight, so the predicted items with negative weights will be discarded;

[0072] Test the data of the power knowledge base and all the data of the FAQ knowledge base, and after the test is completed, input some short sentences and see how the word weights given by the model are:

[0073]

[0074] For the same word in different contexts, the importance will also be different, and an example is given here:

[0075]

[0076] The first sentence in the example is "My phone is broken", and the focus should be on the phone. In the second sentence "the phone's graphics processing capability", the focus is not only on the phone, but also on its graphics processing capability, so the weight value of the phone decreases.

[0077] The document processing method provided by the embodiment of the application can further include the following before each document to be searched is input into the word weight model:

[0078] Uniformly encoding, removing invalid characters and indexing each document to be searched.

[0079] It should be noted that the data preprocessing of all documents in the document set to be searched in the embodiment of the application is convenient for subsequent implementation of corresponding document processing and storage and the like. The data preprocessing can include uniform encoding, removing invalid characters and indexing, etc. The uniform encoding can mean that the text format of all documents is converted into the utf-8 encoding format through encoding and decoding, such as converting the text in the unicode encoding into the text format in the utf-8 encoding. The removing invalid characters can mean removing characters without actual meaning in the document, such as removing spaces and specific symbols. The indexing can be implemented based on any manner in the prior art, and then used in subsequent retrieval and the like.

[0080] The document processing method provided by the embodiment of the present application can further include the following steps before adding each to-be-retrieved document with the obtained word weight of each included word into the knowledge base:

[0081] The word frequency statistics and the part-of-speech tagging are performed on each to-be-retrieved document, and the step of adding each to-be-retrieved document with the obtained word weight of each included word into the knowledge base is executed, so as to realize the retrieval of the to-be-retrieved document in the knowledge base based on the word frequency statistics, the part-of-speech tagging and the word weight of each included word.

[0082] In order to further improve the accuracy of the knowledge base retrieval, the embodiment of the present application also performs the word frequency statistics, the part-of-speech tagging and the like required by the traditional retrieval on all the documents to be retrieved; the word frequency statistics is to obtain the word frequency (tf) of each word through the open source retrieval tool Lucene, which is used to calculate the relevance score of the problem and the document in the Lucene retrieval, and the part-of-speech tagging is to filter out the words with the related part-of-speech such as the virtual words and adverbs by tagging the part-of-speech of each word, so as to realize the stop word filtering.

[0083] The document processing method provided by the embodiment of the present application can further include the following steps after adding each to-be-retrieved document with the obtained word weight of each included word into the knowledge base:

[0084] The problem to be implemented for retrieval is the current problem, and the retrieval of the to-be-retrieved document in the knowledge base is realized through the matching type retrieval, the interactive retrieval or the mixed type retrieval based on the word frequency statistics, the part-of-speech tagging and the word weight of each included word of each to-be-retrieved document.

[0085] In order to further improve the accuracy of the knowledge retrieval, the embodiment of the present application stores the document after the word frequency statistics, the part-of-speech tagging and the word weight calculation into the knowledge base as the to-be-retrieved document, and then the semantic retrieval mode such as the matching type retrieval, the interactive retrieval or the mixed type retrieval can be used to retrieve the related knowledge in the knowledge base according to the user's question and provide the user with the related knowledge; the matching type retrieval uses the Lucene framework to realize the word matching, the interactive retrieval uses the bert-search model to perform the semantic retrieval, and the mixed type retrieval considers the relevance scores of Lucene and bert-search at the same time, and performs the weighted summation of the two to obtain the retrieval result of the final candidate document. In addition, the retrieval based on the word frequency statistics and the part-of-speech tagging has the same meaning as the corresponding concept in the prior art, and thus will not be described in detail here.

[0086] In a specific implementation manner, the implementation process of the document processing method provided by the embodiment of the present application can be as follows Figure 5As shown, the candidate documents are the documents to be retrieved; it can be seen that this application can improve the accuracy of document processing, optimize word weights using the DeepCT model, obtain the weight information of each word in the document, and provide an analytical basis for intent recognition, etc.

[0087] This invention also provides a document processing device, such as... Figure 6 As shown, it may include:

[0088] Document identification module 11 is used to: identify each document to be retrieved as a document to be retrieved;

[0089] The weight determination module 12 is used to: input each document to be retrieved into the word weight model to obtain the word weight of each word in the corresponding document to be retrieved, as output by the word weight model; wherein, the word weight model is trained in advance using multiple texts and the word weight of each word in the corresponding texts;

[0090] The document storage module 13 is used to add each document to be retrieved, for which the word weights of each word have been obtained, to the knowledge base, so as to realize the retrieval of the document to be retrieved in the knowledge base based on the word weights of each word.

[0091] The document processing apparatus provided in this embodiment of the invention may further include:

[0092] The model training module is used to: acquire a dataset containing multiple similar pairs; wherein a single similar pair contains a single question and the document titles of multiple documents corresponding to that single question; perform word segmentation on each similar pair in the dataset, and determine the word weight of each word in each question in each similar pair based on the results of word segmentation using a similar question statistics method; use each question in each similar pair in the dataset and the word weight of each word in each question as training corpus, and train the corresponding word weight model using the training corpus.

[0093] The model training module may include:

[0094] The unit is defined to determine the word weights of each word in each question within each similarity pair according to the following formula:

[0095] ;

[0096] in, For the t-th word in question q, for The word weights are given by D, which is the set of document titles contained in the similarity pair to which question q belongs, and n and |D| are the number of document titles in set D.

[0097] The training unit is configured to train the DeepCT model by using a training corpus to obtain a corresponding word weight model.

[0098] The document processing apparatus provided in the embodiment of the present application can further include:

[0099] The preprocessing module is configured to perform uniform coding, invalid character removal and index establishment on each document to be searched before inputting each document to be searched into the word weight model.

[0100] The document processing apparatus provided in the embodiment of the present application can further include:

[0101] The processing module is configured to perform word frequency statistics and part-of-speech tagging on each document to be searched before adding each document to be searched, for which the word weight of each word contained has been obtained, into the knowledge base, and execute the step of adding each document to be searched, for which the word weight of each word contained has been obtained, into the knowledge base, so as to realize the search of the document to be searched in the knowledge base based on the word frequency statistics, the part-of-speech tagging and the word weight of each word contained.

[0102] The document processing apparatus provided in the embodiment of the present application can further include:

[0103] The search module is configured to acquire a current question as a question to be searched after each document to be searched, for which the word weight of each word contained has been obtained, is added into the knowledge base, and realize the search of the document to be searched in the knowledge base by matching type search, interactive search or mixed search based on the word frequency statistics, the part-of-speech tagging and the word weight of each word contained of each document to be searched.

[0104] The embodiment of the present application further provides an electronic device, which can include:

[0105] The memory is configured to store a computer program;

[0106] The processor is configured to implement the steps of any one of the above document processing methods when executing the computer program.

[0107] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program can implement the steps of any one of the above document processing methods when executed by a processor.

[0108] It should be noted that the descriptions of the related parts of the document processing apparatus, device and storage medium provided in the embodiment of the present application can refer to the detailed descriptions of the corresponding parts in the document processing method provided in the embodiment of the present application, which will not be repeated here. In addition, the parts of the above technical solutions provided in the embodiment of the present application which are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as to avoid excessive repetition.

[0109] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to those skilled in the art and are within the scope of the following claims, defined solely by the words of the claims themselves, in which the patent is sought. Therefore, the application is not limited to the described embodiments and instead is construed to cover all alternatives falling within the scope of the application.

Claims

1. A document processing method characterized by, The method comprises the following steps: determining that each document to be searched is a to-be-searched document; inputting each to-be-searched document into a word weight model to obtain word weights of each word in the corresponding to-be-searched document output by the word weight model; wherein the word weight model is obtained by pre-training using a plurality of texts and word weights of each word in the corresponding texts; adding each to-be-searched document with the obtained word weights of each word to a knowledge base, so as to realize searching of the to-be-searched document in the knowledge base based on the word weights of each word contained in the to-be-searched document; wherein the word weight model is obtained by training using a plurality of texts and word weights of each word in the corresponding texts, comprising: obtaining a data set containing a plurality of similar pairs; wherein a single similar pair contains a single question and document titles of a plurality of documents corresponding to the single question; performing word segmentation processing on each similar pair contained in the data set, and determining word weights of each word in each question contained in each similar pair based on the result of the word segmentation processing by using a similar question statistical method; using each question contained in each similar pair in the data set and the word weights of each word in each question as training corpus, and training a corresponding word weight model using the training corpus; wherein the word weights of each word in each question contained in each similar pair are determined based on the result of the word segmentation processing by using a similar question statistical method, comprising: determining the word weights of each word in each question contained in each similar pair according to the following formula: ； wherein, is the t-th word in the question q, is the word weight of the word, D is the set of document titles contained in the similar pair to which the question q belongs, and n and |D| are the number of document titles in the set D. wherein the corresponding word weight model is obtained by training using the training corpus, comprising: training a DeepCT model using the training corpus to obtain a corresponding word weight model; wherein the DeepCT model generates contextually word embedding through BERT, and predicts word weights through linear regression; wherein before adding each to-be-searched document with the obtained word weights of each word to the knowledge base, the method further comprises: performing word frequency statistics and part-of-speech tagging on each to-be-searched document, and performing the step of adding each to-be-searched document with the obtained word weights of each word to the knowledge base, so as to realize searching of the to-be-searched document in the knowledge base based on the word frequency statistics, the part-of-speech tagging and the word weights of each word contained in the to-be-searched document.

2. The method of claim 1, wherein, Before inputting each to-be-searched document into the word weight model, the method further comprises: performing uniform coding, removing invalid characters and indexing operations on each to-be-searched document.

3. The method of claim 1, wherein, After adding each to-be-searched document with the obtained word weights of each word to the knowledge base, the method further comprises: obtaining a question to be searched as a current question, and realizing searching of the to-be-searched document in the knowledge base by matching type searching, interactive searching or mixed type searching based on the word frequency statistics, the part-of-speech tagging and the word weights of each word contained in each to-be-searched document.

4. A document processing apparatus characterized by comprising: The method comprises the following steps: a document determination module, configured to determine that each document to be searched is a to-be-searched document; The weight determination module is configured to input each of the to-be-retrieved documents into a word weight model to obtain word weights of each word in the corresponding to-be-retrieved document output by the word weight model, wherein the word weight model is obtained by pre-training using a plurality of texts and word weights of each word in the corresponding texts; The document storage module is configured to add each of the to-be-retrieved documents for which the word weights of each word contained therein have been obtained into a knowledge base to implement retrieval of the to-be-retrieved documents in the knowledge base based on the word weights of each word contained therein. The first model training module is configured to obtain a data set containing a plurality of similar pairs, wherein each similar pair contains a single question and document titles of a plurality of documents corresponding to the single question, perform word segmentation processing on each similar pair contained in the data set, determine word weights of each word in each question contained in each similar pair based on the result of the word segmentation processing using a similar question statistical method, use each question contained in each similar pair and the word weights of each word in each question as training corpus, and train the corresponding word weight model using the training corpus. The first model training module is configured to obtain a data set containing a plurality of similar pairs, wherein each similar pair contains a single question and document titles of a plurality of documents corresponding to the single question, perform word segmentation processing on each similar pair contained in the data set, determine word weights of each word in each question contained in each similar pair based on the result of the word segmentation processing using a similar question statistical method, use each question contained in each similar pair and the word weights of each word in each question as training corpus, and train the corresponding word weight model using the training corpus. The word weight calculation module is configured to determine the word weights of each word in each question contained in each similar pair based on the result of the word segmentation processing using a similar question statistical method, including: The word weight calculation module is configured to determine the word weights of each word in each question contained in each similar pair according to the following formula: ； wherein, is the t-th word in the question q, is the word weight of the word, D is the set of document titles contained in the similar pair to which the question q belongs, and n and |D| are the number of document titles in the set D. The second model training module is configured to train the corresponding word weight model using the training corpus. The first model training module is configured to train a DeepCT model using the training corpus to obtain the corresponding word weight model, wherein the DeepCT model generates contextually word embeddings through BERT and predicts word weights through linear regression. The processing module is configured to perform word frequency statistics and part-of-speech tagging on each to-be-retrieved document before adding each of the to-be-retrieved documents for which the word weights of each word contained therein have been obtained into the knowledge base, and execute the step of adding each of the to-be-retrieved documents for which the word weights of each word contained therein have been obtained into the knowledge base, so as to implement retrieval of the to-be-retrieved documents in the knowledge base based on the word frequency statistics, the part-of-speech tagging, and the word weights of each word contained therein.

5. An electronic device, comprising: The memory is configured to store a computer program; The processor is configured to implement the steps of the document processing method of any one of claims 1 to 3 when executing the computer program. The computer program stored on the computer readable storage medium is configured to implement the steps of the document processing method of any one of claims 1 to 3 when executed by the processor.

6. A computer-readable storage medium, characterized in that, ​

Images