Similar document retrieval method, device, electronic device and storage medium

Abstract

The present invention discloses a similar document retrieval method, device, electronic device, and storage medium, which are used to address the technical issues of high time cost and low efficiency of existing similar document retrieval methods. The method comprises the following steps: obtaining a training document library; the training document library includes multiple documents, each document having a corresponding document ID; constructing a training data set based on the documents; training a neural network using the training data set to obtain a target neural network; receiving a target document and generating a target training data set using the target document; inputting the target training data set into the target neural network to obtain a target vector for the target document; calculating the difference between the target vector and a comparison vector in a preset database, and selecting documents corresponding to comparison vectors whose difference is less than a preset threshold as similar documents.

Description

Technical Field

[0001] The present invention relates to the field of document retrieval technology, and in particular to a similar document retrieval method, device, electronic device and storage medium. Background Art

[0002] With the advancement of the information society, more and more documents (such as academic papers, novels, news, etc.) are stored in binary form on storage media such as computer hard drives. Even though the number of documents is growing rapidly, computers can still quickly retrieve documents that meet specific conditions (such as the presence of specific titles or specific keywords) from a huge document library through database-related technologies. However, simply retrieving titles or keywords is not enough to support all application scenarios. Sometimes it is necessary to retrieve a collection of similar documents to the target document. For example, a paper duplicate detection system needs to quickly detect a collection of papers that are semantically similar to the target paper. A self-media publishing review system needs to detect whether an article marked as original is actually original. Similar documents need to be eliminated during the corpus construction process. However, how to quickly retrieve a collection of similar documents to the target document from a huge document library is a difficult problem. To solve this problem, we first need a calculation method that can quantify document similarity.

[0003] Early methods for calculating document similarity used the bag-of-words model as a feature extraction method, converting the document into a vector of numbers. This vector typically uses the inverse document frequency as a weighting factor, and then calculates the cosine similarity between the two vectors to determine the similarity between the two documents. The disadvantage of the bag-of-words model is that it only considers the number of word occurrences, not the order between the words, and therefore ignores the semantic information brought by the word order.

[0004] As document feature extraction technology has further developed, a method called Doc2Vec has been proposed. This method can convert a document into a distributed memory vector. Because the document vector combined with any previous context in the document is sufficient to predict the corresponding next context, it is as if the vector remembers the entire document. Therefore, it is figuratively called a distributed memory document vector.

[0005] However, existing Doc2Vec methods mainly focus on improving the representation of documents so that the similarity between document representation vectors is as consistent as possible with the semantic similarity of documents. However, the problem of how to quickly retrieve similar documents from a huge document library has not been well solved.

[0006] Existing technologies for large-scale similar document retrieval rely on keyword indexing. However, keyword-based retrieval systems cannot meet the needs of practical applications such as originality detection, which require the ability to detect semantically similar documents. While Doc2Vec technology can reflect semantic similarity in documents through the distance between document vectors, it does not address how to retrieve similar documents from a large document repository in constant time. Open-source implementations calculate the similarity between the target document and all documents in the repository and then return a set of similar documents that meet the specified requirements. This approach is time-consuming and inefficient. Summary of the Invention

[0007] The present invention provides a similar document retrieval method, device, electronic device and storage medium, which are used to solve the technical problems of high time cost and low efficiency of existing similar document retrieval methods.

[0008] The present invention provides a similar document retrieval method, comprising:

[0009] Obtain a training document library; the training document library includes multiple documents, each document having a corresponding document ID;

[0010] Constructing a training dataset based on the document;

[0011] Using the training data set to train a neural network to obtain a target neural network;

[0012] receiving a target document, and generating a target training data set using the target document;

[0013] Inputting the target training data set into the target neural network to obtain a target vector for the target document;

[0014] The difference between the target vector and the comparison vectors in the preset database is calculated, and the documents corresponding to the comparison vectors whose difference is less than a preset threshold are regarded as similar documents.

[0015] Optionally, the step of constructing a training dataset based on the documents includes:

[0016] Divide each document into multiple word segments; each word segment has a corresponding word segment ID;

[0017] A training data set is constructed using all the documents and the word segmentation corresponding to each document.

[0018] Optionally, the step of dividing each document into a plurality of segmented words includes:

[0019] Merging consecutive spaces in each of the documents into a single space to obtain a preprocessed document;

[0020] The pre-processed document is divided into a plurality of tokens.

[0021] Optionally, the step of constructing a training dataset using all the documents and the word segmentations corresponding to each document includes:

[0022] Extracting a plurality of first segmented words from the segmented words in the document according to the arrangement order of the segmented words in the document through a preset sliding window;

[0023] The training data is generated by taking the segmentation ID of the last first segmentation in the sliding window as output and the segmentation IDs and corresponding document IDs of the first segmentations other than the last first segmentation in the sliding window as input;

[0024] Moving the sliding window according to the order of the segmented words in the document, and returning through the preset sliding window, extracting a plurality of first segmented words from the document according to the order of the segmented words in the document, until all the segmented words in the document are traversed;

[0025] All the obtained training data are used to generate a training dataset.

[0026] Optionally, the neural network includes an embedding layer, a concatenation layer, a fully connected layer, and a soft layer; and the step of using the training data set to train the neural network to obtain a target neural network includes:

[0027] The word segmentation vectors corresponding to the word segmentation ID of the first word as input are concatenated into a word segmentation matrix;

[0028] Multiplying the word segmentation matrix by a preset first parameter matrix to obtain a first branch output of the embedding layer;

[0029] Multiplying the document vector corresponding to the document ID corresponding to the first word segmentation by a preset second parameter matrix to obtain a second branch output of the embedding layer;

[0030] splicing the first branch output and the second branch output through the splicing layer to obtain the spliced spliced output vector;

[0031] Multiplying the third parameter matrix of the fully connected layer by the concatenated output vector to obtain a fully connected layer output;

[0032] Performing a Softmax transformation on the output of the fully connected layer through a Softmax layer to obtain a target output vector of the neural network;

[0033] Construct the target vector with the word ID of the first word as output;

[0034] Calculating a loss value between the target output vector and the target vector;

[0035] Updating the first parameter matrix, the second parameter matrix, and the third parameter matrix using the loss value;

[0036] Determine whether the current number of training times is equal to the preset number threshold;

[0037] If not, return to the step of concatenating the word vectors corresponding to the word ID of the first word as input into a word segmentation matrix;

[0038] If so, a target neural network is constructed based on the updated first parameter matrix, second parameter matrix, and third parameter matrix.

[0039] Optionally, the target document has multiple target segmentations; the step of receiving the target document and generating a target training dataset using the target document includes:

[0040] receiving a target document, and extracting a plurality of second segmented words from the target segmented words in the target document through the sliding window according to the arrangement order of the target segmented words in the target document;

[0041] The target training data is generated by taking the segmentation ID of the last second segmentation in the sliding window as output and the segmentation IDs of the second segmentations other than the last second segmentation in the sliding window as input;

[0042] The sliding window is moved according to the order of the target segmented words in the target document, and the sliding window is returned to extract a plurality of second segmented words from the target segmented words in the target document according to the order of the target segmented words in the target document, until all the target segmented words in the target document are traversed;

[0043] All the obtained target training data are used to generate a target training dataset.

[0044] The present invention also provides a similar document retrieval device, comprising:

[0045] An acquisition module is used to acquire a training document library; the training document library includes multiple documents, each document having a corresponding document ID;

[0046] A training data set construction module, used to construct a training data set according to the document;

[0047] A target neural network training module is used to train a neural network using the training data set to obtain a target neural network;

[0048] A target training data set generation module is used to receive a target document and generate a target training data set using the target document;

[0049] A target vector acquisition module, configured to input the target training data set into the target neural network to obtain a target vector for the target document;

[0050] The similar document acquisition module is used to calculate the difference between the target vector and the comparison vectors in the preset database, and take the documents corresponding to the comparison vectors whose difference is less than a preset threshold as similar documents.

[0051] Optionally, the training data set construction module includes:

[0052] The word segmentation submodule is used to divide each document into multiple words; each word has a corresponding word ID;

[0053] The training data set construction submodule is used to construct a training data set using all the documents and the word segmentations corresponding to each of the documents.

[0054] The present invention further provides an electronic device, comprising a processor and a memory:

[0055] The memory is used to store program code and transmit the program code to the processor;

[0056] The processor is configured to execute any of the above similar document retrieval methods according to the instructions in the program code.

[0057] The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute any of the similar document retrieval methods described above.

[0058] As can be seen from the above technical solution, the present invention has the following advantages: the present invention discloses a similar document retrieval method, and specifically discloses: obtaining a training document library; the training document library includes multiple documents, each document having a corresponding document ID; constructing a training data set based on the documents; training a neural network using the training data set to obtain a target neural network; receiving a target document and generating a target training data set using the target document; training the target training data set using the target neural network to obtain a target vector for the target document; calculating the difference between the target vector and a comparison vector in a preset database, and selecting documents corresponding to comparison vectors whose difference is less than a preset threshold as similar documents. This reduces the time cost of similar document retrieval and improves retrieval efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0060] Figure 1 A flowchart of a similar document retrieval method according to an embodiment of the present invention;

[0061] Figure 2 A flowchart of a similar document retrieval method according to another embodiment of the present invention;

[0062] Figure 3 This is a structural block diagram of a similar document retrieval device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The embodiments of the present invention provide a similar document retrieval method, device, electronic device and storage medium, which are used to solve the technical problems of high time cost and low efficiency of existing similar document retrieval methods.

[0064] In order to make the purpose, features, and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0065] See also Figure 1 , Figure 1 A flowchart of the steps of a similar document retrieval method provided by an embodiment of the present invention.

[0066] The present invention provides a similar document retrieval method, which may specifically include the following steps:

[0067] Step 101: Obtain a training document library; the training document library includes multiple documents, each document having a corresponding document ID;

[0068] In an embodiment of the present invention, a large number of documents can be captured online by a crawler as a training document library to be used as samples for subsequent neural network training, wherein each document has a corresponding document ID.

[0069] Step 102: construct a training data set based on the document;

[0070] In practical applications, after obtaining the training document library, features that can be used for neural network training can be extracted from it to construct a training dataset.

[0071] Step 103, using the training data set to train the neural network to obtain a target neural network;

[0072] A neural network is a mathematical model that mimics the behavioral characteristics of animal neural networks and performs distributed, parallel information processing. This network relies on the complexity of the system to process information by adjusting the connections between its numerous nodes.

[0073] After constructing the post-training data set, the training data set can be used to train the neural network. The training data set contains the input data and actual output data required by the neural network. After inputting the input data into the neural network for training to obtain the predicted output data, the predicted output data is compared with the actual output data to adjust the parameters of the neural network. By continuously adjusting the parameters of the neural network, the trained target neural network is obtained.

[0074] Step 104: receiving a target document and generating a target training data set using the target document;

[0075] After the target neural network is trained, the target documents that actually need to be retrieved for similarity can be processed to obtain the corresponding target training data set, thereby obtaining the corresponding input data that meets the requirements of the target neural network from the target training data set.

[0076] Step 105: Input the target training data set into the target neural network to obtain the target vector of the target document;

[0077] By training the target document through the target neural network, the output data in vector form can be obtained.

[0078] Step 106 : Calculate the difference between the target vector and the comparison vectors in the preset database, and take the documents corresponding to the comparison vectors whose difference is less than a preset threshold as similar documents.

[0079] The target vector is compared with the comparison vector in the preset database, the difference between the two is calculated, and the documents corresponding to the comparison vectors whose difference is less than the preset threshold are collected, so that a set of similar documents to the target document can be obtained.

[0080] It should be noted that in actual applications, similar document retrieval is generally biased, such as searching in a collection of published academic papers. Therefore, in order to reduce the objects of retrieval, in an embodiment of the present invention, documents within the scope of the retrieval can be collected and stored in a database, and the corresponding comparison vector can be generated through the above-mentioned neural network.

[0081] The present invention discloses a similar document retrieval method, specifically comprising the following steps: obtaining a training document library; the training document library includes multiple documents, each document having a corresponding document ID; constructing a training data set based on the documents; training a neural network using the training data set to obtain a target neural network; receiving a target document and generating a target training data set using the target document; training the target training data set using the target neural network to obtain a target vector for the target document; and calculating the difference between the target vector and a comparison vector in a preset database, and selecting documents corresponding to comparison vectors whose difference is less than a preset threshold as similar documents. This reduces the time cost of similar document retrieval and improves retrieval efficiency.

[0082] See also Figure 2 , Figure 2 This is a flowchart of a similar document retrieval method provided by another embodiment of the present invention. Specifically, the following steps may be included:

[0083] Step 201: Obtain a training document library; the training document library includes multiple documents, each document having a corresponding document ID;

[0084] Step 202: Divide each document into multiple segmented words; each segmented word has a corresponding segmented word ID;

[0085] In the embodiment of the present invention, each document may be segmented to serve as a basis for subsequently generating a training data set.

[0086] In one example, step 202 may include the following sub-steps:

[0087] S21, merging consecutive spaces in each document into a single space to obtain a preprocessed document;

[0088] S22, dividing the preprocessed document into multiple word segments.

[0089] In practical applications, documents in the training document library may be collected from different platforms, so they need to be preprocessed to obtain documents with the same structure. The preprocessing step may include: merging consecutive spaces into a single space to obtain a preprocessed document.

[0090] The document is segmented using a word segmentation technique to obtain word segments for each document, wherein each word segmentation has a corresponding ID. It should be noted that the present invention does not specifically limit the word segmentation technique, and those skilled in the art can select any word segmentation technique to segment the document.

[0091] Step 203: construct a training data set using all documents and the word segmentation corresponding to each document;

[0092] In practical applications, after obtaining the training document library, features that can be used for neural network training can be extracted from it to construct a training dataset.

[0093] In one example, step 203 may include the following sub-steps:

[0094] S31, extracting a plurality of first segmented words from the segmented words of the document according to the arrangement order of the segmented words in the document through a preset sliding window;

[0095] S32, using the segmentation ID of the last first segmentation in the sliding window as output, and the segmentation IDs and corresponding document IDs of the first segmentations other than the last first segmentation in the sliding window as input, to generate training data;

[0096] S33, moving the sliding window according to the order of the word segments in the document, and returning to the step of extracting multiple first word segments from the document according to the order of the word segments in the document through the preset sliding window, until all the word segments in the document are traversed;

[0097] S34, using all the obtained training data to generate a training data set.

[0098] In practical applications, each document in the training document library can correspond to multiple pieces of training data. The present invention uses a sliding window to construct the input and output of the training data set on each pre-processed document. For example, assuming that the size of the sliding window is 5, the IDs of the first four segmented words in the document and the ID of the document can be used as the input of the first training data, and the ID of the fifth segmented word can be used as the output of the first training data. Then the sliding window can slide one segmented word to the right, and the IDs of the second to fifth segmented words and the ID of the document can be used as the input of the second training data, and the sixth word can be used as the output of the second training data. And so on, until the last word in the document. Repeat the above operation for each document to obtain a training data set generated by the entire training document library.

[0099] Step 204: training the neural network using the training data set to obtain a target neural network;

[0100] After constructing the post-training data set, the training data set can be used to train the neural network. The training data set contains the input data and actual output data required by the neural network. After inputting the input data into the neural network for training to obtain the predicted output data, the predicted output data is compared with the actual output data to adjust the parameters of the neural network. By continuously adjusting the parameters of the neural network, the trained target neural network is obtained.

[0101] In one example, step 204 may include the following sub-steps:

[0102] S401, concatenating the word segmentation vectors corresponding to the word segmentation ID of the first word as input into a word segmentation matrix;

[0103] S402, multiplying the word segmentation matrix by a preset first parameter matrix to obtain a first branch output of the embedding layer;

[0104] S403, multiplying the document vector corresponding to the document ID corresponding to the first word segmentation by a preset second parameter matrix to obtain a second branch output of the embedding layer;

[0105] S404, concatenating the first branch output and the second branch output through a concatenation layer to obtain a concatenated output vector;

[0106] S405, multiplying the third parameter matrix of the fully connected layer by the concatenated output vector to obtain the fully connected layer output;

[0107] S406, performing a Softmax transform on the output of the fully connected layer through a Softmax layer to obtain a target output vector of the neural network;

[0108] S407, constructing a target vector using the segmentation ID of the first segmentation as output;

[0109] S408, calculating the loss value between the target output vector and the target vector;

[0110] S409, using the loss value to update the first parameter matrix, the second parameter matrix, and the third parameter matrix;

[0111] S410, determining whether the current number of training times is equal to a preset number threshold;

[0112] If not, step S411 returns to the step of concatenating the word segmentation vectors corresponding to the word segmentation ID of the first word as input into a word segmentation matrix;

[0113] S412: If yes, construct a target neural network based on the updated first parameter matrix, second parameter matrix, and third parameter matrix.

[0114] In practical applications, to train a neural network, we first need to define the desired word vector and document vector sizes. In this embodiment of the present invention, we assume that the word vector size is q and the document vector size is p. Based on this, we construct a neural network and train it using the method proposed in Doc2Vec. Specifically, the process of a training step is as follows:

[0115] The input of the neural network is the one-hot vector converted from four word segmentation IDs, that is, the word segmentation vector and the one-hot vector converted from a document ID, that is, the document vector (each component of the one-hot vector corresponds to a unique ID. To convert the ID to a one-hot vector, just set the component corresponding to the ID to 1 and the rest to 0). Assuming there are M word segmentations, the size of the one-hot vector corresponding to the word segmentation is M. Assuming there are N documents, the size of the one-hot vector corresponding to the document is N.

[0116] The first layer of the neural network is the embedding layer, which has two branches. Let the first matrix parameter of the first branch be W1, whose size is q*m, and the input is represented by X 11 , is a word segmentation matrix formed by concatenating the one-hot vectors corresponding to the four word segmentations, whose size is M*4, and W1 and X 11 After matrix multiplication, the first branch output of this branch can be obtained, which is represented as X 21 , is a q*4 matrix; the second parameter matrix of the second branch can be expressed as W2, its size is p*N, and the input is expressed as X 12 , is the one-hot vector corresponding to the document, whose size is N*1, W2 and X 12 After matrix multiplication, the second branch output of this branch can be obtained, which is represented here as X 22 , is a p*1 vector.

[0117] The second layer of the neural network is the concatenation layer, which can convert the first branch output X 21 and the second branch outputs X 22 Splicing them together, we get the spliced output vector X2 of the layer output, which is a (4q+p)*1 vector.

[0118] The third layer of the neural network is the fully connected layer. The third parameter matrix of this layer is W3, whose size is N*(4q+p). The input is X2. The matrix multiplication operation of W3 and X2 is used to obtain the fully connected layer output X3, which is an N*1 vector.

[0119] The final layer of the neural network is the Softmax layer, which performs a Softmax transform on X3 to obtain the target output vector X4 of the neural network. Due to the characteristics of the Softmax transform, each component of X4 obtained after the transformation has a value range of [0, 1], and the sum of all the components equals 1.

[0120] A one-hot vector is constructed according to the word segmentation ID of the target output as the target vector. As the target of neural network learning, the target is a vector Y of size N*1. The cross entropy calculated by the target vector Y and the target output vector X4 output by the neural network is used as the loss value. The first parameter matrix W1, the second parameter matrix W2, and the third parameter matrix W3 are updated in combination with the loss value and the gradient descent method. The gradients of the first parameter matrix, the second parameter matrix, and the third parameter matrix can be obtained using the back propagation method. For details, please refer to the existing back propagation method, which will not be repeated here.

[0121] When the entire training data set has been calculated once, the neural network has completed a round of training. In an embodiment of the present invention, the neural network can be trained for multiple rounds. When multiple rounds of training are completed, the target neural network can be constructed based on the first parameter matrix, the second parameter matrix, and the third parameter matrix obtained by the latest update.

[0122] Step 205: receiving a target document and generating a target training data set using the target document;

[0123] After the target neural network is trained, the target vector of any document can be obtained through the target neural network. Similarly, the target training dataset of the target document needs to be generated first.

[0124] In one example, step 205 may include the following sub-steps:

[0125] S51, receiving a target document, and extracting a plurality of second segmented words from the target segmented words in the target document according to the arrangement order of the target segmented words in the target document through a sliding window;

[0126] S52, using the segmentation ID of the last second segmentation in the sliding window as output and the segmentation IDs of the second segmentations other than the last second segmentation in the sliding window as input to generate target training data;

[0127] S53, moving the sliding window according to the order of the target segmented words in the target document, and returning to the step of extracting multiple second segmented words from the target segmented words in the target document according to the order of the target segmented words in the target document through the sliding window, until all target segmented words in the target document are traversed;

[0128] S54: Generate a target training data set using all the obtained target training data.

[0129] In a specific implementation, the target neural network can be trained using a sliding window. Multiple second segmentations can be extracted from the target segmentation in the target document according to the order of the target segmentations in the target document. The target training data is then generated using the segmentation ID of the last second segmentation in the sliding window as output and the segmentation IDs of the second segmentations other than the last second segmentation in the sliding window as input. Finally, the sliding window is moved according to the order of the target segmentations in the target document, and the above operation is repeated until all target segmentations in the target document are traversed. The resulting set of all target training data is the target training dataset for the target document.

[0130] Step 206: Input the target training data set into the target neural network to obtain the target vector of the target document;

[0131] It should be noted that when training on the target document, the target document does not have an ID, so the document ID part needs to be ignored when constructing the training set. Then, based on the above neural network training process, some adjustments are made to the current training process. As the training process at this time, the adjustments are as follows:

[0132] The input of the neural network no longer contains the document vector corresponding to the document ID; the branch related to the target document in the embedding layer only retains one X 22 , and X 22 Now it exists as a parameter of the target neural network. At this time, the operation related to W2 no longer exists. In the adjusted entire target neural network, W1 and W3 are treated as constant terms. The calculated loss value is combined with the gradient descent method to update the only remaining parameter X in the target neural network. 22 Since there is only one document and the only updateable parameter is a p*1 vector, the training time for the target document will be relatively short, and the target vector of the target document can be obtained after completion.

[0133] Step 207 : Calculate the difference between the target vector and the comparison vectors in the preset database, and take the documents corresponding to the comparison vectors whose difference is less than a preset threshold as similar documents.

[0134] After obtaining the target document's target vector, we compare it with the comparison vectors in the pre-set database. We search for documents whose differences in each component of the target vector are less than a pre-set threshold d. This yields a set of similar documents to the target document. d is a hyperparameter that can be optimized based on the document library being used and is not specifically limited here.

[0135] It should be noted that in order to conduct targeted similarity retrieval, a collection of collected documents that do not want to be repeated (such as a collection of published academic papers) can be inserted into the database. Specifically, the document ID and character ID of each document can be stored in the same table, and then the vectors of these new documents are calculated and inserted into a comparison table. The comparison table corresponds to p+1 fields, one field stores the document ID, and the other p fields correspond to the values of the p components of the document vector.

[0136] The present invention discloses a similar document retrieval method, specifically comprising the following steps: obtaining a training document library; the training document library includes multiple documents, each document having a corresponding document ID; constructing a training data set based on the documents; training a neural network using the training data set to obtain a target neural network; receiving a target document and generating a target training data set using the target document; training the target training data set using the target neural network to obtain a target vector for the target document; and calculating the difference between the target vector and a comparison vector in a preset database, and selecting documents corresponding to comparison vectors whose difference is less than a preset threshold as similar documents. This reduces the time cost of similar document retrieval and improves retrieval efficiency.

[0137] See also Figure 3 , Figure 3 This is a structural block diagram of a similar document retrieval device provided by an embodiment of the present invention.

[0138] An embodiment of the present invention provides a similar document retrieval device, comprising:

[0139] The acquisition module 301 is used to acquire a training document library; the training document library includes multiple documents, each of which has a corresponding document ID;

[0140] A training data set construction module 302 is used to construct a training data set based on the document;

[0141] A target neural network training module 303 is used to train a neural network using a training data set to obtain a target neural network;

[0142] A target training data set generating module 304 is configured to receive a target document and generate a target training data set using the target document;

[0143] A target vector acquisition module 305 is used to input the target training data set into the target neural network to obtain the target vector of the target document;

[0144] The similar document acquisition module 306 is configured to calculate the difference between the target vector and the comparison vectors in the preset database, and to select documents corresponding to the comparison vectors whose difference is less than a preset threshold as similar documents.

[0145] In this embodiment of the present invention, the training data set construction module 302 includes:

[0146] The word segmentation submodule is used to divide each document into multiple words; each word has a corresponding word ID;

[0147] The training data set construction submodule is used to construct a training data set using all documents and the word segmentation corresponding to each document.

[0148] In an embodiment of the present invention, the word segmentation submodule includes:

[0149] A merging unit, used to merge consecutive spaces in each document into a single space to obtain a preprocessed document;

[0150] The word segmentation unit is used to divide the preprocessed document into multiple words.

[0151] In an embodiment of the present invention, the training data set construction submodule includes:

[0152] A first word segmentation extraction unit is configured to extract a plurality of first word segmentations from the word segmentations of the document according to the arrangement order of the word segmentations in the document through a preset sliding window;

[0153] A training data generating unit is configured to generate training data by taking the segmentation ID of the last first segmentation in the sliding window as output and the segmentation IDs and corresponding document IDs of the first segmentations other than the last first segmentation in the sliding window as input;

[0154] a return unit, configured to move the sliding window according to the order of the word segments in the document, and return to the step of extracting multiple first word segments from the document according to the order of the word segments in the document through the preset sliding window until all the word segments in the document are traversed;

[0155] The training data set generating unit is used to generate a training data set using all the obtained training data.

[0156] In this embodiment of the present invention, the neural network includes an embedding layer, a splicing layer, a fully connected layer, and a soft layer; the target neural network training module 303 includes:

[0157] A word segmentation matrix splicing submodule is used to splice the word segmentation vectors corresponding to the word segmentation ID of the first word as input into a word segmentation matrix;

[0158] A first branch output acquisition submodule is used to multiply the word segmentation matrix with a preset first parameter matrix to obtain a first branch output of the embedding layer;

[0159] A second branch output acquisition submodule is configured to multiply the document vector corresponding to the document ID corresponding to the first word segmentation by a preset second parameter matrix to obtain a second branch output of the embedding layer;

[0160] A splicing output vector acquisition submodule is used to splice the first branch output and the second branch output through a splicing layer to obtain a spliced spliced output vector;

[0161] The fully connected output acquisition submodule is used to multiply the third parameter matrix of the fully connected layer with the concatenated output vector to obtain the fully connected layer output;

[0162] The target output vector acquisition submodule is used to perform Softmax transformation on the output of the fully connected layer through the Softmax layer to obtain the target output vector of the neural network;

[0163] A target vector construction submodule, configured to construct a target vector using the segmentation ID of the first segmentation as output;

[0164] The loss value calculation submodule is used to calculate the loss value between the target output vector and the target vector;

[0165] An updating submodule, configured to update the first parameter matrix, the second parameter matrix, and the third parameter matrix using the loss value;

[0166] The judgment submodule is used to judge whether the current number of training times is equal to the preset number threshold;

[0167] The first return submodule is used for returning the step of splicing the word segmentation vectors corresponding to the word segmentation ID of the first word as input into a word segmentation matrix if no;

[0168] The target neural network construction submodule is used to construct the target neural network based on the updated first parameter matrix, second parameter matrix, and third parameter matrix.

[0169] In this embodiment of the present invention, the target document has multiple target segmentations; the target training data set generation module 304 includes:

[0170] The second word segmentation extraction submodule is used to receive a target document and extract multiple second word segmentations from the target word segmentations of the target document according to the arrangement order of the target word segmentations in the target document through a sliding window;

[0171] A target training data acquisition submodule is used to generate target training data by taking the segmentation ID of the last second segmentation in the sliding window as output and the segmentation IDs of the second segmentations other than the last second segmentation in the sliding window as input;

[0172] a second return submodule, configured to move the sliding window according to the order of the target segmented words in the target document, and return through the sliding window to extract a plurality of second segmented words from the target segmented words in the target document according to the order of the target segmented words in the target document, until all target segmented words in the target document are traversed;

[0173] The target training data set acquisition submodule is used to generate a target training data set using all the obtained target training data.

[0174] An embodiment of the present invention further provides an electronic device, the device including a processor and a memory:

[0175] The memory is used to store program codes and transmit the program codes to the processor;

[0176] The processor is configured to execute the similar document retrieval method of the embodiment of the present invention according to the instructions in the program code.

[0177] An embodiment of the present invention further provides a computer-readable storage medium, which is used to store program code, and the program code is used to execute the similar document retrieval method of the embodiment of the present invention.

[0178] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0179] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0180] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, apparatus, or computer program products. Thus, embodiments of the present invention may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware. Furthermore, embodiments of the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0181] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the process in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0182] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0183] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0184] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic creative concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0185] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0186] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features thereof can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A similar document retrieval method, characterized in that: include: Obtain a training document library; the training document library includes multiple documents, each document having a corresponding document ID; Constructing a training dataset based on the document; Using the training data set to train a neural network to obtain a target neural network; receiving a target document, and generating a target training data set using the target document; Inputting the target training data set into the target neural network to obtain a target vector for the target document; Calculating the difference between the target vector and the comparison vectors in the preset database, and taking the documents corresponding to the comparison vectors whose difference is less than a preset threshold as similar documents, wherein the comparison vectors are collected into the database through the documents within the search range and generated correspondingly by the target neural network; The step of constructing a training data set based on the document includes: Divide each document into multiple word segments; each word segment has a corresponding word segment ID; Constructing a training data set using all the documents and the word segmentation corresponding to each document; The step of constructing a training data set using all the documents and the word segmentation corresponding to each document includes: Extracting a plurality of first segmented words from the segmented words in the document according to the arrangement order of the segmented words in the document through a preset sliding window; The training data is generated by taking the segmentation ID of the last first segmentation in the sliding window as output and the segmentation IDs and corresponding document IDs of the first segmentations other than the last first segmentation in the sliding window as input; Moving the sliding window according to the order of the segmented words in the document, and returning through the preset sliding window, extracting a plurality of first segmented words from the document according to the order of the segmented words in the document, until all the segmented words in the document are traversed; Generate a training data set using all the obtained training data; The neural network includes an embedding layer, a concatenation layer, a fully connected layer, and a Softmax layer; and the step of using the training data set to train the neural network to obtain a target neural network includes: The word segmentation vectors corresponding to the word segmentation ID of the first word as input are concatenated into a word segmentation matrix; Multiplying the word segmentation matrix by a preset first parameter matrix to obtain a first branch output of the embedding layer; Multiplying the document vector corresponding to the document ID corresponding to the first word segmentation by a preset second parameter matrix to obtain a second branch output of the embedding layer; splicing the first branch output and the second branch output through the splicing layer to obtain the spliced spliced output vector; Multiplying the third parameter matrix of the fully connected layer by the concatenated output vector to obtain a fully connected layer output; Performing a Softmax transformation on the output of the fully connected layer through a Softmax layer to obtain a target output vector of the neural network; Construct the target vector with the word ID of the first word as output; Calculating a loss value between the target output vector and the target vector; Updating the first parameter matrix, the second parameter matrix, and the third parameter matrix using the loss value; Determine whether the current number of training times is equal to the preset number threshold; If not, return to the step of concatenating the word vectors corresponding to the word ID of the first word as input into a word segmentation matrix; If so, a target neural network is constructed based on the updated first parameter matrix, second parameter matrix, and third parameter matrix.

2. The method according to claim 1, characterized in that The step of dividing each document into a plurality of word segments includes: Merging consecutive spaces in each of the documents into a single space to obtain a preprocessed document; The pre-processed document is divided into a plurality of tokens.

3. The method according to claim 1, characterized in that The target document has a plurality of target segmentations; the step of receiving the target document and generating a target training data set using the target document includes: receiving a target document, and extracting a plurality of second segmented words from the target segmented words in the target document through the sliding window according to the arrangement order of the target segmented words in the target document; The target training data is generated by taking the segmentation ID of the last second segmentation in the sliding window as output and the segmentation IDs of the second segmentations other than the last second segmentation in the sliding window as input; The sliding window is moved according to the order of the target segmented words in the target document, and the sliding window is returned to extract a plurality of second segmented words from the target segmented words in the target document according to the order of the target segmented words in the target document, until all the target segmented words in the target document are traversed; All the obtained target training data are used to generate a target training dataset.

4. A similar document retrieval device, characterized in that: include: An acquisition module is used to acquire a training document library; the training document library includes multiple documents, each document having a corresponding document ID; A training data set construction module, used to construct a training data set according to the document; A target neural network training module is used to train a neural network using the training data set to obtain a target neural network; A target training data set generation module is used to receive a target document and generate a target training data set using the target document; A target vector acquisition module, configured to input the target training data set into the target neural network to obtain a target vector for the target document; A similar document acquisition module is used to calculate the difference between the target vector and the comparison vectors in the preset database, and select the documents corresponding to the comparison vectors whose difference is less than a preset threshold as similar documents, wherein the comparison vectors are collected into the database through the documents within the scope of the search and generated by the target neural network; The training data set construction module includes: The word segmentation submodule is used to divide each document into multiple words; each word has a corresponding word ID; A training data set construction submodule is used to construct a training data set using all the documents and the word segmentation corresponding to each document; The training data set construction module includes: The word segmentation submodule is used to divide each document into multiple words; each word has a corresponding word ID; A training data set construction submodule is used to construct a training data set using all the documents and the word segmentation corresponding to each document; The training data set construction submodule includes: A first word segmentation extraction unit is configured to extract a plurality of first word segmentations from the word segmentations of the document according to an arrangement order of the word segmentations in the document through a preset sliding window; A training data generating unit is configured to generate training data by taking the segmentation ID of the last first segmentation in the sliding window as output and the segmentation IDs and corresponding document IDs of the first segmentations other than the last first segmentation in the sliding window as input; a returning unit, configured to move the sliding window according to the order of the segmented words in the document, and return to the step of extracting a plurality of first segmented words from the document according to the order of the segmented words in the document through the preset sliding window until all the segmented words in the document are traversed; A training data set generating unit, configured to generate a training data set using all the obtained training data; The neural network includes an embedding layer, a concatenation layer, a fully connected layer, and a Softmax layer; and the target neural network training module includes: A word segmentation matrix splicing submodule is used to splice the word segmentation vectors corresponding to the word segmentation ID of the first word as input into a word segmentation matrix; A first branch output acquisition submodule, configured to multiply the word segmentation matrix by a preset first parameter matrix to obtain a first branch output of the embedding layer; A second branch output acquisition submodule is configured to multiply the document vector corresponding to the document ID corresponding to the first word segmentation by a preset second parameter matrix to obtain a second branch output of the embedding layer; a splicing output vector acquisition submodule, configured to splice the first branch output and the second branch output through the splicing layer to obtain the spliced splicing output vector; A fully connected layer output acquisition submodule, configured to multiply the third parameter matrix of the fully connected layer by the concatenated output vector to obtain a fully connected layer output; A target output vector acquisition submodule is used to perform a Softmax transform on the output of the fully connected layer through a Softmax layer to obtain a target output vector of the neural network; A target vector construction submodule, configured to construct a target vector using the segmentation ID of the first segmentation as output; A loss value calculation submodule, used to calculate the loss value between the target output vector and the target vector; An updating submodule, configured to update the first parameter matrix, the second parameter matrix, and the third parameter matrix using the loss value; The judgment submodule is used to judge whether the current number of training times is equal to the preset number threshold; If not, return to the step of concatenating the word vectors corresponding to the word ID of the first word as input into a word segmentation matrix; The target neural network construction submodule is used to construct the target neural network based on the updated first parameter matrix, second parameter matrix, and third parameter matrix.

5. An electronic device, characterized in that: The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the similar document retrieval method according to any one of claims 1 to 3 according to the instructions in the program code.

6. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store program code, and the program code is used to execute the similar document retrieval method according to any one of claims 1 to 3.

Images