Text retrieval method and device, equipment and storage medium
Through the use of text structure identification model and keyword weight vectors, the accuracy and efficiency of patent plagiarism checking are improved, and the problems of low plagiarism checking efficiency and low accuracy in traditional methods are solved, reducing patent filing failure and resource waste.
Patent Information
- Application Number
- CN202411081326.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-12-06
AI Technical Summary
Traditional enterprises make their own judgments through staff before applying for patents, resulting in low severity check efficiency and low accuracy, resulting in repeated patent submission and waste of resources.
By obtaining the text to be checked, using the text structure recognition model to identify the structural framework, determining the weight values of word segmentation and feature word, selecting keywords, and using the target weight vector to search similar text in the text library to improve retrieval accuracy and efficiency.
It significantly improves the accuracy and efficiency of text retrieval, reduces the failure rate of patent application, and saves corporate resources.
Smart Images

Figure CN119089888A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of retrieval and duplicate checking, and in particular, relates to a text retrieval method, device, equipment and storage medium. Background Art
[0002] As companies pay more and more attention to intellectual property rights, in order to continuously improve the quality of corporate patents and allow patents to better play their own, business and asset values, more attention should be paid to the problem of duplicate checking of patent proposals within the company to avoid the problem of repeated submission of patents with the same technical solutions. In order to check for duplicate application of patents, traditional companies usually make simple judgments through staff in this field before applying for patents. As long as the staff in this field thinks that the technology is advanced and there is no prior art, they will write a patent for application, which will lead to the failure of the patent application. This not only wastes the company's property, but also fails to achieve the effect of patent protection, increases the workload of staff, and wastes manpower and material resources.
[0003] However, the solution determined by the staff on their own is labor-intensive, and the efficiency of duplicate checking and the accuracy of the query are extremely low. As a result, the accuracy of the patent duplicate checking results is low. Summary of the invention
[0004] The embodiments of the present application provide a text retrieval method, apparatus, device and storage medium, which improve the accuracy of text duplicate retrieval.
[0005] According to a first aspect of the present application, an embodiment of the present application provides a text retrieval method, the method comprising:
[0006] Get the text to be checked for duplicates;
[0007] Input the text to be checked for duplicates into a text structure recognition model to obtain multiple structural frameworks in the text to be checked for duplicates;
[0008] The following operations are performed for each structural frame: by performing a word segmentation operation on the text content corresponding to the structural frame, M feature words in the structural frame and a weight value corresponding to each feature word are determined, where M is a positive integer;
[0009] According to the weight value of each feature word, N keywords are selected from the text to be checked for duplicate content, where N is a positive integer and M≥N;
[0010] Using the target weight vectors corresponding to the N keywords, similar texts to the text to be checked for duplicates are retrieved in the text library.
[0011] Optionally, before inputting the to-be-checked duplicate text into the text structure recognition model to obtain a plurality of structure frames in the to-be-checked duplicate text, the method further comprises:
[0012] Obtain a training sample set, the training sample set includes multiple training samples, each training sample includes a one-to-one corresponding preset duplicate check text and a structure framework type label;
[0013] With the preset duplicate check text as input and the structural framework type label as output, the neural network model is trained until the training stop condition is met to obtain the text structure recognition model.
[0014] Optionally, the preset duplicate check text is used as input and the structure framework type label is used as output to train the neural network model until the training stop condition is met, thereby obtaining a text structure recognition model, including:
[0015] The following steps are performed for each training sample: inputting the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text;
[0016] Determine the loss function of the neural network model based on the prediction confidence and the structural framework type label;
[0017] When the loss function does not meet the training stop condition, the model parameters of the neural network model are adjusted to obtain an updated neural network model, and the preset duplicate check text is input into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text;
[0018] Until the training stop condition is met, the text structure recognition model is obtained.
[0019] Optionally, the neural network model includes a decoder and an encoder corresponding to each other;
[0020] Input the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text, including:
[0021] Use the encoder to extract features from the preset duplicate check text to obtain a text feature map;
[0022] Using a decoder to decode the text feature map, a structure frame type segmentation map is obtained, where the structure frame type segmentation map includes multiple actual structure frame types;
[0023] The prediction function is used to predict the confidence of each structural framework type, and the prediction confidence of each actual structural framework type in the preset duplicate check text is determined.
[0024] Optionally, using the target weight vectors corresponding to the N keywords, similar texts to the text to be checked for duplicates are retrieved from the text library, including:
[0025] For each keyword, the following operations are performed: vector characterization is performed on the weight value of the keyword to obtain the target weight vector of the keyword;
[0026] According to the target weight vector of each keyword, similarity search is performed on the text to be checked for duplicates to obtain similar texts.
[0027] Optionally, the weight values of the keywords are vectorized to obtain a target weight vector of the keywords, including:
[0028] The hash function is used to perform vector mapping on the keyword weights to obtain the keyword's target weight vector.
[0029] Optionally, similarity retrieval is performed on the text to be checked for duplicate content according to the target weight vector of each keyword, and the obtained similar texts include:
[0030] Determine the first text weight vector of the text to be checked for duplicates according to the sum of the target weight vectors of each keyword;
[0031] Obtaining second text weight vectors of a plurality of preset texts in a database;
[0032] Performing feature comparison on the first text weight vector and each second text weight vector respectively to determine the similarity between the first text weight vector and each second text weight vector;
[0033] The preset text corresponding to the second text weight vector whose similarity is greater than the preset similarity is determined as the target similar text.
[0034] Optionally, by performing a word segmentation operation on the text content corresponding to the structural frame, M feature words in the structural frame and a weight value corresponding to each feature word are determined, including:
[0035] Performing a word segmentation operation on the text content corresponding to the structural framework according to the preset word segmentation list to obtain feature words corresponding to the preset word segmentation list;
[0036] Get the word frequency and inverse document frequency of feature words;
[0037] The product of the word frequency and the inverse document frequency is determined as the weight value corresponding to the feature word.
[0038] According to a second aspect of the present application, an embodiment of the present application provides a text search device, the text search device comprising:
[0039] An acquisition module is used to obtain the text to be checked for duplicates;
[0040] A recognition module is used to recognize the text to be checked for duplicates using a text structure recognition model to obtain multiple structural frameworks in the text to be checked for duplicates;
[0041] The word segmentation module is used to perform the following operations for each structural frame: by performing word segmentation operations on the text content corresponding to the structural frame, M feature words in the structural frame and the weight value corresponding to each feature word are determined, where M is a positive integer;
[0042] The selection module is used to select N keywords from the text to be checked for duplicate content according to the weight value of each feature word, where N is a positive integer and M≥N;
[0043] The retrieval module is used to use the target weight vectors corresponding to the N keywords to retrieve similar texts to the text to be checked for duplicates in the text library.
[0044] According to a third aspect of the present application, there is provided a text retrieval device, the device comprising: a processor and a memory storing computer program instructions;
[0045] When the processor executes the computer program instructions, the text retrieval method according to any one of the first aspects is implemented.
[0046] According to the fourth aspect of the present application, an embodiment of the present application provides a computer storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the text retrieval method of any one of the first aspects is implemented.
[0047] According to the fifth aspect of the present application, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the text retrieval method of any one of the first aspects.
[0048] The text retrieval method, device, equipment and storage medium of the embodiment of the present application obtains the text to be checked for duplicates, inputs the text to be checked for duplicates into the text structure recognition model, and identifies the text to be checked for duplicates through the text structure recognition model, thereby obtaining the structural framework of the text to be checked for duplicates, thereby performing word segmentation operations on the text content under each structural framework, determining the feature words in the text content under each structural framework, and selecting the keywords that best reflect the text to be checked for duplicates from multiple feature words according to the weight value of each feature word, thereby searching the text library according to the target weight vector of the keyword to obtain similar texts similar to the text to be checked for duplicates. Based on this, in the retrieval process, by extracting the keywords in the text content corresponding to each structural framework in the text to be checked, similar texts are retrieved according to the target weight vector of the keyword, thereby improving the retrieval accuracy and retrieval efficiency of similar texts. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the technical solution of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0050] Figure 1 is a flow chart of a text retrieval method according to an exemplary embodiment;
[0051] Figure 2 is a structural block diagram of a text retrieval device according to an exemplary embodiment;
[0052] Figure 3 It is a structural block diagram of a text retrieval device according to an exemplary embodiment. DETAILED DESCRIPTION
[0053] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present application by illustrating the examples of the present application.
[0054] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-otherwise inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.
[0055] As mentioned in the background technology section, before traditional enterprises apply for patents, they usually make simple judgments through the staff in the field. As long as the staff in the field think that the technology is advanced and there is no prior art, they will write a patent for application, which will lead to failure after the patent application. This not only wastes the company's property, but also fails to achieve the effect of patent protection, increases the workload of the staff, and wastes manpower and material resources. However, the solution judged by the staff on their own has a large workload, and the efficiency of duplicate checking and the accuracy of query are extremely low. Therefore, the accuracy of the patent duplicate checking results is low.
[0056] In order to solve the problems of the prior art, the present application extracts keywords from the text content corresponding to each structural framework in the text to be searched during the retrieval process, and retrieves similar texts according to the target weight vector of the keywords, thereby improving the retrieval accuracy and efficiency of similar texts.
[0057] Based on this, the present application provides a text search method, device, equipment and storage medium. The text search method provided in the embodiment of the present application is first introduced below.
[0058] Figure 1 A flowchart of a text retrieval method provided by an embodiment of the present application is shown.
[0059] like Figure 1 As shown, it may include the following steps:
[0060] S101, obtaining the text to be checked for duplicates;
[0061] S102, inputting the text to be checked for duplicates into a text structure recognition model to obtain multiple structural frameworks in the text to be checked for duplicates;
[0062] S103, performing the following operations for each structural frame: performing a word segmentation operation on the text content corresponding to the structural frame to determine M feature words in the structural frame and a weight value corresponding to each feature word, where M is a positive integer;
[0063] S104, selecting N keywords from the text to be checked for duplicate content according to the weight value of each feature word, where N is a positive integer and M≥N;
[0064] S105, using the target weight vectors corresponding to the N keywords, retrieve similar texts of the text to be checked for duplicates in the text library.
[0065] Based on the above embodiments, the present application obtains the text to be checked for duplicates, inputs the text to be checked for duplicates into the structural recognition model of the present invention, and identifies the text to be checked for duplicates through the text structure recognition model, thereby obtaining the structural framework of the text to be checked for duplicates. Thus, the text content under each structural framework can be segmented, the characteristic words in the text content under each structural framework can be determined, and the keywords that best reflect the text to be checked for duplicates can be selected from multiple characteristic words according to the weight value of each characteristic word. Thus, the text library can be searched according to the target weight vector of the keyword to obtain similar texts similar to the text to be checked for duplicates. Based on this, in the retrieval process, by extracting the keywords in the text content corresponding to each structural framework in the text to be checked, similar texts are retrieved according to the target weight vector of the keyword, thereby improving the retrieval accuracy and retrieval efficiency of similar texts.
[0066] In the above S101, the text to be checked for plagiarism is obtained.
[0067] In the above S102, the text to be checked for duplicates is input into the text structure recognition model, and the text structure recognition model is used to perform structural framework recognition on the text to be checked for duplicates, so as to determine multiple structural frameworks in the text to be checked for duplicates.
[0068] As an example, the structural framework can be similar to the secondary title of an article, such as: the name of the invention, the technical field, the technical solution of the prior art, the shortcomings of the prior art and the technical problems to be solved by the proposal of this application, a detailed description of the technical solution of the proposal of this application, the key points of the proposal of this application and the points to be protected, etc.
[0069] In the above S103, the text to be checked for duplicates has corresponding text content under each structural framework, and the text content is composed of many words and sentences, and there are several groups of words in the multiple words and sentences that can reflect the core ideas and meanings of the entire text content. Therefore, the text content of each structural frame can be segmented to determine the M feature words of the text content in each structural frame, as well as the weight value of each feature word.
[0070] Specifically, in one embodiment, in order to improve the recognition accuracy of feature words and the accuracy of feature word weights, S103 may include:
[0071] S1031, performing a word segmentation operation on the text content corresponding to the structural framework according to a preset word segmentation list to obtain feature words corresponding to the preset word segmentation list;
[0072] S1032, obtaining the word frequency and inverse document frequency of the feature word;
[0073] S1033, determining the product of the word frequency and the inverse document frequency as the weight value corresponding to the feature word.
[0074] Based on the above embodiment, by performing word segmentation operations on the text content corresponding to the structural framework according to a preset word segmentation list, it is possible to determine the feature words according to an accurate word segmentation standard, thereby improving the accuracy of the feature words, and the word frequency and inverse document frequency can reflect the number of times the feature words appear in the text content, thereby determining the weight value of the feature words based on the word frequency and inverse document frequency, which can be more accurate.
[0075] In the above S1031, the preset word segmentation list is set in advance by the user, and usually contains the attributes of words and sentences that best reflect the core ideas and meanings of the text content. Therefore, the words and sentences in the text content are segmented according to the attributes of the words and sentences in the preset list, and M feature words corresponding to the attributes of the words and sentences in the preset list are determined.
[0076] As an example, the preset list may include: keywords, graph features, formulas, and may also include basic information such as proposal type, proposal name, name of the proposal inventor, and department of the proposal inventor.
[0077] In the above S1032, the number of occurrences of each feature word is counted, thereby calculating the word frequency of each feature word and the inverse text frequency.
[0078] Specifically, the term frequency (TF) is the number of times a feature word appears in the text content under the structure framework. This number is usually normalized (usually the term frequency divided by the total number of words in the text content) to prevent it from being biased towards long documents. (The same feature word may have a higher term frequency in a long text content than a short feature word, but it is not necessarily more important.)
[0079] Inverse Document Frequency (IDF) is a measure of the general importance of a feature word. The IDF of a feature word can be obtained by dividing the number of the feature word in the text to be checked for duplicates by the number of the feature word in the text content of the structural framework containing the word, and then taking the logarithm of the quotient. If the fewer structural frameworks containing the feature word, the larger the IDF, it means that the feature word has a good ability to distinguish categories.
[0080] In the above S1033, the product of the term frequency and the inverse document frequency is determined as the term frequency-inverse document frequency (TF-IDF), which is a weighted technology used in information retrieval and text mining. It evaluates the importance of a feature word to the entire text to be checked for duplicates by counting the frequency of a feature word appearing in the text content under the structural framework and the rarity of the feature word in the entire text to be checked for duplicates.
[0081] In the above S104, after the weight value of each feature word is determined, the larger the weight value is, the more important the feature word is to the text to be checked for duplicates. Therefore, the feature words ranked in the top N weight values from the M feature words can be selected as keywords.
[0082] In the above S105, the target weight vector can reflect the importance of the N keywords in the text to be checked for duplicates; and, during retrieval, we only need the target weight vectors corresponding to the N keywords, and compare them with the target weight vectors of the keywords of the preset text stored in the database. The similarity between the target weight vectors of the keywords can be used to determine the degree of text similarity, thereby greatly improving the retrieval efficiency. Therefore, searching the text library for similar texts based on the target weight vector can improve the retrieval accuracy and efficiency.
[0083] Specifically, in one embodiment, to determine each characteristic word, S105 may include:
[0084] S1051, performing the following operations for each keyword: performing vector characterization on the weight value of the keyword to obtain a target weight vector of the keyword;
[0085] S1052, performing similarity search on the text to be checked for duplicate content according to the target weight vector of each keyword to obtain similar texts.
[0086] In the above S1051, the weight value of each keyword is mapped by a hash function to generate a K-bit binary string, and this K-bit binary string is the target weight vector of the keyword.
[0087] In the above S1052, all key target weight vectors in the entire text to be checked for duplicates are accumulated to form a text weight vector of the text to be checked for duplicates, thereby comparing the text weight vector with the text weight vector of the preset text stored in the database to obtain the preset text whose text similarity is greater than the preset text similarity as the similar text.
[0088] Specifically, in one embodiment, in order to improve the retrieval accuracy, S1052 may include:
[0089] S10521, determining a first text weight vector of the to-be-checked duplicate text according to the sum of the target weight vectors of each keyword;
[0090] S10522, obtaining second text weight vectors of a plurality of preset texts in a database;
[0091] S10523, performing feature comparison on the first text weight vector and each second text weight vector respectively to determine the similarity between the first text weight vector and each second text weight vector;
[0092] S10524: Determine the preset text corresponding to the second text weight vector whose similarity is greater than the preset similarity as the target similar text.
[0093] Based on the above embodiment, by combining the target weight vector of each keyword and the first text weight vector of the text to be checked for duplicates, and comparing the similarity between the first text weight vector and the second text weight vector, a comparison between the first text and the second text is achieved, thereby improving the accuracy of the similarity between the text to be checked for duplicates and the preset text, and further improving the accuracy of similar texts retrieved based on the text to be checked for duplicates.
[0094] In the above S10521, the target weight vectors of the keywords in each structural framework in the text to be checked for duplicates are accumulated to obtain the first text weight vector of the entire text to be checked for duplicates.
[0095] In the above S10522, by pre-processing the preset texts in the database according to the determination method of the above-mentioned first text weight vector, the second text weight vector of each preset text is obtained, and the second text weight vector is stored in the database, thereby directly obtaining the second text weight vector of each preset text in the database.
[0096] In the above S10523, the first text weight vector and the second text weight vector can be calculated by using a cosine similarity calculation method or a graph convolutional neural network text similarity calculation method to determine the similarity between the text to be checked for duplicates and each preset text.
[0097] In the above S10524, the preset texts whose similarity between the text to be checked for duplicates and each preset text is greater than the preset similarity are determined as similar texts to the text to be checked for duplicates.
[0098] In order to obtain an accurate text structure recognition model, the present application also provides another implementation of the text retrieval method.
[0099] Specifically, before the above S102, the method may further include the following steps:
[0100] S201, obtaining a training sample set, the training sample set including a plurality of training samples, each training sample including a one-to-one corresponding preset duplicate check text and a structural framework type label;
[0101] S202, using the preset duplicate check text as input and the structure framework type label as output, training the neural network model until the training stop condition is met to obtain a text structure recognition model.
[0102] Based on the above embodiment, by obtaining a training sample set, the neural network model is trained using the good samples in the training sample set until the stopping condition is met, so that the text structure recognition model can accurately identify the framework structure type in the preset text to be checked for duplicates, thereby improving the accuracy of the text structure recognition model in recognizing the framework structure through training with a large number of training samples.
[0103] In the above S201, a preset duplicate checking text prepared in advance is obtained, and a corresponding structural framework type label is added to each structural framework type in the preset duplicate checking text.
[0104] In the above S202, the structural framework type label and the preset duplicate check text are input into the neural network model, and the neural network model is trained to identify the structural framework type label until the training stop condition is met, thereby obtaining a text structure recognition model.
[0105] Specifically, in one embodiment, S202 may include:
[0106] S2021, performing the following steps for each training sample: inputting the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text;
[0107] S2022, determining a loss function of the neural network model based on the prediction confidence and the structural framework type label;
[0108] S2023, when the loss function does not meet the training stop condition, adjust the model parameters of the neural network model to obtain an updated neural network model, and return to input the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text;
[0109] S2024, until the training stop condition is met, a text structure recognition model is obtained.
[0110] In the above S2021, the preset duplicate check text is input into the neural network model, and the neural network model outputs the prediction confidence of each structural framework type. These prediction confidences represent the prediction probability of the neural network model for each structural framework type.
[0111] In one embodiment, the neural network model may include a decoder and an encoder corresponding to each other, and S2021 may include:
[0112] S20211, using an encoder to extract features from a preset duplicate check text to obtain a text feature map;
[0113] S20212, using a decoder to perform feature decoding on the text feature map to obtain a structure frame type segmentation map, where the structure frame type segmentation map includes multiple actual structure frame types;
[0114] S20213, using the prediction function to predict the confidence of each structural framework type, and determine the prediction confidence of each actual structural framework type in the preset duplicate check text.
[0115] In the above S20211, the encoder performs preprocessing operations such as cleaning, word segmentation, and stop word removal on the preset duplicate check text. Then, a deep learning model (such as convolutional neural network CNN, recurrent neural network RNN, long short-term memory network LSTM or Transformer, etc.) is used to extract features of the preprocessed preset duplicate check text to obtain a text feature map.
[0116] As an example, the text feature map is a high-dimensional vector or matrix representation, i.e., a text feature map, which is obtained by converting a preset duplicate check text into a high-dimensional vector or matrix representation by an encoder, and contains important information of the text.
[0117] In the above S20212, the decoder network is used to decode the text feature map output by the encoder and convert it back into a structured representation. The output of the decoder is a structure frame type segmentation map, in which each area represents a part of the text and is marked with the corresponding actual structure frame type (such as introduction, body, conclusion, etc.).
[0118] The decoder can be a convolutional neural network (CNN) or a recurrent neural network (RNN / LSTM), etc., whose input is the text feature map and output is the structure frame type segmentation map. The segmentation map can be implemented by a classification layer (such as a softmax layer), which assigns a structure frame type label to each region.
[0119] Each encoder corresponds to a decoder. The encoder part consists of several convolutional layers, BNc layers, RELU and pooling layers. The pooling index of the maximum pooling layer is used for upsampling. Finally, the output feature map is input into the softmax layer to classify each pixel and obtain the probability that each pixel belongs to K categories. The output of the m-layer pooling layer in the encoding part is extracted as features of different scales, and the features of various scales are respectively input into the text structure recognition model. The text structure recognition model outputs the confidence of each part of the structure after calculation, and the semantic segmentation branch outputs the segmentation map after calculation; the segmentation model in the text structure recognition model is used to segment the various parts of the technical disclosure book to obtain the content of the invention name, the content of the technical field, the content of the problem solved by the invention, the content of the technical solution, and the content of the key point protection.
[0120] When pooling, maximum pooling is used to distribute the input matrix into a series of non-overlapping rectangles, and then each rectangle only outputs its maximum value. This can make the network representation invariant to small translations or rotations, that is, when the input produces a small translation or rotation, the pooling result remains unchanged; by replacing k pixels in the pooling matrix with a statistical value, the final matrix size is reduced, which can reduce the amount of calculation and the risk of overfitting, and improve the network calculation efficiency.
[0121] During training, the corresponding text structure content labels can be pre-set in the training sample data. Use the labeled technical briefing to train the text structure recognition model. First, the encoder is used to extract image features, and then the obtained feature map is sent to the decoder for decoding. After the d_conv and upsampling modules, the text content corresponding to each segmented structure is finally obtained; the d_conv operation process of the decoder part includes a convolution with a mask size of n*n, normalization batchnormalization and activation function relu, and the upsampling module includes a convolution with a mask size of n*n, normalization batchnormalization, activation function relu and UpSample upsampling operation.
[0122] In the above S20213, for each structural frame type output by the decoder, a prediction function is used to calculate its prediction confidence. According to the output of the prediction function, the prediction confidence of each preset structural frame type in the preset duplicate check text is determined.
[0123] The prediction function can be a simple classifier (such as logistic regression, support vector machine SVM or a branch of a neural network), whose input is the feature vector of a certain region output by the decoder, and the output is the prediction confidence of the structural framework type corresponding to the region. The confidence is usually a value between 0 and 1, indicating the degree of certainty of the model for the prediction.
[0124] In the above S2022, the loss function of the neural network model is calculated according to the prediction confidence and the true value (structural framework type label). The loss function measures the difference between the predicted value of the neural network model and the true value.
[0125] In the above S2023, an optimization algorithm (such as gradient descent, Adam, etc.) is used to adjust the parameters of the neural network model according to the gradient of the loss function. An updated neural network model is obtained. Step S2021 is repeated again, and the preset duplicate check text is input into the updated neural network model to obtain a new prediction confidence again.
[0126] As an example, the training stop condition may include any one of the following conditions: reaching a preset number of iterations, a loss value being less than a certain threshold, or the performance of the validation set no longer improving.
[0127] In the above S2034, after the training stop condition is met, the new neural network model can be determined as a text structure recognition model.
[0128] It should be noted that the application scenarios described in the above-mentioned embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Ordinary technicians in this field can know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems; and new embodiments of the present application can be combined with each other, and new solutions formed by the combination of various embodiments are all within the protection scope of the present application.
[0129] Based on the same inventive concept, the present application also provides a text retrieval device 200. Figure 2 Provide detailed explanation.
[0130] An acquisition module 210 is used to acquire the text to be checked for duplicates;
[0131] The recognition module 220 is used to recognize the text to be checked for duplicates by using a text structure recognition model to obtain multiple structural frameworks in the text to be checked for duplicates;
[0132] The word segmentation module 230 is used to perform the following operations for each structural frame: by performing a word segmentation operation on the text content corresponding to the structural frame, determine M feature words in the structural frame and a weight value corresponding to each feature word, where M is a positive integer;
[0133] The selection module 240 is used to select N keywords from the text to be checked for duplicate content according to the weight value of each feature word, where N is a positive integer and M≥N;
[0134] The search module 250 is used to search the text library for similar texts to the text to be checked for duplicates using the target weight vectors corresponding to the N keywords.
[0135] In the text retrieval device 200 provided in the present embodiment, the acquisition module 210 acquires the duplicate text to be checked, and the identification module 220 inputs the duplicate text to be checked into the text structure recognition model, and identifies the duplicate text to be checked through the text structure recognition model, thereby obtaining the structural framework of the duplicate text to be checked, thereby, the word segmentation module 230 can perform word segmentation operations on the text content under each structural framework, determine the feature words in the text content under each structural framework, and the selection module 240 selects the keywords that best reflect the duplicate text to be checked from multiple feature words according to the weight value of each feature word, and the retrieval module 250 can search in the text library according to the target weight vector of the keyword to obtain similar texts similar to the duplicate text to be checked. Based on this, in the retrieval process, by extracting the keywords in the text content corresponding to each structural framework in the text to be checked, and retrieving and obtaining similar texts according to the target weight vector of the keyword, the retrieval accuracy and retrieval efficiency of similar texts are improved.
[0136] Optionally, the text retrieval device 200 may further include:
[0137] A first acquisition module is used to acquire a training sample set, the training sample set includes a plurality of training samples, each training sample includes a one-to-one corresponding preset duplicate check text and a structural framework type label;
[0138] The training module is used to train the neural network model with the preset duplicate check text as input and the structural framework type label as output until the training stop condition is met to obtain the text structure recognition model.
[0139] Optionally, the training module may include:
[0140] The input unit is used to perform the following steps for each training sample: input the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text;
[0141] A first determination unit is used to determine a loss function of a neural network model according to a prediction confidence and a structural framework type label;
[0142] An adjustment unit, used for adjusting the model parameters of the neural network model when the loss function does not meet the training stop condition, obtaining an updated neural network model, and returning to input the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text;
[0143] The unit is obtained until the training stop condition is met to obtain a text structure recognition model.
[0144] Optionally, the neural network model may include a decoder and an encoder corresponding to each other;
[0145] Input units may include:
[0146] A feature extraction subunit is used to extract features from a preset duplicate check text using an encoder to obtain a text feature map;
[0147] A feature encoding subunit is used to decode the text feature map using a decoder to obtain a structure frame type segmentation map, where the structure frame type segmentation map includes multiple actual structure frame types;
[0148] The prediction function is used to predict the confidence of each structural framework type, and the prediction confidence of each actual structural framework type in the preset duplicate check text is determined.
[0149] Optionally, the retrieval module 250 may include:
[0150] The vector characterization unit is used to perform the following operations for each keyword: vector characterization of the weight value of the keyword to obtain a target weight vector of the keyword;
[0151] The retrieval unit is used to perform similarity retrieval on the text to be checked for duplicate content according to the target weight vector of each keyword to obtain similar texts.
[0152] Optionally, the vector characterization unit may include:
[0153] The vector mapping subunit is used to perform vector mapping on the weight of the keyword by using a hash function to obtain a target weight vector of the keyword.
[0154] Optionally, the word segmentation module 230 may include:
[0155] The molecular unit is used to perform a segmentation operation on the text content corresponding to the structural frame according to a preset segmentation list to obtain feature words corresponding to the preset segmentation list;
[0156] An acquisition unit, used to acquire the word frequency and inverse document frequency of the feature word;
[0157] The second determining unit is used to determine the product of the word frequency and the inverse document frequency as the weight value corresponding to the feature word.
[0158] Optionally, the retrieval unit may include:
[0159] A first determination subunit is used to determine a first text weight vector of the to-be-checked duplicate text according to the sum of the target weight vectors of each keyword;
[0160] An acquisition subunit, used for acquiring second text weight vectors of a plurality of preset texts in a database;
[0161] A second determination subunit is used to perform feature comparison on the first text weight vector and each second text weight vector respectively to determine the similarity between the first text weight vector and each second text weight vector;
[0162] The third determining subunit is used to determine the preset text corresponding to the second text weight vector whose similarity is greater than the preset similarity as the target similar text.
[0163] The text search device 200 provided in the embodiment of the present application can realize Figure 1 The various processes implemented by the method embodiment can achieve the same technical effect, and to avoid repetition, they will not be described here.
[0164] Figure 3 A schematic diagram of the hardware structure of a text retrieval device provided by an embodiment of the present invention is shown.
[0165] The text retrieval device may include a processor 301 and a memory 302 storing computer program instructions.
[0166] Specifically, the processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiment of the present invention.
[0167] The memory 302 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive or a combination of two or more of these. In appropriate cases, the memory 302 may include a removable or non-removable (or fixed) medium. In appropriate cases, the memory 302 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 302 is a non-volatile solid-state memory.
[0168] In certain embodiments, the memory 302 may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical or other physical / tangible memory storage device. Thus, typically, the memory 302 includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors 301), it is operable to perform the operations described with reference to the method according to an aspect of the present application.
[0169] The processor 301 implements any one of the text retrieval methods in the above embodiments by reading and executing the computer program instructions stored in the memory 302 .
[0170] In one example, the text retrieval device may further include a communication interface 303 and a bus 304. As shown in the figure, the processor 301, the memory 302, and the communication interface 303 are connected via the bus 304 and communicate with each other.
[0171] The communication interface 303 is mainly used to implement the communication between the modules, devices, units and / or equipment in the embodiment of the present invention.
[0172] Bus 304 includes hardware, software or both, and will be coupled to each other in the parts of text retrieval equipment.For example, but not limitation, bus 304 may include accelerated graphics port (AGP) or other graphics bus, enhanced industrial standard architecture (EISA) bus, front side bus (FSB), hypertransport (HT) interconnection, industrial standard architecture (ISA) bus, wireless bandwidth interconnection, low pin count (LPC) bus, memory bus, micro channel architecture (MCA) bus, peripheral control interconnection (PCI) bus, PCI-Express (PCI-X) bus, serial advanced technology attachment (SATA) bus, video electronics standard association local (VLB) bus or other suitable bus or two or more of these combinations. In suitable cases, bus 304 may include one or more buses 304. Although the present application embodiment describes and shows a specific bus 304, the application considers any suitable bus 304 or interconnection.
[0173] The text retrieval device can execute the text retrieval method in the embodiment of the present application based on the text retrieval device 200, thereby realizing the combination of Figure 1 Describes the text retrieval method.
[0174] In addition, in combination with the text retrieval method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the text retrieval methods in the above embodiment is implemented.
[0175] An embodiment of the present application also provides a computer program product, including a computer program, which implements any one of the text retrieval methods in the above embodiments when the computer program is processed and executed.
[0176] It should be clear that the present application is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present application.
[0177] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.
[0178] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps, that is, the steps can be performed in the order mentioned in the embodiment, or in a different order from the embodiment, or several steps can be performed simultaneously.
[0179] Aspects of the present disclosure are described above with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable text retrieval device to produce a machine so that these instructions executed by the processor of the computer or other programmable text retrieval device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It can also be understood that each box in the block diagram and / or flowchart and the combination of the boxes in the block diagram and / or flowchart can also be implemented by dedicated hardware that performs a specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions.
[0180] The above is only a specific implementation of the present application. Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the protection scope of the present application is not limited to this. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the protection scope of this application.
Claims
1. A text retrieval method, It is characterized in that The method comprises: Get the text to be checked for duplicates; Inputting the text to be checked for duplicates into a text structure recognition model to obtain a plurality of structural frameworks in the text to be checked for duplicates; The following operations are performed for each of the structural frames: by performing a word segmentation operation on the text content corresponding to the structural frame, M feature words in the structural frame and a weight value corresponding to each of the feature words are determined, where M is a positive integer; According to the weight value of each of the feature words, N keywords are selected from the text to be checked for duplicates, where N is a positive integer and M≥N; Using the target weight vectors corresponding to the N keywords, similar texts to the text to be checked for duplicates are retrieved in a text library.
2. The method according to claim 1, It is characterized in that Before inputting the to-be-checked duplicate text into a text structure recognition model to obtain a plurality of structure frames in the to-be-checked duplicate text, the method further comprises: Obtain a training sample set, wherein the training sample set includes a plurality of training samples, each training sample includes a one-to-one corresponding preset duplicate check text and a structural framework type label; The preset duplicate check text is used as input and the structural framework type label is used as output to train the neural network model until the training stop condition is met, thereby obtaining the text structure recognition model.
3. The method according to claim 2, It is characterized in that The method uses the preset duplicate check text as input and the structural framework type label as output to train the neural network model until the training stop condition is met to obtain the text structure recognition model, including: The following steps are respectively performed for each of the training samples: the preset duplicate checking text is input into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate checking text; Determining a loss function of the neural network model according to the prediction confidence and the structural framework type label; In the case where the loss function does not satisfy the training stop condition, adjusting the model parameters of the neural network model to obtain an updated neural network model, and returning to input the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text; Until the training stop condition is met, the text structure recognition model is obtained.
4. The method according to claim 3, It is characterized in that The neural network model includes a decoder and an encoder corresponding to each other; The step of inputting the preset duplicate check text into the neural network model to obtain the prediction confidence of each actual structural framework type included in the preset duplicate check text includes: Using the encoder to extract features from a preset duplicate check text to obtain a text feature graph; Using the decoder to perform feature decoding on the text feature map to obtain a structure frame type segmentation map, wherein the structure frame type segmentation map includes a plurality of actual structure frame types; A prediction function is used to predict the confidence of each of the structural framework types, and the prediction confidence of each of the actual structural framework types in the preset duplicate check text is determined.
5. The method according to claim 1, It is characterized in that The method of using the target weight vectors corresponding to the N keywords to retrieve similar texts of the text to be checked for duplicate content in a text library includes: For each of the keywords, the following operations are performed respectively: vector characterization is performed on the weight value of the keyword to obtain the target weight vector of the keyword; A similarity search is performed on the text to be checked for duplicate content according to the target weight vector of each keyword to obtain similar texts.
6. The method according to claim 5, It is characterized in that The step of performing vector characterization on the weight value of the keyword to obtain the target weight vector of the keyword includes: A hash function is used to perform vector mapping on the weight of the keyword to obtain the target weight vector of the keyword.
7. The method according to claim 5, It is characterized in that The similarity search of the to-be-checked duplicate texts according to the target weight vector of each keyword to obtain similar texts includes: Determining a first text weight vector of the to-be-checked duplicate text according to the sum of the target weight vectors of each of the keywords; Obtaining second text weight vectors of a plurality of preset texts in a database; Performing feature comparison on the first text weight vector and each of the second text weight vectors respectively to determine the similarity between the first text weight vector and each of the second text weight vectors; The preset text corresponding to the second text weight vector whose similarity is greater than the preset similarity is determined as the target similar text.
8. The method according to any one of claims 1 to 7, It is characterized in that The method of performing a word segmentation operation on the text content corresponding to the structural frame to determine M feature words in the structural frame and a weight value corresponding to each feature word includes: Performing a word segmentation operation on the text content corresponding to the structural framework according to a preset word segmentation list to obtain the feature words corresponding to the preset word segmentation list; Obtaining the word frequency and inverse document frequency of the feature word; The product of the word frequency and the inverse document frequency is determined as the weight value corresponding to the feature word.
9. A text retrieval device, It is characterized in that The device also includes: An acquisition module is used to obtain the text to be checked for duplicates; A recognition module, used for recognizing the text to be checked for duplicates by using a text structure recognition model, and obtaining a plurality of structural frameworks in the text to be checked for duplicates; The word segmentation module is used to perform the following operations for each of the structural frames: by performing a word segmentation operation on the text content corresponding to the structural frame, determine M feature words in the structural frame and a weight value corresponding to each of the feature words, where M is a positive integer; A selection module is used to select N keywords from the text to be checked for duplicate content according to the weight value of each of the feature words, where N is a positive integer and M≥N; The retrieval module is used to use the target weight vectors corresponding to the N keywords to retrieve similar texts of the text to be checked for duplicates in the text library.
10. A text retrieval device, It is characterized in that The device comprises: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the method according to any one of claims 1 to 8 is implemented.
11. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.
12. A computer program product, It is characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the method according to any one of claims 1 to 8.
