Document search method and device

Abstract

The present application provides a document search method and device, wherein the document search method comprises: firstly, by extracting the target text content of the document to be searched and the first semantic vector of the target text content, searching the first candidate document and searching the second candidate document based on the target text content and the first semantic vector, the search can be combined with the text level and the semantic level, so that the candidate documents obtained by the search are more comprehensive. Then, because the semantic level and the text level are combined, the document to be searched, the first candidate document and the second candidate document are analyzed for similarity, and the target similar document of the document to be searched is determined, thereby improving the accuracy of the determined target similar document.

Description

Technical Field

[0001] The present application relates to the field of document processing technology, and more particularly to a document search method. The present application also relates to a document search device, a computing device, and a computer-readable storage medium. Background Art

[0002] With the development of Internet technology, a large number of documents are needed to record project content in various project scenarios. Existing documents with similar content have an impact on the project. For example, in the project scenario of academic document publication, if there are already academic documents with similar content, it will cause trouble for the academic document publication. Therefore, it is necessary to search and obtain similar documents from a large number of existing documents.

[0003] In the prior art, a text-level search is performed, that is, a search is performed based on the text content of the document to be searched, multiple candidate documents are obtained through the search, and then a target similar document of the document to be searched is determined based on the similarity of the text content between the document to be searched and the multiple candidate documents.

[0004] However, such a text-level search based only on the text content to be searched cannot be combined with the semantic features of the document to be searched, resulting in insufficient comprehensiveness of the candidate documents obtained from the search. In addition, the target similar documents are determined only based on the similarity of the text content, resulting in insufficient accuracy in the determined target similar documents. Summary of the invention

[0005] In view of this, the embodiment of the present application provides a document search method to solve the technical defects existing in the prior art. The embodiment of the present application also provides a document search device, a computing device, and a computer-readable storage medium.

[0006] According to a first aspect of an embodiment of the present application, a document search method is provided, comprising:

[0007] Acquire a document to be searched, extract target text content of the document to be searched and a first semantic vector of the target text content;

[0008] Based on the target text content, searching for a first candidate document, and based on the first semantic vector, searching for a second candidate document;

[0009] A similarity analysis is performed on the document to be searched, the first candidate document and the second candidate document to determine a target similar document of the document to be searched.

[0010] According to a second aspect of an embodiment of the present application, there is provided a document search device, comprising:

[0011] An extraction module is configured to obtain a document to be searched, extract a target text content of the document to be searched and a first semantic vector of the target text content;

[0012] A search module configured to search for a first candidate document based on the target text content, and to search for a second candidate document based on the first semantic vector;

[0013] The determination module is configured to perform similarity analysis on the document to be searched, the first candidate document and the second candidate document to determine a target similar document of the document to be searched.

[0014] According to a third aspect of an embodiment of the present application, a computing device is provided, including:

[0015] Memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor implements the steps of the document search method when executing the computer-executable instructions.

[0017] According to a fourth aspect of an embodiment of the present application, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of the document search method are implemented.

[0018] According to a fifth aspect of an embodiment of the present application, a chip is provided, which stores a computer program, and the computer program implements the steps of the document search method when executed by the chip.

[0019] The document search method provided by the present application first extracts the target text content of the document to be searched and the first semantic vector of the target text content, and searches the first candidate document and the second candidate document based on the target text content and the first semantic vector, and can combine the text level and the semantic level to search, so that the candidate documents obtained by the search are more comprehensive. Then, because the semantic level and the text level are combined, the similarity analysis is performed on the document to be searched, the first candidate document and the second candidate document, and the target similar document of the document to be searched is determined, thereby improving the accuracy of the determined target similar document. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a structural diagram of a document search system provided by an embodiment of the present application;

[0021] Figure 2 is a flowchart of a document search method provided by an embodiment of the present application;

[0022] Figure 3 This is a flow chart of a first semantic vector extraction method in a document search method provided in one embodiment of the present application;

[0023] Figure 4 is a flow chart of a first candidate document search method in a document search method provided in an embodiment of the present application;

[0024] Figure 5 is a flow chart of a first candidate document search method in another document search method provided in an embodiment of the present application;

[0025] Figure 6 is a flow chart of a first candidate document search method in another document search method provided in an embodiment of the present application;

[0026] Figure 7 is a flow chart of a second candidate document search method in a document search method provided in an embodiment of the present application;

[0027] Fig. 8A It is a flow chart of a method for determining a target similar document in a document search method provided in an embodiment of the present application;

[0028] Figure 8B It is a schematic diagram of the processing flow of a double-tower model in a document search method provided in an embodiment of the present application;

[0029] Figure 8C It is a schematic diagram of the processing flow of an interactive model in a document search method provided in an embodiment of the present application;

[0030] Fig. 9 It is a flow chart of a method for determining target similar documents in another document search method provided in an embodiment of the present application;

[0031] Fig.10 It is a flow chart of a method for determining target similar documents in another document search method provided in an embodiment of the present application;

[0032] Fig.11 It is a flow chart of a method for determining target similar documents in another document search method provided in an embodiment of the present application;

[0033] Fig.12 It is a flow chart of a method for determining target similar documents in another document search method provided in an embodiment of the present application;

[0034] Fig.13 This is a flow chart of a method for determining target similar documents in another document search method provided in an embodiment of the present application.

[0035] Fig.14 It is a flow chart of a method for determining target similar documents in another document search method provided in an embodiment of the present application;

[0036] Fig.15 It is a processing flow chart of a document search method applied to search based on the file name and original text content of the document to be searched, provided in one embodiment of the present application;

[0037] Fig.16 It is a system flow diagram of a document search method provided by an embodiment of the present application;

[0038] Fig.17 It is a structural schematic diagram of a document search device provided by an embodiment of the present application;

[0039] Fig.18 It is a structural block diagram of a computing device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0040] Many specific details are described in the following description to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present application, so the present application is not limited by the specific implementation disclosed below.

[0041] The terms used in one or more embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present application. The singular forms of "a", "said" and "the" used in one or more embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present application refers to and includes any or all possible combinations of one or more associated listed items.

[0042] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present application, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0043] First, the terms involved in one or more embodiments of the present invention are explained.

[0044] Semantic vector: semantic information extracted by the model.

[0045] ES (Elasticsearch): is a search server based on Lucene.

[0046] Lucene: It is an open source full-text search engine toolkit and a full-text search engine architecture that provides a complete query engine and index engine, as well as a partial text analysis engine.

[0047] Faiss: An open source vector recall engine for finding the N most similar vectors to a given vector.

[0048] BERT model (Bidirectional Encoder Representation from Transformers): is a pre-trained language representation model.

[0049] Original text content: original text information.

[0050] TextRank: It is a common keyword extraction algorithm in the field of natural language processing. It can be used to extract keywords, phrases and automatically generate text summaries.

[0051] IOU (Intersection over Union) calculation: the value of the intersection of two sets over the union of the two sets.

[0052] In the present application, a document search method is provided. The present application also relates to a document search device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0053] Figure 1 The schematic diagram of the structure of a document search system provided by an embodiment of the present application is shown. The execution subject of the document search method provided by the embodiment of the present application is a server 101 with the function of extracting text content and semantic vectors; the server 101 includes: a pre-processing layer 1011, a search layer 1012 and a post-processing layer 1013. The pre-processing layer 1011 has the function of obtaining the document to be searched and extracting the target text content and the first semantic vector, the search layer 1012 has the function of searching candidate documents, and the post-processing layer 1013 has the function of performing similarity analysis on the document to be searched and the candidate documents and determining the target similar document. The execution subject for storing sample documents in the document search method provided by the embodiment of the present application is the storage terminal 102, and the storage terminal 102 can be a storage terminal with a document storage function. The above-mentioned server 101 can be any electronic product that can perform human-computer interaction with users, such as PC (Personal Computer), mobile phone, Pocket PC, tablet computer, etc.

[0054] The server 101 and storage 102 in the document search system can be integrated into one device or can be independent devices. In the case of independent devices, the server 101 and storage 102 can be a server, a server cluster composed of multiple servers, or a cloud computing service center, which is not limited in the present embodiment.

[0055] The pre-processing layer 1011 in the server 101 with document search function obtains the document to be searched, extracts the target text content of the document to be searched and the first semantic vector of the target text content, the search layer 1012 in the server 101 searches for the first candidate document based on the target text content, and searches for the second candidate document based on the first semantic vector, and the post-processing layer 1013 in the server 101 performs similarity analysis on the document to be searched, the first candidate document and the second candidate document to determine the target similar document of the document to be searched. The storage end 102 can receive the document search request of the server 101 and return the stored candidate documents to the server 101.

[0056] The specific document search method will be described in detail in the subsequent embodiments.

[0057] Figure 2 A flowchart of a document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0058] Step S202: obtaining a document to be searched, extracting target text content of the document to be searched and a first semantic vector of the target text content.

[0059] The core of the embodiment of the present application is to combine the text features of the document to be searched and the semantic features of the target text content to perform the search. For documents to be searched in different formats, the document search process is basically the same. The document search process is introduced in detail below.

[0060] The document to be searched is a reference document for which a similar document search is required. The document to be searched may be a scientific research project document, an engineering project document, a publishing project document or other text content document, or a computer code document, or a web page, a picture or other document. The embodiment of the present application is described using a text content document as an example. The document to be searched may be a document in a variety of formats, for example, a document in txt format, a document in xml format, a document in html format, a document in doc mode, a document in pdf format, a document in caj format, a document in vbs format, etc., which are not limited here.

[0061] The target text content is the content of the text features of the document to be searched, for example, the author name of the document to be searched, the file name of the document to be searched, the directory of the document to be searched, the original text content of the document to be searched, the summary content of the document to be searched, and the keywords of the document to be searched, etc., which are not limited here.

[0062] The first semantic vector of the target text content is a vector of the semantic features of the target text content of the document to be searched, which is a semantic feature vector obtained by vectorizing the target text content based on the semantic feature level of the target text content. Semantic features include: word features, part-of-speech features, character features, position features, etc. For example, the word feature of "weather" in the text content "The weather is good today" is token_1 (E_token_1 determined according to a pre-set feature rule), the part-of-speech feature is a noun (E_cls_n), the character feature is 2 characters (E_num_2), and the position feature is the second word in the text content (E_pos-2). After vectorizing "weather", the semantic feature vector obtained is {E_token_1, E_cls_n, E_num_2, E_pos-2}.

[0063] The method of obtaining the document to be searched can be to directly call from the local storage end through the document call port, or to send a call request to the remote storage end through the document call interface and obtain the document to be searched returned by the remote storage end. Generally, according to the project scenario requirements, the document to be searched will be encrypted. The encryption methods include: AES encryption (Advanced Encryption Standard), RSA encryption (RSA algorithm, asymmetric encryption algorithm), md5 encryption algorithm, etc. After obtaining the document to be searched, the document to be searched is decrypted according to the corresponding encryption algorithm.

[0064] The method of extracting the target text content of the document to be searched can be to use a pre-trained target text content extraction model for extraction, or to extract by manual selection, or to extract according to a pre-established text content query table, or to identify the text content in the image document through text recognition technology (OCR, optical character recognition), which is not limited here.

[0065] The method for extracting the first semantic vector of the target text content can be to use a pre-trained natural language model to encode the semantic features of the target text content to obtain the first semantic vector; to manually encode the semantic features of the target text content to obtain the first semantic vector; or to obtain the first semantic vector by querying a pre-established semantic feature query table, without limitation here.

[0066] Specifically, a document to be searched is obtained, target text content of the document to be searched is extracted, and a first semantic vector of the target text content is obtained according to semantic features of the target text content.

[0067] For example, obtain the document to be searched "Mathematical Principles of Natural Philosophy", extract the target text content "On the Movement of Objects", and according to the semantic features of the target text content {On - E_token_1, Objects - E_token_2, Of - E_token_3, Movement - E_token_4}, obtain the first semantic vector of the target text content as {E_token_1, E_token_2, E_token_3, E_token_4}.

[0068] By acquiring the document to be searched, and extracting the target text content of the document to be searched and the first semantic vector of the target text content, an indexing foundation is laid for the subsequent search of the first candidate document and the second candidate document.

[0069] Step S204: searching for a first candidate document based on the target text content, and searching for a second candidate document based on the first semantic vector.

[0070] The first candidate document and the second candidate document are multiple sample documents obtained through search and stored in the storage end 102. The first candidate document is one or more sample documents corresponding to the target text content obtained by searching based on the target text content of the document to be searched, and the second candidate document is one or more sample documents corresponding to the first semantic vector obtained by searching based on the first semantic vector of the target text content. The first candidate document and the second candidate document may contain the same document.

[0071] The method of searching for the first candidate document is to send a first document search request to the storage end 102, wherein the first document search request includes the target text content of the document to be searched. After receiving the first document search request, the storage end 102 queries multiple stored sample documents and selects one or more sample documents corresponding to the target text content as the first candidate document.

[0072] The method for searching for the second candidate document is to send a second document search request to the storage end 102, wherein the second document search request includes a first semantic vector of the target text content. After receiving the second document search request, the storage end 102 queries multiple stored sample documents and selects one or more sample documents corresponding to the first semantic vector as the second candidate document.

[0073] Specifically, based on the target text content, one or more sample documents corresponding to the target text content in multiple sample documents are determined as first candidate documents, and based on the first semantic vector, one or more sample documents corresponding to the first semantic vector in multiple sample documents are determined as second candidate documents.

[0074] Continuing with the above example, based on the target text content “On the Movement of Objects”, the sample documents A1, A14, A27 corresponding to the target text content “On the Movement of Objects” among multiple sample documents A1, A2, A3…A30 are determined as the first candidate documents (A1, A14, A27), and based on the first semantic vector being {E_token_1, E_token_2, E_token_3, E_token_4}, the sample documents A4, A19, A30 corresponding to the first semantic vector among multiple sample documents A1, A2, A3…A30 are determined as the second candidate documents (A4, A19, A30).

[0075] Based on the target text content and the first semantic vector, the first candidate document and the second candidate document are searched. The search may be performed at the text level and the semantic level, so that the candidate documents obtained by the search are more comprehensive.

[0076] Step S206: performing similarity analysis on the document to be searched, the first candidate document and the second candidate document to determine a target similar document to the document to be searched.

[0077] The target similar document is one or more documents among the first candidate document and the second candidate document that have the highest similarity with the document to be searched at the text level and the semantic level.

[0078] The similarity analysis method can be to perform a similarity analysis on the search document and the first candidate document, and to perform a similarity analysis on the search document and the second candidate document, and determine the target similar document based on the results of the two analyses. The first candidate document and the second candidate document can also be combined into a document set, and a similarity analysis is performed on the search document and each document in the document set to determine the target similar document. There is no limitation here.

[0079] Specifically, similarity analysis is performed on the document to be searched, the first candidate document and the second candidate document, and one or more documents in the first candidate document and the second candidate document that have the highest similarity with the document to be searched at the text level and the semantic level are determined as target similar documents.

[0080] For example, if the document to be searched is "the mathematical principles of natural philosophy", the first candidate documents include "the principles of natural science", "the principles of mechanics", and "the principles of nature", and the second candidate documents include "Newton's mechanics", "the principles of mechanics", and "the three laws and the law of universal gravitation". A similarity analysis is performed on the document to be searched, the first candidate document, and the second candidate document, and it is determined that the three documents with the highest similarity at the text level and the semantic level are "Newton's mechanics", "the principles of mechanics", and "the three laws and the law of universal gravitation". The target similar documents are "Newton's mechanics", "the principles of mechanics", and "the three laws and the law of universal gravitation".

[0081] In the embodiment of the present application, firstly, by extracting the target text content of the document to be searched and the first semantic vector of the target text content, searching for the first candidate document and searching for the second candidate document based on the target text content and the first semantic vector, the search can be combined at the text level and the semantic level, so that the candidate documents obtained by the search are more comprehensive. Then, because the semantic level and the text level are combined, the similarity analysis is performed on the document to be searched, the first candidate document and the second candidate document, and the target similar document of the document to be searched is determined, thereby improving the accuracy of the determined target similar document.

[0082] Figure 3 A flowchart of a method for extracting a first semantic vector in a document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0083] Step S2022: extracting a first semantic vector of the target text content using a pre-trained natural language model according to the target text content.

[0084] The natural language model is a neural network model that can encode the text content of natural language into feature vectors. The natural language model can be a DOC2VEC model, a WORD2VEC model, an LSA model, a GLOVE model, a BERT model, etc. In the embodiment of the present application, the BERT model is used for illustration. The BERT model can be encoded based on the context in the text content, so that the word vector in the feature vector can contain the semantic features of the context, and the obtained semantic vector is more accurate.

[0085] The natural language model is pre-trained using a parallel sample set, which includes unlabeled text content samples and text content samples labeled with semantic features.

[0086] Specifically, the target text content is input into a pre-trained natural language model, and based on the semantic features of the target text content, the target text content is encoded with a feature vector to obtain a first semantic vector of the target text content.

[0087] For example, the target text content is "The Yangtze River system has rich biological resources", the natural language model is the BERT model, the target text content is input into the pre-trained BERT model, and the target text content is feature vector encoded based on the semantic features of the target text content to obtain the first semantic vector of the target text content {E_token_1 (Yangtze River system), E_token_2 (has), E_token_3 (rich), E_token_4 (of), E_token_5 (biological resources)}.

[0088] The first semantic vector of the target text is extracted using a pre-trained natural language model, which improves the efficiency of extracting the first semantic vector and thus improves the efficiency of the overall document search. In addition, the first semantic vector extracted by the trained natural language model is more accurate, which can ensure that the subsequent search for the second candidate document based on the first semantic vector is more accurate.

[0089] Figure 4 A flowchart of a first candidate document search method in a document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0090] Step S2042: extracting keywords from the target text content.

[0091] Keywords are words that can represent key information of the target text content. They can be the words that appear most frequently in the target text content, or they can be core entity words in the target text content, or they can be words such as time and place that can be used for subsequent precise searches. The target text content can be a file name, original text content, author name, directory, etc. Correspondingly, keywords can be file name keywords, text content keywords, author name keywords, directory keywords, etc. For example, in the target text content "On the Movement of Objects", the keyword is "movement"; the target text content "XX City Aquatic Products Situation Report in 2022" has keywords "2022", "XX City", and "Aquatic Products"; the target text content is "Author: XX, YY, ZZZ", where the first author XXX is the core author (entity word), and the keyword is "XX".

[0092] The keyword extraction method of the target text content can be to extract using a pre-trained keyword extraction model, or to extract using a manual selection method, or to extract using a pre-established keyword query table. In the embodiment of the present application, the Textrank model is used for keyword extraction.

[0093] Specifically, keywords in the target text content are extracted according to the content type of the target text content.

[0094] exist Figure 2In the embodiment, it has been explained that the target text content is the content of the text features of the document to be searched. It can be the directory of the document to be searched, the author name of the document to be searched, the original text content of the document to be searched, etc. Correspondingly, the content types of the target text content include directory, author name, original text content, etc. For the directory type, because the directory is generally a multi-level directory, the directory name of the first-level directory contains more information, and it can be searched as a keyword to obtain a more comprehensive candidate document. For the original text content, because the text content of the first paragraph in the general original text content is a summary content, it contains more comprehensive information, and it can be searched as a keyword to obtain a more comprehensive candidate document.

[0095] Step S2044: Searching for the first candidate document based on the target text content and keywords.

[0096] The method of searching for the first candidate document is to send a first document search request to the storage end 102, wherein the first document search request includes target text content and keywords of the document to be searched. After receiving the first document search request, the storage end 102 queries multiple stored sample documents and selects one or more sample documents corresponding to the target text content and keywords as the first candidate document.

[0097] Specifically, based on the target text content and keywords, one or more sample documents corresponding to the target text content and keywords among the multiple sample documents are determined as first candidate documents.

[0098] For example, based on the target text content "Author: XX, YY, ZZZ" and the keyword "XX", multiple sample documents B1 ("Author: YY, ZZZ", keyword: YY), B2 ("Author: XX, YY, ZZZ", keyword: XX), B3 ("Author: WW, XX, YY", keyword: WW), B4 ("Author: XX", keyword: XX) are determined, and B2 and B4 corresponding to the target text content "Author: XX, YY, ZZZ" and the keyword "XX" are determined as the first candidate documents.

[0099] Based on the target text content and keywords, the first candidate document is searched. The search is combined with keywords, so that the first candidate document obtained by the search is more accurate, and the target similar documents determined subsequently are more accurate.

[0100] The target text content in the embodiment of the present application includes the file name and original text content of the document to be searched. Figure 5 A flowchart of a first candidate document search method in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0101] Step S2046: extract file name keywords from the file name, and extract text content keywords from the original text content.

[0102] The target text content includes the file name and original text content of the document to be searched.

[0103] The file name is the file name of the document to be searched, and the original text content is the text that records the project content in the document to be searched. For example, the document to be searched is a certain engineering project document, and the file name is "General Overview of a Certain Project", and the original text content is "Chapter 1 General Overview of the Project 1. Project Overview 1.1 Introduction to the Bidding Situation Project Name: XX Housing Project in XX City Project Location: XX City Construction Unit: XX City Engineering Construction Company Site Conditions: The site is ready for construction..."

[0104] The file name keywords are words in the file name that can represent the key information of the file name. They can be the words that appear most frequently in the file name, or they can be the core entity words in the file name, or they can be the words that can be used for subsequent precise searches such as time and place in the file name, etc. The text content keywords are words in the original text content that can represent the key information of the target text content. They can be the words that appear most frequently in the original text content, or they can be the core entity words in the original text content, or they can be the words that can be used for subsequent precise searches such as time and place in the original text content, etc. Continuing with the above example, the extracted file name keywords are "a certain project", and the extracted text content keywords are "project overview", "XX housing project", and "XX city".

[0105] The method of extracting file name keywords from the file name and extracting text content keywords from the original text content can be performed by using a pre-trained file name keyword extraction model and a text content keyword extraction model, or by manual selection, or by extracting according to a pre-established file name keyword query table and a text content keyword query table, which is not limited here.

[0106] Step S2048: Searching for the first candidate document based on the file name, original text content, file name keywords and text content keywords.

[0107] Specifically, based on the file name, original text content, file name keywords and text content keywords, one or more sample documents corresponding to the file name, original text content, file name keywords and text content keywords in multiple sample documents are determined as first candidate documents.

[0108] For example, based on the file name "Overview of a certain project", the original text content "Chapter 1 Overall Overview of the Project 1. Project Overview 1.1 Introduction to the Bidding Situation Project Name: XX Housing Project in XX City Project Location: XX City Construction Unit: XX City Engineering Construction Company On-site conditions: The site is ready for commencement of construction...", the file name keywords "a certain project" and the text content keywords "Project Overview", "XX Housing Project", "XX City", two sample documents C6 and C7 corresponding to the file name, original text content, file name keywords and text content keywords in multiple sample documents C1, C2, C3...C9 are determined as the first candidate documents (C6, C7).

[0109] Extract file name keywords from the file name, extract text content keywords from the original text content, and search for the first candidate document based on the file name, the original text content, the file name keywords and the text content keywords. Combine multiple text-level indexes to perform the search, so that the first candidate document obtained is more comprehensive.

[0110] Figure 6 A flowchart of a first candidate document search method in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0111] Step 2050: Based on the file name, search a pre-established document database to obtain a candidate file name document.

[0112] Specifically, based on the file name, the ES search server is used to search a pre-established document database to obtain the file name candidate documents.

[0113] For example, based on the file name "Overview of a certain project", the ES search server is used to search the pre-established document database to obtain multiple file name candidate documents C1, C2, C3...C9.

[0114] By searching for file names, corresponding candidate documents of the file names are obtained, which improves the efficiency and pertinence of the search and lays a document foundation for the subsequent determination of the first candidate document.

[0115] Step 2052: Based on the original text content, search a pre-established document database to obtain candidate documents of the original text content.

[0116] Specifically, based on the original text content, the ES search server is used to search a pre-established document database to obtain candidate documents of the original text content.

[0117] For example, based on the original text content "Chapter 1 Overall Overview of the Project 1. Project Overview 1.1 Introduction to the Bidding Situation Project Name: XX Housing Project in XX City Project Location: XX City Construction Unit: XX City Engineering Construction Company Site Conditions: The site is ready for construction...", the ES search server is used to search the pre-established document database to obtain multiple original text content candidate documents C1, C2, C3...C15.

[0118] By searching the original text content, corresponding original text content candidate documents are obtained, which improves the efficiency and pertinence of the search and lays a document foundation for the subsequent determination of the first candidate document.

[0119] Step 2054: Based on the file name keyword, search the pre-established document database to obtain the file name keyword candidate documents.

[0120] Specifically, based on the file name keyword, the ES search server is used to search a pre-established document database to obtain the file name keyword candidate documents.

[0121] For example, based on the file name keyword "a certain project", the ES search server is used to search the pre-established document database to obtain the file name keyword candidate documents C1, C6, C7...C12.

[0122] By searching for file name keywords, corresponding file name keyword candidate documents are obtained, which improves the efficiency and pertinence of the search and lays a document foundation for the subsequent determination of the first candidate document.

[0123] Step 2056: Based on the text content keywords, search the pre-established document database to obtain text content keyword candidate documents.

[0124] Specifically, based on the text content keywords, the ES search server is used to search a pre-established document database to obtain text content keyword candidate documents.

[0125] For example, based on the text content keywords "project overview", "XX housing project", and "XX city", the ES search server is used to search and obtain text content keyword candidate documents C1, C6, C9...C15.

[0126] By searching for text content keywords, corresponding text content keyword candidate documents are obtained, which improves the efficiency and pertinence of the search and lays a document foundation for the subsequent determination of the first candidate document.

[0127] Step 2058: Merge the file name candidate document, the original text content candidate document, the file name keyword candidate document and the text content keyword candidate document into a first candidate document.

[0128] Specifically, a document intersection set of a plurality of file name candidate documents, a plurality of original text content candidate documents, a plurality of file name keyword candidate documents, and a plurality of text content keyword candidate documents is selected, and the document intersection set is determined to be the first candidate document.

[0129] For example, the document intersection set of the file name candidate documents C1, C2, C3...C9, the original text content candidate documents C1, C2, C3...C15, the file name keyword candidate documents C1, C6, C7...C12 and the text content keyword candidate documents C1, C6, C9...C15 is C1, C2, C3...C15, and the document intersection set is determined to be the first candidate document C1, C2, C3...C15.

[0130] In the embodiment of the present application, by searching and obtaining file name candidate documents, original text content candidate documents, file name keyword candidate documents, and text content keyword candidate documents based on the file name, original text content, file name keywords, and text content keyword respectively, the search conditions are reduced, more search results are obtained, the documents included in the first candidate document are more comprehensive, and the target similar documents determined subsequently are more accurate.

[0131] Figure 7 A flowchart of a second candidate document search method in a document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0132] Step S2060: extracting the second semantic vector of the target text content in each sample document.

[0133] The sample documents are multiple documents stored in the storage end 102 , from which the first candidate document and the second candidate document are determined according to the document to be searched.

[0134] The second semantic vector of the target text content is a vector of the semantic features of the target text content of the sample document, which is a semantic feature vector obtained by vectorizing the target text content based on the semantic feature level of the target text content. Semantic features include: word features, part of speech features, character features, position features, etc.

[0135] Specifically, according to the semantic features of the target text content of each sample document, a second semantic vector of the target text content is obtained.

[0136] For example, the sample documents include "Newtonian Mechanics", "Principles of Mechanics", and "Three Laws and the Law of Universal Gravitation". The target text contents are "The relationship between force and motion", "Force is the interaction between objects", and "Newton's Second Law: The magnitude of an object's acceleration is proportional to the force applied". Based on the semantic features, the second semantic vectors of the target texts are B1, B2, and B3 respectively.

[0137] Step S2062: Compare the first semantic vector with each second semantic vector respectively, and determine a similar vector to the first semantic vector from each second semantic vector.

[0138] Compare the first semantic vector with each second semantic vector respectively, compare each word vector of the first semantic vector with each word vector of the second semantic vector one by one, count the proportion of the same word vector in the total word vector of the first semantic vector and the second semantic vector, and obtain the similarity value, or determine the cosine value of the first semantic vector and the second semantic vector as the similarity value. Then determine the similarity vector according to the similarity value, and the method for determining the similarity vector according to the similarity value can be to determine one or more second semantic vectors with the highest similarity value as the similarity vector, or to compare each similarity value with a preset similarity value threshold, and determine the second semantic vector greater than the similarity value threshold as the similarity vector.

[0139] The similarity value is calculated as follows:

[0140]

[0141]

[0142] Among them, A represents the first semantic vector, B represents the second semantic vector, and A i represents the word vector in the first semantic vector, B i Represents the word vector in the second semantic vector.

[0143] Specifically, the first semantic vector is compared with each second semantic vector to obtain a similarity value, and a similar vector of the first semantic vector is determined from each second semantic vector according to the similarity value.

[0144] Continuing with the above example, the first semantic vector A is {12, 34, 87, 46, 72}, the second semantic vector B1 is {12, 20, 87, 44, 75}, the second semantic feature vector B2 is {12, 34, 85, 46, 75}, and the third semantic feature vector B3 is {13, 20, 87, 43, 72}. According to formula 1, the similarity value between A and B1 is 0.25, the similarity value between A and B2 is 0.375, and the similarity value between A and B3 is 0.375. The similarity values ​​between A and B2 and B3 are the highest, and B2 and B3 are determined to be similar vectors.

[0145] Step S2064: Determine the sample document corresponding to the similarity vector as the second candidate document.

[0146] Continuing with the above example, the sample documents corresponding to B2 and B3 are "Principles of Mechanics", "Three Major Laws and the Law of Universal Gravitation", and "Principles of Mechanics" and "Three Major Laws and the Law of Universal Gravitation" are determined as the second candidate documents.

[0147] In an embodiment of the present application, a similarity vector is obtained by comparing the first semantic vector with the second semantic vector, and a corresponding second candidate document is determined, thereby ensuring that the determined second candidate document can accurately correspond to the semantic features of the document to be searched, thereby improving the accuracy of the second candidate document, and further improving the accuracy of subsequent determination of target similar documents.

[0148] Fig. 8A A flow chart of a method for determining a target similar document in a document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0149] Step S2066: Obtain designated feature data of the document to be searched, the first candidate document, and the second candidate document.

[0150] The specified feature data is a numerical value indicating the literal features and / or semantic features between the document to be searched, the first candidate document and the second candidate document. For example, a file name semantic vector indicating the semantic features of the file name, a file name literal feature indicating the literal features of the file name, an original text content semantic vector indicating the semantic features of the original text content, a text literal feature indicating the literal features of the original text content, a keyword weight indicating the literal features and / or semantic features of the keywords, etc. The specified feature data will be determined according to the specific project scenario, and the specified feature data can be one or more. For example, the file name literal feature value for the engineering project scenario, the text literal features and keyword weights of the literal features of the original text content for the academic and scientific research project scenario, etc.

[0151] Specifically, according to a specific project scenario, the specified feature data of the document to be searched, the first candidate document and the second candidate document are obtained.

[0152] For example, the specific project scenario is an academic research project scenario, and the text literal features and keyword weights of the document to be searched, the first candidate document, and the second candidate document are obtained.

[0153] Step S2068: Calculate a first similarity between the specified feature data of the document to be searched and the specified feature data of the first candidate document, and a second similarity between the specified feature data of the document to be searched and the specified feature data of the second candidate document.

[0154] Specifically, the document to be searched and the first candidate document or the second candidate document are input into a pre-trained similarity calculation model, and based on the specified feature data, the first similarity and the second similarity are calculated.

[0155] The similarity calculation model is a neural network model that can calculate the similarity of two documents based on the target text content and / or the semantic vector of the target text of the two documents. Optionally, the similarity calculation model can be an interactive model or a dual-tower model.

[0156] Continuing with the above example, the document to be searched and the first candidate document or the second candidate document are input into the pre-trained similarity calculation model, and based on the literal features of the text and the keyword weights, the first similarities A1 (0.6, 0.6), A2 (0.3, 0.3), A3 (0.5, 0.4) and the second similarities B1 (0.7, 0.5), B2 (0.8, 0.8) are calculated.

[0157] Figure 8B A schematic diagram of the processing flow of a double-tower model in a document search method provided in an embodiment of the present application is shown. Figure 8C A schematic diagram of the processing flow of an interactive model in a document search method provided in an embodiment of the present application is shown.

[0158] in, Figure 8B The processing steps for the dual tower model are as follows:

[0159] After the original text contents of the document to be searched and the first candidate document or the second candidate document are respectively input into two pre-trained BERT1 and BERT2 models, they are respectively subjected to pooling processing to obtain the specified feature data of the original text content of the document to be searched and the specified feature data of the original text content of the first candidate document or the second candidate document, and the two specified feature data are jointly input into the fully connected layer and the normalization layer to obtain the first similarity or the second similarity.

[0160] in, Figure 8C The processing steps of the interactive model are as follows:

[0161] The semantic features of the original text content of the document to be searched and the first candidate document or the second candidate document are extracted, and after embedding processing, the semantic vectors of the original text content of the document to be searched and the first candidate document or the second candidate document are obtained, and a pre-trained BERT model is input to perform regression classification to obtain the first text similarity or the second text similarity between the document to be searched and the first candidate document or the second candidate document.

[0162] Compared with the interactive model and the dual-tower model, the dual-tower model has a faster similarity calculation speed, but a lower similarity calculation accuracy. The interactive model has a higher similarity calculation accuracy, but a lower similarity calculation speed. Therefore, the specific use of the interactive model or the dual-tower model needs to be determined according to the specific similarity calculation scenario and similarity calculation requirements.

[0163] Step S2070: Based on the first similarity and the second similarity, determine a target similar document of the document to be searched from the first candidate document and the second candidate document.

[0164] Specifically, based on the first similarity and the second similarity, the first candidate document and the second candidate document are sorted, and according to the sorting result, a target similar document to the document to be searched is determined.

[0165] According to the sorting results, the method of determining the target similar document of the document to be searched can be to determine one or more candidate documents with the highest first similarity and second similarity as the target similar document, or to compare the first similarity or the second similarity of the candidate document with a preset similarity threshold and determine the candidate document with a similarity greater than the similarity threshold as the target similar document.

[0166] Continuing with the above example, based on the first similarities A1 (0.6, 0.6), A2 (0.3, 0.3), A3 (0.5, 0.4) and the second similarities B1 (0.7, 0.5), B2 (0.8, 0.8), the first candidate documents and the second candidate documents are sorted B2, B1, A1, A3, A2, and according to the sorting results, the candidate document B2 with the highest first similarity and second similarity is determined as the target similar document.

[0167] By calculating the first similarity and the second similarity according to the specified feature data, the similarity can be determined in a targeted manner according to different features, so that the corresponding target similar documents can be determined subsequently, and the target similar documents are determined according to the first similarity and the second similarity, thereby improving the accuracy of the determined target similar documents.

[0168] Fig. 8A The specified feature data in the embodiment includes a file name semantic vector. Fig. 9 A flow chart of a method for determining a target similar document in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0169] Step S2072: Calculate the first file name similarity between the document to be searched and the first candidate document, and the second file name similarity between the document to be searched and the second candidate document, respectively, based on the file name semantic vectors of the document to be searched, the first candidate document, and the second candidate document.

[0170] The file name semantic vector is a vector that represents the semantic features of the file name.

[0171] Specifically, the file name of the document to be searched and the file name of the first candidate document or the second candidate document are input into a pre-trained similarity calculation model, and based on the file name semantic vector, the first file name similarity and the second file name similarity are calculated.

[0172] For example, the file name of the document to be searched is "Mathematical Principles of Natural Philosophy", the file name of the first candidate document A is "Principles of Natural Science", and the file name of the second candidate document B is "Principles of Mechanics". "Mathematical Principles of Natural Philosophy" and "Principles of Natural Science" or "Principles of Mechanics" are input into the pre-trained similarity calculation model, and based on the file name semantic vector, the similarity of the first file name is calculated to be 0.6 and the similarity of the second file name is 0.4.

[0173] Step S2074: Based on the first file name similarity and the second file name similarity, determine a target similar document of the document to be searched from the first candidate document and the second candidate document.

[0174] Specifically, based on the first file name similarity and the second file name similarity, the first candidate document and the second candidate document are sorted, and according to the sorting result, a target similar document of the document to be searched is determined.

[0175] Continuing with the above example, based on the similarity of the first file name being 0.6 and the similarity of the second file name being 0.4, the first candidate document and the second candidate document are sorted into A, B, and according to the sorting result, the candidate document A having the highest similarity between the first file name and the second file name is determined as the target similar document.

[0176] By calculating the first file name similarity and the second file name similarity based on the file name semantic vector, the file name similarity can be determined in a targeted manner based on the file name semantic features, so that the corresponding target similar documents can be determined subsequently, and the target similar documents can be determined based on the first file name similarity and the second file name similarity. In the application scenario of the file name semantic features, the accuracy of the determined target similar documents is improved.

[0177] The specified feature data in the embodiment of the present application includes file name surface features. Fig.10 A flow chart of a method for determining a target similar document in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0178] Step S2076: Calculate the first file name face similarity between the document to be searched and the first candidate document, and the second file name face similarity between the document to be searched and the second candidate document, respectively, based on the file name face features of the document to be searched, the first candidate document, and the second candidate document.

[0179] Specifically, the file name of the document to be searched and the file name of the first candidate document or the second candidate document are input into a pre-trained similarity calculation model, and based on the face features of the file names, the first file name face similarity and the second file name face similarity are calculated.

[0180] For example, the file name of the document to be searched is "Mathematical Principles of Natural Philosophy", the file name of the first candidate document A is "Principles of Natural Science", and the file name of the second candidate document B is "Principles of Mechanics". "Mathematical Principles of Natural Philosophy" and "Principles of Natural Science" or "Principles of Mechanics" are input into the pre-trained similarity calculation model, and based on the file name semantic vector, the face similarity of the first file name is calculated to be 0.8 and the face similarity of the second file name is 0.2.

[0181] Step S2078: Based on the face similarity of the first file name and the face similarity of the second file name, determine a target similar document to be searched from the first candidate document and the second candidate document.

[0182] Specifically, based on the face similarity of the first file name and the face similarity of the second file name, the first candidate document and the second candidate document are sorted, and according to the sorting result, the target similar document of the document to be searched is determined.

[0183] Continuing with the above example, based on the first file name face similarity of 0.8 and the second file name similarity of 0.2, the first candidate document and the second candidate document are sorted into A, B. According to the sorting results, the candidate document A with the highest face similarity between the first file name and the second file name is determined as the target similar document.

[0184] By calculating the face similarity of the first file name and the face similarity of the second file name according to the file name semantic vector, the file name face similarity can be determined in a targeted manner according to the file name face features, so as to subsequently determine the corresponding target similar documents, and determine the target similar documents according to the first file name face similarity and the second file name face similarity, which improves the accuracy of the determined target similar documents in the application scenario of the file name face features.

[0185] Fig. 8A The specified feature data in the embodiment includes original text content semantic vectors. Fig.11 A flow chart of a method for determining a target similar document in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0186] Step S2080: Calculate the first text similarity between the document to be searched and the first candidate document, and the second text similarity between the document to be searched and the second candidate document, respectively, based on the original text content semantic vectors of the document to be searched, the first candidate document, and the second candidate document.

[0187] Specifically, the original text content of the document to be searched and the original text content of the first candidate document or the second candidate document are input into a pre-trained similarity calculation model, and the first text similarity and the second text similarity are calculated based on the original text content semantic vector.

[0188] For example, the original text content of the document to be searched is "Newton's Second Law: The magnitude of an object's acceleration is proportional to the force applied to it", the original text content of the first candidate document A is "The relationship between force and motion", and the original text content of the second candidate document B is "Force is the interaction between objects". "Newton's Second Law: The magnitude of an object's acceleration is proportional to the force applied to it" and "The relationship between force and motion" or "Force is the interaction between objects" are input into the pre-trained similarity calculation model, and based on the text content semantic vector, the first text similarity is calculated to be 0.7 and the second text similarity is 0.3.

[0189] Step S2082: Based on the first text similarity and the second text similarity, determine a target similar document to be searched from the first candidate document and the second candidate document.

[0190] Specifically, based on the first text similarity and the second text similarity, the first candidate document and the second candidate document are sorted, and according to the sorting result, a target similar document to the document to be searched is determined.

[0191] Continuing with the above example, based on the first text similarity of 0.7 and the second text similarity of 0.3, the first candidate document and the second candidate document are sorted into A, B, and according to the sorting result, the candidate document A with the highest first text similarity and second text similarity is determined as the target similar document.

[0192] By calculating the first text similarity and the second text similarity based on the semantic vector of the original text content, the text similarity can be determined in a targeted manner based on the semantic vector of the original text content, so that the corresponding target similar documents can be determined subsequently, and the target similar documents can be determined based on the first text similarity and the second text similarity. In the application scenario of the semantic features of the original text content, the accuracy of the determined target similar documents is improved.

[0193] Fig. 8A The specified feature data in the embodiment includes text literal features of the original text content. Fig.12 A flow chart of a method for determining a target similar document in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0194] Step S2084: Calculate the first textual similarity between the document to be searched and the first candidate document, and the second textual similarity between the document to be searched and the second candidate document, respectively, based on the textual features of the original text contents of the document to be searched, the first candidate document, and the second candidate document.

[0195] Specifically, the original text content of the document to be searched and the original text content of the first candidate document or the second candidate document are input into a pre-trained similarity calculation model, and the first text literal similarity and the second text literal similarity are calculated based on the text literal features of the original text content.

[0196] For example, the original text content of the document to be searched is "Newton's Second Law: The magnitude of an object's acceleration is proportional to the force applied to it", the original text content of the first candidate document A is "The relationship between force and motion", and the original text content of the second candidate document B is "Force is the interaction between objects". "Newton's Second Law: The magnitude of an object's acceleration is proportional to the force applied to it" and "The relationship between force and motion" or "Force is the interaction between objects" are input into the pre-trained similarity calculation model, and based on the text content semantic vector, the literal similarity of the first text is calculated to be 0.2 and the literal similarity of the second text is calculated to be 0.3.

[0197] Step S2086: Based on the first text literal similarity and the second text literal similarity, determine a target similar document to be searched from the first candidate document and the second candidate document.

[0198] Specifically, based on the first text literal similarity and the second text literal similarity, the first candidate document and the second candidate document are sorted, and according to the sorting result, a target similar document to the document to be searched is determined.

[0199] Continuing with the above example, based on the first text literal similarity of 0.2 and the second text literal similarity of 0.3, the first candidate document and the second candidate document are sorted A, B, and according to the sorting results, the candidate document B with the highest literal similarity between the first text and the second text is determined as the target similar document.

[0200] By calculating the first text literal similarity and the second text literal similarity based on the text literal features of the original text content, the text literal similarity can be determined in a targeted manner based on the text literal features of the original text content, so as to subsequently determine the corresponding target similar documents, and the target similar documents are determined based on the first text literal similarity and the second text literal similarity, which improves the accuracy of the determined target similar documents in the application scenario of the text literal features of the original text content.

[0201] Fig. 8A The specified feature data in an embodiment includes keyword weights. Fig.13 A flow chart of a method for determining a target similar document in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0202] Step S2088: Calculate the first keyword similarity between the document to be searched and the first candidate document, and the second keyword similarity between the document to be searched and the second candidate document according to the keyword weights of the document to be searched, the first candidate document and the second candidate document.

[0203] The keyword weight is the weight assigned to the keyword when using Textrank to extract the keyword. The specific method is to perform IOU calculation on the keywords of the search document, the keywords of the first candidate document, and the keywords of the second candidate document.

[0204] Specifically, the keywords of the document to be searched and the keywords of the first candidate document or the second candidate document are input into a pre-trained similarity calculation model, and based on the keyword weights, the first keyword similarity and the second keyword similarity are calculated.

[0205] For example, the keywords of the document to be searched are "acceleration" and "force", the keywords of the first candidate document A are "force" and "motion", and the keywords of the second candidate document B are "force" and "interaction". "Acceleration", "force" and "force", "motion" or "force", "interaction" are input into the pre-trained similarity calculation model, and based on the keyword weights, the similarity of the first keyword is calculated to be 0.8 and the similarity of the second keyword is 0.5.

[0206] Step S2090: Based on the first keyword similarity and the second text literal similarity, determine a target similar document of the document to be searched from the first candidate document and the second candidate document.

[0207] Specifically, based on the first keyword similarity and the second text literal similarity, the first candidate document and the second candidate document are sorted, and according to the sorting result, a target similar document to the document to be searched is determined.

[0208] Continuing with the above example, based on the first keyword similarity of 0.8 and the second keyword similarity of 0.5, the first candidate document and the second candidate document are sorted into A, B, and according to the sorting results, the candidate document A with the highest similarity between the first keyword and the second keyword is determined as the target similar document.

[0209] By calculating the first keyword similarity and the second keyword similarity according to the keyword weight, the keyword similarity can be determined in a targeted manner according to the keyword weight, so as to subsequently determine the corresponding target similar document, and the target similar document is determined according to the first keyword similarity and the second keyword similarity. In the application scenario of the keyword weight, the accuracy of the determined target similar document is improved.

[0210] Fig. 8AThe first similarity in the embodiment includes: first file name similarity, first file name literal similarity, first text similarity, first text literal similarity and first keyword similarity; the second similarity includes: second file name similarity, second file name literal similarity, second text similarity, second text literal similarity and second keyword similarity. Fig.14 A flow chart of a method for determining a target similar document in another document search method provided according to an embodiment of the present application is shown, which specifically includes the following steps:

[0211] Step S2092: performing weighted calculation on the first file name similarity, the first file name literal similarity, the first text similarity, the first text literal similarity and the first keyword similarity to obtain a first weight.

[0212] The weighted calculation may be performed by allocating an average weight or by performing a weighted calculation according to a preset weight. For example, if the similarities are 0.8 and 0.2 respectively, the average weight is allocated for weighted calculation 0.8*0.5+0.2*0.5=0.5, and the weighted calculation according to the preset weight is 0.8*0.8+0.2*0.2=0.68.

[0213] The first weight is a weight indicating the text-level features and semantic-level features of the document to be searched and the first candidate document.

[0214] Step S2094: performing weighted calculation on the second file name similarity, the second file name literal similarity, the second text similarity, the second text literal similarity and the second keyword similarity to obtain a second weight.

[0215] The second weight is a weight indicating the text-level features and the semantic-level features of the document to be searched and the second candidate document.

[0216] Step S2096: Sort the first candidate document and the second candidate document according to the first weight and the second weight to obtain a sorting result.

[0217] The first candidate document and the second candidate document are sorted according to the first weight and the second weight, which may be first sorting the first candidate document to obtain a first sorting result, then sorting the second candidate document to obtain a second sorting result, and finally determining the sorting result based on the first sorting result and the second sorting result. Alternatively, the first candidate document and the second candidate document are sorted together to obtain a sorting result.

[0218] Step S2098: Based on the ranking result, determine a target similar document of the document to be searched from the first candidate document and the second candidate document.

[0219] The target similar document of the document to be searched is determined based on the ranking result. One or more candidate documents with the highest ranking may be determined as the target similar document, or the first weight, the second weight and a preset weight threshold may be compared, and the candidate document with a weight greater than the weight threshold may be determined as the target similar document.

[0220] The first weight is calculated based on the first file name similarity, the first file name face similarity, the first text similarity, the first text literal similarity and the first keyword similarity, and the second weight is calculated based on the second file name similarity, the second file name face similarity, the second text similarity, the second text literal similarity and the second keyword similarity. The first candidate document and the second candidate document are sorted according to the first weight and the second weight to obtain the sorting result, and the target similar document is determined based on the sorting result. The semantic level features and text level features of the document to be searched, the first candidate document and the second candidate document are combined to calculate the first weight and the second weight, and the target similar document is determined by sorting according to the first weight and the second weight, thereby improving the accuracy of the determined target similar document, improving the accuracy of document search, and improving user experience.

[0221] The following combination Fig.15 Taking the document search method provided by the present application as an example, the document search method is further described by applying the search application based on the file name and original text content of the document to be searched. Fig.15 A processing flow chart of a document search method provided by an embodiment of the present application for searching based on the file name and original text content of the document to be searched is shown, which specifically includes the following steps:

[0222] Step S1502: Obtain the document to be searched.

[0223] Step S1504: Obtain the file name and original text content of the document to be searched.

[0224] Step S1506: extract file name keywords from the file name, and extract text content keywords from the original text content.

[0225] Extraction is performed using the pre-trained file name keyword extraction model and text content keyword extraction model.

[0226] Step S1508: extracting the first semantic vector of the file name and the original text content using a pre-trained natural language model according to the file name and the original text content.

[0227] The pre-trained BERT model is used to extract the first semantic vectors of the file name and the original text content.

[0228] Step S1510: Searching for the first candidate document based on the file name, original text content, file name keywords and text content keywords.

[0229] Based on the file name, original text content, file name keywords and text content keywords, 4N first candidate documents are searched and obtained.

[0230] Step S1512: searching for a second candidate document based on the file name and the first semantic vector of the original text content.

[0231] Based on the first semantic vector of the file name and the original text content, 2N second candidate documents are searched and obtained.

[0232] Step S1514: Obtain designated feature data of the document to be searched, the first candidate document, and the second candidate document.

[0233] The specified feature data includes a file name semantic vector, a file name face feature, an original text content semantic vector, a text literal feature of the original text content, and a keyword weight.

[0234] Step S1516: Calculate a first similarity between the specified feature data of the document to be searched and the specified feature data of the first candidate document, and a second similarity between the specified feature data of the document to be searched and the specified feature data of the second candidate document.

[0235] The first similarity includes: first file name similarity, first file name literal similarity, first text similarity, first text literal similarity and first keyword similarity; the second similarity includes: second file name similarity, second file name literal similarity, second text similarity, second text literal similarity and second keyword similarity.

[0236] The interactive model is used to calculate a first similarity between the designated feature data of the document to be searched and the designated feature data of the first candidate document, and a second similarity between the designated feature data of the document to be searched and the designated feature data of the second candidate document.

[0237] Step S1518: Perform weighted calculation on the first similarity to obtain a first weight.

[0238] A weighted calculation is performed on five similarities, namely, first file name similarity, first file name literal similarity, first text similarity, first text literal similarity and first keyword similarity, to obtain a first weight.

[0239] Step S1520: performing weighted calculation on the second similarity to obtain a second weight.

[0240] A weighted calculation is performed on five similarities, namely, the second file name similarity, the second file name literal similarity, the second text similarity, the second text literal similarity and the second keyword similarity, to obtain a second weight.

[0241] Step S1522: Sort the first candidate document and the second candidate document according to the first weight and the second weight to obtain a sorting result.

[0242] The 6N first candidate documents and second candidate documents are sorted according to the first weight and the second weight to obtain a sorting result.

[0243] Step S1524: Based on the ranking result, determine a target similar document of the document to be searched from the first candidate document and the second candidate document.

[0244] Based on the ranking result, the M candidate documents with the highest order are determined from the 6N first candidate documents and second candidate documents as target similar documents.

[0245] In an embodiment of the present application, firstly, by extracting the file name and original text content of the document to be searched and the first semantic vector of the file name and original text content, the first candidate document and the second candidate document are searched based on the file name, original text content and the first semantic vector, and the text level and semantic level of the file name and original text content can be fully combined for searching, so that the candidate documents obtained by the search are more comprehensive. Then, because the text level and semantic level of the file name and original text content are combined, the similarity of the document to be searched, the first candidate document and the second candidate document is calculated, and the weight is determined according to the similarity for sorting, and the target similar document is obtained according to the sorting result, which improves the accuracy of the determined target similar document.

[0246] Fig.16 A system flow diagram of a document search method provided by an embodiment of the present application is shown. The details are as follows:

[0247] After the user inputs the file name and original text content of the document to be searched, the document to be searched is decrypted, and the decryption methods include: ASE decryption method and RSA decryption method. The naming rules and file name formats of the file name of the document to be searched are judged for validity. If they are valid, the file name and original text content of the document to be searched are pre-processed accordingly, including: file name semantic vectorization, file name, keyword extraction, original text content vectorization, and original text content, and the following document search indexes are obtained: file name semantic vector, file name, keyword, original text content semantic representation vector, and original text content. Based on the above indexes, candidate documents are obtained after the corresponding searches, and the candidate documents are respectively input into the text content classification model and the file name classification model, and the candidate documents are post-processed to remove duplicate candidate documents to obtain the intersection set of candidate documents. The similarity is calculated based on each candidate document in the intersection set of candidate documents and the document to be searched, and each candidate document in the intersection set of candidate documents is sorted according to the similarity to determine the highest ranked candidate document to the target candidate document.

[0248] Corresponding to the above method embodiment, the present application also provides a document search device embodiment, Fig.17 FIG. 1 is a schematic diagram showing the structure of a document search device provided by an embodiment of the present application. Fig.17 As shown, the device comprises:

[0249] An extraction module 1702 is configured to obtain a document to be searched, extract a target text content of the document to be searched and a first semantic vector of the target text content;

[0250] A search module 1704 is configured to search for a first candidate document based on the target text content, and to search for a second candidate document based on the first semantic vector;

[0251] The determination module 1706 is configured to perform similarity analysis on the document to be searched, the first candidate document and the second candidate document to determine a target similar document to the document to be searched.

[0252] Optionally, the extraction module 1702 includes:

[0253] The first semantic vector extraction submodule is configured to extract the semantic vector of the target text content based on the target text content using a pre-trained natural language model.

[0254] Optionally, the search module 1704 includes:

[0255] A keyword extraction submodule, configured to extract keywords from target text content;

[0256] The first search submodule is configured to search for a first candidate document based on target text content and keywords.

[0257] Optionally, the keyword extraction submodule includes:

[0258] A file name keyword extraction unit is configured to extract file name keywords from the file name and extract text content keywords from the original text content;

[0259] The keyword extraction submodule includes:

[0260] The first search unit is configured to search for a first candidate document based on the file name, the original text content, the file name keyword, and the text content keyword.

[0261] Optionally, the first searching unit includes:

[0262] The file name candidate document search subunit is configured to search a pre-established document database based on the file name to obtain the file name candidate document;

[0263] The original text content candidate document search subunit is configured to search a pre-established document database based on the original text content to obtain the original text content candidate document;

[0264] The file name keyword candidate document search subunit is configured to search a pre-established document database based on the file name keyword to obtain the file name keyword candidate document;

[0265] The text content keyword candidate document search subunit is configured to search a pre-established document database based on the text content keyword to obtain the text content keyword candidate document;

[0266] The first candidate document merging subunit is configured to merge the file name candidate document, the original text content candidate document, the file name keyword candidate document and the text content keyword candidate document into a first candidate document.

[0267] Optionally, the search module 1704 includes:

[0268] A second semantic vector extraction submodule is configured to extract a second semantic vector of the target text content in each sample document;

[0269] A similarity vector determination submodule is configured to compare the first semantic vector with each second semantic vector respectively, and determine a similarity vector of the first semantic vector from each second semantic vector;

[0270] The second candidate document determination submodule is configured to determine a sample document corresponding to the similarity vector as a second candidate document.

[0271] Optionally, the determining module 1706 includes:

[0272] A designated feature data acquisition submodule is configured to acquire designated feature data of the document to be searched, the first candidate document, and the second candidate document;

[0273] A similarity determination submodule is configured to calculate a first similarity between the specified feature data of the document to be searched and the specified feature data of the first candidate document, and a second similarity between the specified feature data of the document to be searched and the specified feature data of the second candidate document;

[0274] The determination submodule is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the first similarity and the second similarity.

[0275] Optionally, the specified feature data includes a file name semantic vector, and the similarity determination submodule includes:

[0276] A file name similarity determination unit is configured to calculate a first file name similarity between the document to be searched and the first candidate document, and a second file name similarity between the document to be searched and the second candidate document, respectively, based on the file name semantic vectors of the document to be searched, the first candidate document, and the second candidate document;

[0277] The submodules include:

[0278] The first determining unit is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the first similarity and the second similarity.

[0279] Optionally, the specified feature data includes file name features, and the similarity determination submodule includes:

[0280] A file name similarity determination unit is configured to calculate a first file name name similarity between the file to be searched and the first candidate document, and a second file name name similarity between the file to be searched and the second candidate document, respectively, based on file name name features of the file to be searched, the first candidate document, and the second candidate document;

[0281] The submodules include:

[0282] The second determining unit is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the face similarity of the first file name and the face similarity of the second file name.

[0283] Optionally, the specified feature data includes a semantic vector of original text content, and the similarity determination submodule includes:

[0284] A file similarity determination unit is configured to calculate a first text similarity between the document to be searched and the first candidate document, and a second text similarity between the document to be searched and the second candidate document, respectively, based on original text content semantic vectors of the document to be searched, the first candidate document, and the second candidate document;

[0285] The submodules include:

[0286] The third determining unit is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the first text similarity and the second text similarity.

[0287] Optionally, the specified feature data includes text literal features of the original text content, and the similarity determination submodule includes:

[0288] A word similarity determination unit is configured to calculate a first text word similarity between the document to be searched and the first candidate document, and a second text word similarity between the document to be searched and the second candidate document, respectively, based on text name word features of original text contents of the document to be searched, the first candidate document, and the second candidate document;

[0289] The submodules include:

[0290] The fourth determining unit is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the first text literal similarity and the second text literal similarity.

[0291] Optionally, the specified feature data includes keyword weights, and the similarity determination submodule includes:

[0292] A keyword similarity determination unit is configured to calculate a first keyword similarity between the document to be searched and the first candidate document, and a second keyword similarity between the document to be searched and the second candidate document, respectively, according to keyword weights of the document to be searched, the first candidate document, and the second candidate document;

[0293] The submodules include:

[0294] The fifth determining unit is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the first keyword similarity and the second keyword similarity.

[0295] Optionally, the first similarity includes: first file name similarity, first file name literal similarity, first text similarity, first text literal similarity and first keyword similarity; the second similarity includes: second file name similarity, second file name literal similarity, second text similarity, second text literal similarity and second keyword similarity;

[0296] Correspondingly, the determination submodule includes:

[0297] A first weight calculation unit is configured to perform weighted calculation on the first file name similarity, the first file name literal similarity, the first text similarity, the first text literal similarity and the first keyword similarity to obtain a first weight;

[0298] A second weight calculation unit is configured to perform weighted calculation on the second file name similarity, the second file name literal similarity, the second text similarity, the second text literal similarity and the second keyword similarity to obtain a second weight;

[0299] A sorting unit, configured to sort the first candidate document and the second candidate document according to the first weight and the second weight to obtain a sorting result;

[0300] The sixth determining unit is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the ranking result.

[0301] In the embodiment of the present application, firstly, by extracting the target text content of the document to be searched and the first semantic vector of the target text content, searching for the first candidate document and searching for the second candidate document based on the target text content and the first semantic vector, the search can be combined at the text level and the semantic level, so that the candidate documents obtained by the search are more comprehensive. Then, because the semantic level and the text level are combined, the similarity analysis is performed on the document to be searched, the first candidate document and the second candidate document, and the target similar document of the document to be searched is determined, thereby improving the accuracy of the determined target similar document.

[0302] The above is a schematic scheme of a document search device of the present embodiment. It should be noted that the technical scheme of the document search device and the technical scheme of the document search method described above belong to the same concept. For details not described in detail in the technical scheme of the document search device, please refer to the description of the technical scheme of the document search method described above. In addition, each component in the device embodiment should be understood as a functional module that must be established to implement each step of the program flow or each step of the method, and each functional module is not an actual functional division or separation definition. A device claim defined by such a group of functional modules should be understood as a functional module architecture that mainly implements the solution through a computer program recorded in the specification, and should not be understood as a physical device that mainly implements the solution through hardware.

[0303] Fig.18 The structure block diagram of a computing device 1800 provided according to an embodiment of the present application is shown. The components of the computing device 1800 include but are not limited to a memory 1810 and a processor 1820. The processor 1820 is connected to the memory 1810 via a bus 1830, and the database 1850 is used to store data.

[0304] The computing device 1800 also includes an access device 1840 that enables the computing device 1800 to communicate via one or more networks 1860. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1840 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world microwave interconnection access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0305] In one embodiment of the present application, the above components of the computing device 1800 and Fig.18 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig.18 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present application. Those skilled in the art may add or replace other components as needed.

[0306] Computing device 1800 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. Computing device 1800 may also be a mobile or stationary server.

[0307] The processor 1820 is used to execute computer executable instructions of the document search method.

[0308] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the document search method described above are of the same concept, and the details not described in detail in the technical scheme of the computing device can be found in the description of the technical scheme of the document search method described above.

[0309] An embodiment of the present application further provides a computer-readable storage medium storing computer instructions, which are used in a document search method when executed by a processor.

[0310] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the document search method described above are of the same concept, and the details not described in detail in the technical scheme of the storage medium can be found in the description of the technical scheme of the document search method described above.

[0311] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0312] An embodiment of the present application further provides a chip storing a computer program, which implements the steps of the document search method when executed by the chip.

[0313] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0314] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0315] The preferred embodiments of the present application disclosed above are only used to help explain the present application. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present application. The present application selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present application, so that those skilled in the art can understand and use the present application well. The present application is only limited by the claims and their full scope and equivalents.

Claims

1. A document search method, characterized in that: include: Acquire a document to be searched, extract target text content of the document to be searched and a first semantic vector of the target text content; Based on the target text content, searching for a first candidate document, and based on the first semantic vector, searching for a second candidate document; Acquire specified feature data of the document to be searched, the first candidate document, and the second candidate document; Calculating a first similarity between the specified feature data of the document to be searched and the specified feature data of the first candidate document, and a second similarity between the specified feature data of the document to be searched and the specified feature data of the second candidate document; Based on the first similarity and the second similarity, a target similar document of the document to be searched is determined from the first candidate document and the second candidate document.

2. The method according to claim 1, characterized in that The step of extracting a first semantic vector of the target text content includes: According to the target text content, a first semantic vector of the target text content is extracted using a pre-trained natural language model.

3. The method according to claim 1, characterized in that The step of searching for a first candidate document based on the target text content includes: Extracting keywords from the target text content; Based on the target text content and the keyword, a first candidate document is searched.

4. The method according to claim 3, characterized in that The target text content includes the file name and original text content of the document to be searched; The extracting keywords from the target text content includes: Extract file name keywords from the file name, and extract text content keywords from the original text content; The step of searching for a first candidate document based on the target text content and the keyword includes: A first candidate document is searched based on the file name, the original text content, the file name keyword, and the text content keyword.

5. The method according to claim 4, characterized in that The step of searching for the first candidate document based on the file name, the original text content, the file name keyword, and the text content keyword includes: Based on the file name, searching a pre-established document database to obtain a candidate file name document; Based on the original text content, searching a pre-established document database to obtain candidate documents of the original text content; Based on the file name keyword, searching a pre-established document database to obtain file name keyword candidate documents; Based on the text content keywords, searching a pre-established document database to obtain text content keyword candidate documents; The file name candidate document, the original text content candidate document, the file name keyword candidate document and the text content keyword candidate document are merged into a first candidate document.

6. The method according to claim 1, characterized in that The step of searching for a second candidate document based on the first semantic vector includes: Extracting a second semantic vector of the target text content in each sample document; Comparing the first semantic vector with each of the second semantic vectors respectively, and determining a similar vector of the first semantic vector from each of the second semantic vectors; A sample document corresponding to the similarity vector is determined as a second candidate document.

7. The method according to claim 1, characterized in that The specified feature data includes a file name semantic vector; The calculating of a first similarity between the designated feature data of the document to be searched and the designated feature data of the first candidate document, and a second similarity between the designated feature data of the document to be searched and the designated feature data of the second candidate document, comprises: Calculate, according to the file name semantic vectors of the document to be searched, the first candidate document, and the second candidate document, respectively a first file name similarity between the document to be searched and the first candidate document, and a second file name similarity between the document to be searched and the second candidate document; The determining, based on the first similarity and the second similarity, a target similar document of the document to be searched from the first candidate document and the second candidate document comprises: Based on the first file name similarity and the second file name similarity, a target similar document of the document to be searched is determined from the first candidate document and the second candidate document.

8. The method according to claim 1, characterized in that The specified feature data includes file name features; The calculating of a first similarity between the designated feature data of the document to be searched and the designated feature data of the first candidate document, and a second similarity between the designated feature data of the document to be searched and the designated feature data of the second candidate document, comprises: Calculate, based on the face features of the file names of the document to be searched, the first candidate document, and the second candidate document, respectively a first file name face similarity between the document to be searched and the first candidate document, and a second file name face similarity between the document to be searched and the second candidate document; The determining, based on the first similarity and the second similarity, a target similar document of the document to be searched from the first candidate document and the second candidate document comprises: Based on the first file name face similarity and the second file name face similarity, a target similar document of the document to be searched is determined from the first candidate document and the second candidate document.

9. The method according to claim 1, characterized in that: The specified feature data includes the original text content semantic vector; The calculating of a first similarity between the designated feature data of the document to be searched and the designated feature data of the first candidate document, and a second similarity between the designated feature data of the document to be searched and the designated feature data of the second candidate document, comprises: Calculate, according to the original text content semantic vectors of the document to be searched, the first candidate document and the second candidate document, a first text similarity between the document to be searched and the first candidate document, and a second text similarity between the document to be searched and the second candidate document; The determining, based on the first similarity and the second similarity, a target similar document of the document to be searched from the first candidate document and the second candidate document comprises: Based on the first text similarity and the second text similarity, a target similar document of the document to be searched is determined from the first candidate document and the second candidate document.

10. The method according to claim 1, characterized in that The specified feature data includes text literal features of the original text content; The calculating of a first similarity between the designated feature data of the document to be searched and the designated feature data of the first candidate document, and a second similarity between the designated feature data of the document to be searched and the designated feature data of the second candidate document, comprises: Calculate, according to the textual features of the original text contents of the document to be searched, the first candidate document, and the second candidate document, respectively a first textual similarity between the document to be searched and the first candidate document, and a second textual similarity between the document to be searched and the second candidate document; The determining, based on the first similarity and the second similarity, a target similar document of the document to be searched from the first candidate document and the second candidate document comprises: Based on the first text literal similarity and the second text literal similarity, a target similar document to the document to be searched is determined from the first candidate document and the second candidate document.

11. The method according to claim 1, characterized in that: The specified feature data includes keyword weights; The calculating of a first similarity between the designated feature data of the document to be searched and the designated feature data of the first candidate document, and a second similarity between the designated feature data of the document to be searched and the designated feature data of the second candidate document, comprises: Calculating, according to the keyword weights of the document to be searched, the first candidate document, and the second candidate document, respectively, a first keyword similarity between the document to be searched and the first candidate document, and a second keyword similarity between the document to be searched and the second candidate document; The determining, based on the first similarity and the second similarity, a target similar document of the document to be searched from the first candidate document and the second candidate document comprises: Based on the first keyword similarity and the second keyword similarity, a target similar document of the document to be searched is determined from the first candidate document and the second candidate document.

12. The method according to claim 1, characterized in that The first similarity includes: first file name similarity, first file name face similarity, first text similarity, first text literal similarity and first keyword similarity; the second similarity includes: second file name similarity, second file name face similarity, second text similarity, second text literal similarity and second keyword similarity; The determining, based on the first similarity and the second similarity, a target similar document of the document to be searched from the first candidate document and the second candidate document comprises: Performing weighted calculation on the first file name similarity, the first file name literal similarity, the first text similarity, the first text literal similarity and the first keyword similarity to obtain a first weight; Performing weighted calculation on the second file name similarity, the second file name literal similarity, the second text similarity, the second text literal similarity and the second keyword similarity to obtain a second weight; sorting the first candidate document and the second candidate document according to the first weight and the second weight to obtain a sorting result; Based on the ranking result, a target similar document of the document to be searched is determined from the first candidate document and the second candidate document.

13. A document search device, characterized in that: include: An extraction module is configured to obtain a document to be searched, extract a target text content of the document to be searched and a first semantic vector of the target text content; A search module configured to search for a first candidate document based on the target text content, and to search for a second candidate document based on the first semantic vector; A designated feature data acquisition submodule is configured to acquire designated feature data of the document to be searched, the first candidate document and the second candidate document; a similarity determination submodule, configured to calculate a first similarity between the specified feature data of the document to be searched and the specified feature data of the first candidate document, and a second similarity between the specified feature data of the document to be searched and the specified feature data of the second candidate document; The determination submodule is configured to determine a target similar document of the document to be searched from the first candidate document and the second candidate document based on the first similarity and the second similarity.

14. A computing device, characterized in that include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the steps of the document search method according to any one of claims 1 to 12.

15. A computer-readable storage medium storing computer instructions, characterized in that: When the instruction is executed by a processor, the steps of the document search method described in any one of claims 1 to 12 are implemented.

Images