Document search method, device, computer equipment and computer readable storage medium

Abstract

The present application belongs to the field of data search technology, and the present application provides a document search method, device, computer equipment and computer-readable storage medium. The present application obtains a search text, and obtains the word segmentation contained in the search text and the semantic features of the search text, and obtains the word segmentation index, title semantic features and text semantic features of each preset document based on a preset document library. According to the similarity between the word segmentation and the word segmentation index, and the semantic similarity between the search text semantic features and the title semantic features and the text semantic features, the semantic similarity between the search text and the preset document is calculated, and the preset document whose semantic similarity meets the preset semantic similarity condition is obtained as the search document corresponding to the search text. Since the word similarity and semantic similarity between the search text and each document are integrated, it can support document search based on natural language, improve the convenience and efficiency of document search, and improve the accuracy of document search.

Description

Technical Field

[0001] The present application relates to the field of data search technology, and in particular to a document search method, device, computer equipment and computer-readable storage medium. Background Art

[0002] With the continuous development of IoT-related technologies and platforms, the number of related technical manuals and documents has also increased rapidly. At present, the help documents of various IoT platform-related products and services are becoming increasingly complex and diversified, which makes it difficult for users to find suitable help documents and related tutorial documents. Therefore, providing an intelligent and effective document search system on various IoT platforms is an urgent need for both platform service providers and users. Existing common document search implementation solutions include:

[0003] 1. Index retrieval method. That is, full-text indexing is performed on the files and characters to be retrieved, and the index is quickly searched during the search to obtain the search position. This position is associated with the document path where the search term appears, and then the search results are returned to the user.

[0004] 2. Retrieval method based on short text vector similarity. The original data is split into multiple short texts to form a search text vector. The semantic understanding interface of the artificial intelligence platform is called to calculate the similarity between the search text and the index library text, and then the search result set is obtained.

[0005] Among them, the index retrieval method is just a mechanical match from the glyph, which will miss a lot of information that represents the same concept but with different text expressions. Since this method can only match words, when the descriptive text changes, it cannot be retrieved. The retrieval method based on short text vector similarity splits the original data into multiple short texts to form a search text vector, and calculates the similarity between the search text and the index library text by calling the semantic understanding interface of the artificial intelligence platform, thereby returning a list of search results. However, this method is difficult to completely match the document chapter-level semantic information, and cannot solve professional field problems well by calling the general language model of the artificial intelligence platform. Therefore, the existing document search methods all have the problem of low accuracy in document search. Summary of the invention

[0006] The present application provides a document search method, apparatus, computer device and computer-readable storage medium, which can solve the technical problem of low document search accuracy in traditional technologies.

[0007] In a first aspect, the present application provides a document search method, comprising: obtaining a search text, and obtaining search text semantic features corresponding to the search text and segmented words contained in the search text; based on a preset document library, obtaining a segmented word index for each of the preset documents, and obtaining text semantic features and title semantic features of each of the preset documents; calculating the semantic similarity between the search text and the preset documents based on the word similarity between the segmented words and the segmented word index, and the local semantic similarity corresponding to the search text semantic features and the title semantic features and the text semantic features; based on the semantic similarities corresponding to all the preset documents contained in the preset document library, obtaining a preset document whose semantic similarity meets a preset semantic similarity condition as the search document corresponding to the search text.

[0008] In a second aspect, the present application also provides a document search device, comprising: a first acquisition unit, used to acquire a search text, and acquire search text semantic features corresponding to the search text and segmented words contained in the search text; a second acquisition unit, used to acquire a segmented word index of each of the preset documents based on a preset document library, and acquire the main text semantic features and title semantic features of each of the preset documents; a first calculation unit, used to calculate the semantic similarity between the search text and the preset document based on the word similarity between the segmented words and the segmented word index, and the local semantic similarity corresponding to the search text semantic features and the title semantic features and the main text semantic features; a third acquisition unit, used to acquire a preset document whose semantic similarity meets a preset semantic similarity condition based on the semantic similarities corresponding to all the preset documents contained in the preset document library, as the search document corresponding to the search text.

[0009] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the document search method when executing the computer program.

[0010] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the document search method.

[0011] The present application provides a document search method, device, computer equipment and computer-readable storage medium. The present application obtains a search text, obtains the search text semantic features corresponding to the search text and the segmented words contained in the search text, obtains the segmented word index, title semantic features and text semantic features of each preset document based on a preset document library, calculates the semantic similarity between the search text and the preset document according to the word similarity between the segmented words and the segmented word index, and the local semantic similarity between the search text semantic features and the title semantic features and the text semantic features, and obtains the preset document whose semantic similarity meets the preset semantic similarity condition based on the semantic similarity of all the preset documents contained in the preset document library as the search document corresponding to the search text. Since the present application combines the word similarity between the search text and the segmented word index of each document, and the semantic similarity between the search text and the document title and the document body, the document with the highest semantic similarity to the search text is obtained as the search document corresponding to the search text, which can support document search based on natural language, improve the convenience and efficiency of document search, and improve the accuracy of document search. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A flowchart of a document search method provided in an embodiment of the present application;

[0014] Figure 2 A schematic diagram of the first sub-process of the document search method provided in an embodiment of the present application;

[0015] Figure 3 A schematic diagram of the framework flow of the document search method provided in the embodiment of the present application;

[0016] Figure 4 A schematic diagram of a second sub-process of the document search method provided in an embodiment of the present application;

[0017] Figure 5 A schematic diagram of a third sub-process of the document search method provided in an embodiment of the present application;

[0018] Figure 6 A schematic diagram of the fourth sub-process of the document search method provided in an embodiment of the present application;

[0019] Figure 7 A schematic block diagram of a document search device provided in an embodiment of the present application;

[0020] Figure 8 A schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0023] See also Figure 1 , Figure 1 A flowchart of a document search method provided in an embodiment of the present application. Figure 1 As shown, the method includes the following steps S11-S14:

[0024] S11. Obtain a search text, and obtain search text semantic features corresponding to the search text and segmented words contained in the search text, wherein the segmented words are semantic units obtained by performing segmentation processing on the search text.

[0025] Specifically, if the user needs to search for the required documents, a search text will generally be provided. For example, the search text is entered through the search box. The search text can be a word, a phrase, a sentence or a paragraph. If the search text is multiple words, phrases, sentences or a paragraph, the search text can be segmented based on a preset word segmentation word library and a preset word segmentation method, which can be an existing word segmentation method, to obtain the semantic units contained in the search text. The semantic units are word segments, which are used to describe the language units with independent meanings contained in the search text, so as to obtain the word features of the search text from the perspective of words. In addition, the search text is semantically understood through a semantic understanding algorithm to extract the search text semantic features corresponding to the semantics described by the search text. In particular, when the search text describes the semantics with a clear theme and complete meaning through a phrase, sentence or paragraph, the semantic features of the search text can be used to accurately describe the semantics corresponding to the search text.

[0026] S12. Based on the preset document library, obtain the word segmentation index of each preset document, and obtain the text semantic features and title semantic features of each preset document.

[0027] Specifically, the preset document library contains multiple preset documents. For each of the preset documents, the word segmentation processing can be performed in advance to obtain the word segmentation contained in each of the preset documents, and based on the word segmentation contained in each of the preset documents, the word segmentation index of each of the preset documents is constructed. The word segmentation index is a key word segmentation that can describe the main content of each of the preset documents and represent each of the preset documents. The word segmentation index can be a high-frequency word segmentation contained in each of the preset documents, and the high-frequency word segmentation is a word segmentation whose appearance frequency is greater than or equal to a preset word frequency threshold. For example, the BM25 index of each of the preset documents can be constructed based on word segmentation and BM25 retrieval algorithm, wherein BM25 is an algorithm used to evaluate the correlation between search terms and documents, and is an algorithm proposed based on a probabilistic retrieval model, so that the correlation between word segmentation and documents is evaluated based on BM25, and then the word segmentation with high correlation with the preset document is used as the index of the preset document.

[0028] At the same time, for the title and text of each of the preset documents, semantic understanding and feature extraction are performed based on the semantic understanding algorithm to obtain the title semantic features and text semantic features describing the preset document. The title semantic features are used to describe the features corresponding to the semantics of the document title of the preset document, and the text semantic features are used to describe the features corresponding to the semantics of the document text of the preset document, so that for each of the preset documents, the word segmentation index, title semantic features and text semantic features of the preset document are obtained.

[0029] Then, after the search text is obtained, when a document search is performed, the word segmentation index, title semantic features, and text semantic features of each of the preset documents are obtained based on the preset document library.

[0030] S13. Calculate the semantic similarity between the search text and the preset document based on the word similarity between the word segmentation and the word segmentation index, and the local semantic similarities between the semantic features of the search text and the semantic features of the title and the semantic features of the text.

[0031] Specifically, after obtaining the search text semantic features corresponding to the search text and the segmented words contained in the search text, and obtaining the segmented word index, title semantic features and text semantic features of each of the preset documents, the segmented words, the search text semantic features, the segmented word index, the title semantic features and the text semantic features are respectively vectorized to facilitate the calculation of each similarity. At the same time, since the word segmentation and the word segmentation index both describe the search text and the preset document from the perspective of word features, the search text semantic features and the title semantic features and the text semantic features all describe the search text and the preset document from the perspective of semantic features, therefore, the word similarity between the word segmentation and the word segmentation index can be calculated, and the word similarity is used to describe the word similarity between the word segmentation of the search text and the word segmentation of the word segmentation index. At the same time, the local semantic similarity corresponding to the search text semantic features and the title semantic features and the text semantic features is calculated. The local semantic similarity includes the title semantic similarity and the text semantic similarity. The title semantic similarity is used to describe the semantic similarity between the search text and the document title contained in the preset document, and the text semantic similarity is used to describe the semantic similarity between the search text and the document body contained in the preset document. Then, the word similarity and the local semantic similarity are combined to obtain the semantic similarity corresponding to the comprehensive and overall similarity between the search text and the preset document.

[0032] S14. Based on the semantic similarities corresponding to all the preset documents contained in the preset document library, a preset document whose semantic similarity meets a preset semantic similarity condition is obtained as a search document corresponding to the search text.

[0033] Specifically, after obtaining the semantic similarity between the search text and each of the preset documents, based on all the semantic similarities corresponding to all the preset documents contained in the preset document library, the preset document with the highest semantic similarity is screened out as the search document corresponding to the search text, and the search document is displayed to be provided to the terminal user corresponding to the search text.

[0034] In an embodiment of the present application, a search text is obtained, and search text semantic features corresponding to the search text and segmented words contained in the search text are obtained. Based on a preset document library, a segmented word index, a title semantic feature and a text semantic feature of each preset document are obtained. According to the word similarity between the segmented word and the segmented word index, and the local semantic similarity corresponding to each of the search text semantic feature and the title semantic feature and the text semantic feature, the semantic similarity between the search text and the preset document is calculated. Based on the semantic similarities corresponding to all the preset documents contained in the preset document library, a preset document whose semantic similarity meets a preset semantic similarity condition is obtained, for example, a preset document with the highest semantic similarity is obtained, or a plurality of preset documents whose semantic similarity is greater than or equal to a preset semantic similarity threshold are obtained. If the preset documents that meet the preset semantic similarity condition include multiple documents, the multiple preset documents can also be sorted in descending order of the semantic similarity, so that the user can be given more search documents as references, as search documents corresponding to the search text. By combining the word similarity between the search text and the main word segments of each document, and the semantic similarity between the search text and the document title and document body, the document with the highest semantic similarity to the search text is obtained. This document is used as the search document corresponding to the search text, which can support document search based on natural language, improve the convenience and efficiency of document search, and improve the accuracy of document search.

[0035] In one embodiment, the obtaining the search text semantic feature corresponding to the search text includes:

[0036] Based on a preset semantic feature extraction model of a deep bidirectional converter, semantic features are extracted from the search text to obtain search text semantic features corresponding to the search text.

[0037] Among them, the deep bidirectional transformer is a model that performs deep bidirectional pre-training by adjusting the context in all layers in the model. For example, the BERT model implements deep bidirectional pre-training of context based on Transformer. The full name of BERT is Bidirectional Encoder Representation from Transformers, which is a pre-training model.

[0038] Specifically, a semantic feature extraction model based on a deep bidirectional converter is pre-constructed, where the semantic feature extraction model may be a general model based on a deep bidirectional converter, and the semantic feature extraction model is trained to obtain a preset semantic feature extraction model based on a deep bidirectional converter. By using the semantic feature extraction model based on a deep bidirectional converter, the semantic feature information of the semantic level can be fully extracted, and the accuracy of semantic feature extraction can be improved, thereby improving the accuracy of document search.

[0039] For further information, see Figure 2 , Figure 2 The first sub-process diagram of the document search method provided in the embodiment of the present application is as follows: Figure 2 As shown, in this embodiment, when training the semantic feature extraction model based on the deep bidirectional converter, the following steps may be included:

[0040] S21, using a preset first number of first corpus training samples to preliminarily pre-train a preset general model based on a deep bidirectional converter to obtain a first pre-training model, wherein the first corpus training samples are corpora that have not been manually annotated;

[0041] S22. Use a preset second number of second corpus training samples to pre-train the first pre-training model again to obtain a preset semantic feature extraction model based on a deep bidirectional converter, wherein the second corpus training samples are manually annotated corpuses, and the preset first number is greater than the preset second number.

[0042] Specifically, when training a semantic feature extraction model based on a deep bidirectional converter, the first step can be to use a large number of unlabeled first corpus training samples to preliminarily pre-train the preset general model based on the deep bidirectional converter to obtain a first pre-trained model, and then use a limited amount of manually labeled second corpus training samples to pre-train the first pre-trained model again, thereby obtaining a preset semantic feature extraction model based on a deep bidirectional converter, improving the accuracy of the preset semantic feature extraction model, saving manpower for manually labeling the corpus, and improving the training efficiency of the preset semantic feature extraction model. For example, in the field of Internet of Things, using a general BERT model as a semantic feature extraction model, in the first step, a large number of unlabeled first corpus training samples in the field of Internet of Things can be used to pre-train the general BERT model, so that the BERT model can initially learn the feature extraction capabilities of Internet of Things-related proprietary vocabulary and sentences based on the deep bidirectional converter. In the second step, a limited amount of manually annotated question-answer pairs in the field of Internet of Things are used as second corpus training samples to further pre-train the BERT model, so that the BERT model can be further fine-tuned, so that the BERT model can perform similarity modeling on questions and corresponding answers, and learn the potential mapping relationship between questions and their corresponding answers. The BERT model pre-trained in the first and second steps, as the final preset semantic feature extraction BERT model based on the deep bidirectional converter, can use a limited amount of manually annotated corpus training samples to improve the accuracy of the preset semantic feature extraction BERT model, save manpower for manually annotating the corpus, and improve the pre-training efficiency of the preset semantic feature extraction BERT model.

[0043] See also Figure 3 , Figure 3 A schematic diagram of the framework flow of the document search method provided in the embodiment of the present application, such as Figure 3 As shown, after obtaining the search text, a preset semantic feature extraction model based on a deep bidirectional converter is used to perform semantic understanding on the search text to extract semantic features of the search text, thereby obtaining the search text semantic features corresponding to the search text, which can improve the accuracy of the semantic feature extraction of the search text, and further improve the accuracy of document search.

[0044] Please continue reading Figure 3 When extracting semantic features from the title and text of each preset document in the preset document library, the preset semantic feature extraction model based on the deep bidirectional converter can also be used to understand the semantics to extract semantic features, thereby respectively obtaining the title semantic features corresponding to the title of the preset document and the text semantic features corresponding to the text of the preset document, which can also improve the accuracy of the extraction of the title semantic features and the text semantic features, thereby improving the accuracy of document search.

[0045] See also Figure 4 , Figure 4 A second sub-process diagram of the document search method provided in the embodiment of the present application is shown in FIG. Figure 4 As shown, in this embodiment, obtaining the text semantic features of each of the preset documents includes:

[0046] S121, obtaining the text of each of the preset documents, and dividing the text into different sentences in sequence according to the order of sentences contained in the text and using punctuation marks contained in the text as segmentation marks, to obtain text sentences contained in the text;

[0047] S122. Based on the preset semantic feature extraction model, extract semantic features from the text sentences to obtain text semantic features corresponding to the text.

[0048] Specifically, when extracting the text semantic features corresponding to the text contained in each of the preset documents, the text of each of the preset documents can be obtained, and according to the sentence order of the sentences contained in the text and taking the punctuation marks contained in the text as segmentation marks, the text is sequentially segmented into different sentences to obtain the text sentences contained in the text, thereby extracting the text sentences corresponding to the sentences contained in the text in a sliding window manner for the long text corresponding to the text, and then performing semantic feature extraction on the text sentences based on the preset semantic feature extraction model, extracting the semantic features of each of the text sentences, thereby obtaining the text semantic features corresponding to the text, and realizing the long text part of the text. The sliding window semantic feature extraction of the sentences can make the semantic feature extraction of the document text more comprehensive and accurate, thereby improving the accuracy and performance of document search.

[0049] See also Figure 5 , Figure 5 A third sub-process diagram of the document search method provided in the embodiment of the present application is shown in FIG. Figure 5 As shown, in this embodiment, according to the word similarity between the segmentation and the segmentation index, and the local semantic similarity corresponding to each other between the semantic feature of the search text and the semantic feature of the title and the semantic feature of the text, the semantic similarity between the search text and the preset document is calculated, including:

[0050] S131, calculating the index word similarity corresponding to the word similarity between the segmentation and the segmentation index;

[0051] S132, calculating the title semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the title semantic feature;

[0052] S133, calculating the text semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the text semantic feature;

[0053] S134: linearly sum the index term similarity, the title semantic similarity and the text semantic similarity to obtain the semantic similarity between the search text and the preset document.

[0054] Specifically, in order to more accurately count the similarity between the search text and the preset document, the similarity between words and the similarity between semantics are measured to obtain a more accurate similarity between the search text and the preset document. The index word similarity corresponding to the word similarity between the word segmentation and the word segmentation index can be first calculated. The word segmentation and the word segmentation index can be vectorized to obtain their corresponding word vectors, and then the similarity between the word vectors can be calculated. For example, the index word similarity is calculated using cosine similarity to obtain the index word similarity between the word segmentation and the word segmentation index. Similarly, the title semantic similarity corresponding to the local semantic similarity between the semantic features of the search text and the semantic features of the title can be calculated, and the text semantic similarity corresponding to the local semantic similarity between the semantic features of the search text and the semantic features of the text can be calculated. Since the index term similarity, the title semantic similarity and the text semantic similarity are respectively values ​​between [1, 1], the index term similarity, the title semantic similarity and the text semantic similarity are linearly summed. The index term similarity, the title semantic similarity and the text semantic similarity can be directly added together, or the index term similarity, the title semantic similarity and the text semantic similarity can be respectively set with corresponding weights to reflect the different roles and importance of each part in calculating the similarity between the search text and the preset document, thereby improving the accuracy of calculating the similarity, and then adding them together to obtain the semantic similarity between the search text and the preset document.

[0055] See also Figure 6 , Figure 6 The fourth sub-process diagram of the document search method provided in the embodiment of the present application is as follows: Figure 6 As shown, in this embodiment, obtaining the word segmentation index of each preset document includes:

[0056] S61, obtaining each of the preset documents, and performing word segmentation processing on the preset documents to obtain document word segments contained in the preset documents;

[0057] S62, based on the BM25 algorithm, counting the relevance score between each document segment and the preset document;

[0058] S63, determining whether the relevance score is greater than or equal to a preset relevance score threshold;

[0059] S64: If the relevance score is greater than or equal to a preset relevance score threshold, use the document segmentation as the target document segmentation, and construct a segmentation index of the preset document based on the target document segmentation to obtain the segmentation index of the preset document;

[0060] S65: If the relevance score is less than a preset relevance score threshold, the document segmentation is not used as a target document segmentation.

[0061] Specifically, in order to more accurately obtain the keywords describing each of the preset documents, and to establish the index of the preset documents through the keywords, thereby improving the accuracy of document search, each of the preset documents is first obtained, and the preset documents are segmented to obtain the document segmentation contained in the preset documents, thereby obtaining the document segmentation from the preset documents. Then, based on all the document segmentations contained in the preset documents, based on the BM25 algorithm, the relevance score of each document segmentation and the preset document is counted, thereby comparing and analyzing the document segmentation with the preset document, and then judging whether the relevance score is greater than or equal to the preset relevance score threshold. If the relevance score is greater than or equal to the preset relevance score threshold, the document segmentation is used as the target document segmentation, and the index of the preset document is constructed based on the target document segmentation, thereby obtaining the segmentation index of the preset document. If the relevance score is less than the preset relevance score threshold, the document segmentation is not used as the target document segmentation. Since the BM25 algorithm is an algorithm used to evaluate the correlation between the document segmentation and the preset document, and the correlation between the document segmentation and the preset document is quantitatively described by the correlation score, the higher the correlation score, the greater the correlation between the document segmentation and the preset document, and the more the document segmentation can describe the preset document, thereby better describing the main content of the preset document. The preset document segmentation can be used as the main keyword of the preset document to obtain the document segmentation from the preset document and then use it to compare with the preset document, thereby obtaining the target key segmentation of the preset document, which is used to construct the index of the preset document and obtain the segmentation index of the preset document, which can improve the accuracy of the index of the preset document and further improve the accuracy of document search.

[0062] It should be noted that the document search method described in the above embodiments can recombine the technical features contained in different embodiments as needed to obtain a combined implementation plan, but all of them are within the protection scope required by this application.

[0063] See also Figure 7 , Figure 7 A schematic block diagram of a document search device provided in an embodiment of the present application. Corresponding to the document search method described above, an embodiment of the present application also provides a document search device. Figure 7 As shown, the document search device includes a unit for executing the document search method described above. The document search device can be configured in a computer device such as a server to implement a document search engine. Figure 7 The document search device 70 includes a first acquisition unit 71, a second acquisition unit 72, a first calculation unit 73 and a third acquisition unit 74.

[0064] The first acquisition unit 71 is used to acquire the search text, and acquire the search text semantic features corresponding to the search text and the segmented words contained in the search text;

[0065] The second acquisition unit 72 is used to acquire the word segmentation index of each of the preset documents based on the preset document library, and acquire the text semantic features and title semantic features of each of the preset documents;

[0066] A first calculation unit 73 is used to calculate the semantic similarity between the search text and the preset document according to the word similarity between the word segmentation and the word segmentation index, and the local semantic similarity between the search text semantic feature and the title semantic feature and the text semantic feature respectively corresponding to each other;

[0067] The third acquisition unit 74 is used to acquire, based on the semantic similarities corresponding to all the preset documents included in the preset document library, a preset document whose semantic similarity meets a preset semantic similarity condition as a search document corresponding to the search text.

[0068] In one embodiment, the first acquisition unit 71 is specifically configured to extract semantic features from the search text based on a preset semantic feature extraction model of a deep bidirectional converter to obtain search text semantic features corresponding to the search text.

[0069] In one embodiment, the second obtaining unit 72 includes:

[0070] A segmentation subunit is used to obtain the text of each of the preset documents, and according to the sentence order of the sentences contained in the text, use the punctuation marks contained in the text as segmentation marks, and sequentially segment the text into different sentences to obtain the text sentences contained in the text;

[0071] The first extraction subunit is used to extract semantic features from the text sentences based on the preset semantic feature extraction model to obtain text semantic features corresponding to the text.

[0072] In one embodiment, the document search device 70 further includes:

[0073] A first pre-training unit is used to perform preliminary pre-training on a preset general model based on a deep bidirectional converter by using a preset first number of first corpus training samples to obtain a first pre-training model, wherein the first corpus training samples are corpuses that are not manually annotated;

[0074] The second pre-training unit is used to use a preset second number of second corpus training samples to pre-train the first pre-training model again to obtain a preset semantic feature extraction model based on a deep bidirectional converter, wherein the second corpus training samples are manually annotated corpuses, and the preset first number is greater than the preset second number.

[0075] In one embodiment, the preset semantic feature extraction model is a BERT model.

[0076] In one embodiment, the first calculation unit 73 includes:

[0077] A first calculation subunit, configured to calculate an index word similarity corresponding to a word similarity between the segmented word and the segmented word index;

[0078] A second calculation subunit is used to calculate the title semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the title semantic feature;

[0079] A third calculation subunit is used to calculate the text semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the text semantic feature;

[0080] The summing subunit is used to linearly sum the index word similarity, the title semantic similarity and the text semantic similarity to obtain the semantic similarity between the search text and the preset document.

[0081] In one embodiment, the second obtaining unit 72 includes:

[0082] A first acquisition subunit is used to acquire each of the preset documents and perform word segmentation processing on the preset documents to obtain document word segments contained in the preset documents;

[0083] A statistical subunit, used for counting the relevance score between each document segment and the preset document based on the BM25 algorithm;

[0084] A determination subunit, used to determine whether the relevance score is greater than or equal to a preset relevance score threshold;

[0085] A construction subunit is used to, if the relevance score is greater than or equal to a preset relevance score threshold, use the document segmentation as the target document segmentation, and construct a segmentation index of the preset document based on the target document segmentation to obtain the segmentation index of the preset document.

[0086] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned document search device and each unit can refer to the corresponding description in the aforementioned method embodiment, and for the convenience and brevity of description, it will not be repeated here.

[0087] At the same time, the division and connection method of each unit in the above-mentioned document search device is only for illustrative purposes. In other embodiments, the document search device can be divided into different units as needed, and the units in the document search device can also be connected in different orders and methods to complete all or part of the functions of the above-mentioned document search device.

[0088] The document search device can be implemented in the form of a computer program. Figure 8 Runs on the computer device shown.

[0089] See also Figure 8 , Figure 8 1 is a schematic block diagram of a computer device provided in an embodiment of the present application. The computer device 500 may be a computer device such as a server, or a component or part in other devices.

[0090] See also Figure 8 The computer device 500 includes a processor 502, a memory and a network interface 505 connected via a system bus 501, wherein the memory may include a non-volatile storage medium 503 and an internal memory 504, and the memory may also be a volatile storage medium.

[0091] The non-volatile storage medium 503 can store an operating system 5031 and a computer program 5032. When the computer program 5032 is executed, the processor 502 can execute the document search method described above.

[0092] The processor 502 is used to provide computing and control capabilities to support the operation of the entire computer device 500 .

[0093] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute the above-mentioned document search method.

[0094] The network interface 505 is used to communicate with other devices over the network. Figure 8The structure shown in the figure is only a block diagram of a part of the structure related to the present application scheme, and does not constitute a limitation on the computer device 500 to which the present application scheme is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. For example, in some embodiments, the computer device may only include a memory and a processor. In such an embodiment, the structure and function of the memory and the processor are the same as those of the present application scheme. Figure 8 The embodiments shown are consistent and will not be described again here.

[0095] The processor 502 is used to run a computer program 5032 stored in a memory to implement the following steps: obtaining a search text, and obtaining search text semantic features corresponding to the search text and segmented words contained in the search text; based on a preset document library, obtaining a segmented word index for each of the preset documents, and obtaining text semantic features and title semantic features of each of the preset documents; calculating the semantic similarity between the search text and the preset document based on the word similarity between the segmented words and the segmented word index, and the local semantic similarity corresponding to the search text semantic features and the title semantic features and the text semantic features; based on the semantic similarities corresponding to all the preset documents contained in the preset document library, obtaining a preset document whose semantic similarity meets the preset semantic similarity condition as the search document corresponding to the search text.

[0096] In one embodiment, the processor 502 implements the following steps when obtaining the search text semantic feature corresponding to the search text:

[0097] Based on a preset semantic feature extraction model of a deep bidirectional converter, semantic features are extracted from the search text to obtain search text semantic features corresponding to the search text.

[0098] In one embodiment, the processor 502 implements the following steps when acquiring the text semantic features of each of the preset documents:

[0099] Obtaining the text of each of the preset documents, dividing the text into different sentences in sequence according to the order of sentences contained in the text and taking punctuation marks contained in the text as segmentation marks, to obtain text sentences contained in the text;

[0100] Based on the preset semantic feature extraction model, semantic features are extracted from the text sentences to obtain text semantic features corresponding to the text.

[0101] In one embodiment, before implementing the preset semantic feature extraction model based on the deep bidirectional converter and extracting semantic features from the search text to obtain the search text semantic features corresponding to the search text, the processor 502 further implements the following steps:

[0102] Using a preset first number of first corpus training samples, preliminarily pre-training a preset general model based on a deep bidirectional converter is performed to obtain a first pre-training model, wherein the first corpus training samples are corpora that have not been manually annotated;

[0103] The first pre-trained model is pre-trained again using a preset second number of second corpus training samples to obtain a preset semantic feature extraction model based on a deep bidirectional converter, wherein the second corpus training samples are manually annotated corpora, and the preset first number is greater than the preset second number.

[0104] In one embodiment, when the processor 502 implements the preset semantic feature extraction model, the preset semantic feature extraction model is a BERT model.

[0105] In one embodiment, the processor 502 specifically implements the following steps when calculating the semantic similarity between the search text and the preset document based on the word similarity between the word segmentation and the word segmentation index, and the local semantic similarities corresponding to the semantic features of the search text and the semantic features of the title and the semantic features of the text:

[0106] Calculating the index word similarity corresponding to the word similarity between the segmentation and the segmentation index;

[0107] Calculating the title semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the title semantic feature;

[0108] Calculating the text semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the text semantic feature;

[0109] The index word similarity, the title semantic similarity and the text semantic similarity are linearly summed to obtain the semantic similarity between the search text and the preset document.

[0110] In one embodiment, when the processor 502 implements the acquisition of the word segmentation index of each preset document, the processor 502 specifically implements the following steps:

[0111] Obtain each of the preset documents, and perform word segmentation processing on the preset documents to obtain document word segments contained in the preset documents;

[0112] Based on the BM25 algorithm, the relevance score between each document segment and the preset document is counted;

[0113] Determining whether the relevance score is greater than or equal to a preset relevance score threshold;

[0114] If the relevance score is greater than or equal to a preset relevance score threshold, the document segmentation is used as the target document segmentation, and a segmentation index of the preset document is constructed based on the target document segmentation to obtain the segmentation index of the preset document.

[0115] It should be understood that in the embodiment of the present application, the processor 502 may be a central processing unit (CPU), and the processor 502 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0116] Those skilled in the art will appreciate that all or part of the processes in the method for implementing the above embodiment can be completed by a computer program, which can be stored in a computer-readable storage medium. The computer program is executed by at least one processor in the computer system to implement the process steps of the embodiment of the above method.

[0117] Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, and the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the following steps:

[0118] A computer program product, when running on a computer, enables the computer to execute the steps of the document search method described in the above embodiments.

[0119] The computer-readable storage medium may be an internal storage unit of the aforementioned device, such as a hard disk or memory of the device. The computer-readable storage medium may also be an external storage device of the device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of the device and an external storage device.

[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described equipment, devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0121] The storage medium is a physical, non-transient storage medium, for example, it can be a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk, etc., which can store computer programs.

[0122] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0123] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of each unit is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0124] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs. The units in the device of the embodiment of the present application can be combined, divided and deleted according to actual needs. In addition, the functional units in the various embodiments of the present application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0125] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for an electronic device (which can be a personal computer, terminal, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0126] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A document search method, characterized in that: The method comprises: Obtaining a search text, and obtaining a search text semantic feature corresponding to the search text and a segmentation word contained in the search text; Based on the preset document library, obtain the word segmentation index of each preset document, and obtain the text semantic features and title semantic features of each preset document, wherein the word segmentation index includes key words and high-frequency words contained in each preset document; Calculate the semantic similarity between the search text and the preset document according to the word similarity between the word segmentation and the word segmentation index, and the local semantic similarity between the semantic feature of the search text and the semantic feature of the title and the semantic feature of the text; Based on the semantic similarities corresponding to all the preset documents contained in the preset document library, a preset document whose semantic similarity meets a preset semantic similarity condition is obtained as a search document corresponding to the search text; The obtaining of the search text semantic feature corresponding to the search text includes: Based on a preset semantic feature extraction model of a deep bidirectional converter, the search text is subjected to semantic feature extraction to obtain search text semantic features corresponding to the search text; The obtaining of the text semantic features of each of the preset documents includes: Obtaining the text of each of the preset documents, dividing the text into different sentences in sequence according to the order of sentences contained in the text and taking punctuation marks contained in the text as segmentation marks, to obtain text sentences contained in the text; Based on the preset semantic feature extraction model, semantic features are extracted from the text sentences to obtain text semantic features corresponding to the text.

2. The document search method according to claim 1, characterized in that: The preset semantic feature extraction model based on the deep bidirectional converter extracts semantic features from the search text to obtain the search text semantic features corresponding to the search text, including: Using a preset first number of first corpus training samples, preliminarily pre-training a preset general model based on a deep bidirectional converter is performed to obtain a first pre-training model, wherein the first corpus training samples are corpora that have not been manually annotated; The first pre-trained model is pre-trained again using a preset second number of second corpus training samples to obtain a preset semantic feature extraction model based on a deep bidirectional converter, wherein the second corpus training samples are manually annotated corpora, and the preset first number is greater than the preset second number.

3. The document search method according to any one of claims 1-2, characterized in that: The preset semantic feature extraction model is the BERT model.

4. The document search method according to claim 1, characterized in that: Calculating the semantic similarity between the search text and the preset document according to the word similarity between the word segmentation and the word segmentation index, and the local semantic similarities between the semantic features of the search text and the semantic features of the title and the semantic features of the text, including: Calculating the index word similarity corresponding to the word similarity between the segmentation and the segmentation index; Calculating the title semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the title semantic feature; Calculating the text semantic similarity corresponding to the local semantic similarity between the search text semantic feature and the text semantic feature; The index word similarity, the title semantic similarity and the text semantic similarity are linearly summed to obtain the semantic similarity between the search text and the preset document.

5. The document search method according to claim 1, characterized in that: The step of obtaining the word segmentation index of each preset document includes: Obtain each of the preset documents, and perform word segmentation processing on the preset documents to obtain document word segments contained in the preset documents; Based on the BM25 algorithm, the relevance score between each document segment and the preset document is counted; Determining whether the relevance score is greater than or equal to a preset relevance score threshold; If the relevance score is greater than or equal to a preset relevance score threshold, the document segmentation is used as the target document segmentation, and a segmentation index of the preset document is constructed based on the target document segmentation to obtain the segmentation index of the preset document.

6. A document search device, characterized in that: include: A first acquisition unit is used to acquire a search text, and acquire a search text semantic feature corresponding to the search text and a segmentation word included in the search text; A second acquisition unit is used to acquire a word segmentation index of each of the preset documents based on a preset document library, and acquire a text semantic feature and a title semantic feature of each of the preset documents, wherein the word segmentation index includes key words and high-frequency words contained in each of the preset documents; A first calculation unit is used to calculate the semantic similarity between the search text and the preset document according to the word similarity between the segmentation and the segmentation index, and the local semantic similarities corresponding to the semantic features of the search text and the semantic features of the title and the semantic features of the text; A third acquisition unit is used to acquire, based on the semantic similarities corresponding to all the preset documents contained in the preset document library, a preset document whose semantic similarity meets a preset semantic similarity condition as a search document corresponding to the search text; The first acquisition unit is specifically configured to extract semantic features from the search text based on a preset semantic feature extraction model of a deep bidirectional converter to obtain a search text semantic feature corresponding to the search text; The second acquiring unit includes: A segmentation subunit is used to obtain the text of each of the preset documents, and according to the sentence order of the sentences contained in the text, use the punctuation marks contained in the text as segmentation marks, and sequentially segment the text into different sentences to obtain the text sentences contained in the text; The first extraction subunit is used to extract semantic features from the text sentences based on the preset semantic feature extraction model to obtain text semantic features corresponding to the text.

7. A computer device, characterized in that: The computer device comprises a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program to execute the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 can be implemented.

Images