Document retrieval method and device, electronic equipment and storage medium

Abstract

The invention provides a document retrieval method and device, electronic equipment and a storage medium. The method comprises the steps that transverse rows of a clustering classification table are all classification libraries in a classification document library, vertical columns of the clustering classification table are all clustering clusters, and the value of the clustering classification table is the first contribution degree of each clustering cluster to each classification library; each classification library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; based on the clustering classification table, determining a classification document library to which the to-be-retrieved document belongs, and determining a clustering cluster to which each sentence in the to-be-retrieved document belongs; determining a first vector based on the to-be-retrieved document and the cluster, and determining a second vector based on all documents in the classification document library and the cluster; and determining a target document of the to-be-retrieved document based on a target similarity between the first vector and the second vector. Therefore, similarity matching of the to-be-retrieved document in each clustering direction is carried out, the accuracy of document retrieval is improved from the level of finer granularity, and the requirement for rapidly and accurately matching a long document is met.

Description

Technical Field

[0001] The present invention relates to the field of document retrieval technology, and in particular to a document retrieval method, device, electronic equipment and storage medium. Background Art

[0002] Text matching is a core problem in natural language understanding, and it can be applied to a large number of natural language processing tasks, such as information retrieval, question-answering systems, paraphrasing, dialogue systems, machine translation, etc. These natural language processing tasks can be abstracted into text matching problems to a large extent. For example, information retrieval can be reduced to the matching of search terms and document resources, question-answering systems can be reduced to the matching of questions and candidate answers, paraphrasing can be reduced to the matching of two synonymous sentences, dialogue systems can be reduced to the matching of the previous dialogue and the reply, and machine translation can be reduced to the matching of two languages.

[0003] Commonly used text retrieval methods include vocabulary-based full-text retrieval and semantic-based vector similarity retrieval. The first method is suitable for full-text retrieval of short texts, while the second method has large similarity errors and poor matching effects at the document-level text granularity, and slow calculation speed and low matching efficiency at the sentence-level text granularity. Summary of the invention

[0004] The present invention provides a document retrieval method, device, electronic device and storage medium, which are used to solve the defects that the vocabulary-based full-text retrieval method in the prior art is suitable for full-text retrieval of short texts, and the semantic-based vector similarity retrieval method has large similarity errors and poor matching effects at the document-level text granularity, and slow calculation speed and low matching efficiency at the sentence-level text granularity, so as to achieve fast and accurate matching of long documents.

[0005] The present invention provides a document retrieval method, comprising the following steps: Obtain documents to be retrieved, and construct a cluster classification table of the classification document library; the horizontal rows of the cluster classification table are the classification libraries in the classification document library, the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classification library; each classification library corresponds to multiple cluster clusters, and the multiple cluster clusters are obtained based on sentence granularity clustering; Based on the cluster classification table, determining the classification document library to which the document to be retrieved belongs, and determining the cluster to which each sentence in the document to be retrieved belongs; Determine a first vector based on the document to be retrieved and the cluster, and determine a second vector based on all documents in the classified document library and the cluster; Based on the target similarity between the first vector and the second vector, a target document of the document to be retrieved is determined.

[0006] According to a document retrieval method provided by the present invention, the step of determining the first contribution of each cluster to each classification library includes: Determine a first contribution value of each word in the classified document library to each category; Determine the first similarity between each cluster; Determining a second similarity between the classified document library and each cluster; Based on the first contribution value and / or the first similarity, a second contribution value of each cluster to each classification library is determined, and based on the second contribution value and the second similarity, the first contribution is determined.

[0007] According to a document retrieval method provided by the present invention, determining the first contribution value of each word in the classified document library to each category includes: Determine the reverse document frequency of each word and determine the discrimination of each word for each category; Based on the inverse document frequency and the discrimination, the first contribution value is determined.

[0008] A document retrieval method provided by the present invention further includes: Splitting the document to be retrieved by sentences to obtain a sentence set of the document to be retrieved; Determining a third similarity between each sentence in the sentence set and each cluster, and determining a first contribution of each cluster to each classification library; Based on the third similarity and the first contribution, the attribution degree of each sentence in the sentence set to the classified document library is determined, and based on the attribution degree, the target category of the document to be retrieved in the classified document library is determined.

[0009] According to a document retrieval method provided by the present invention, determining the target category of the document to be retrieved in the classified document library based on the attribution degree includes: Determining a target degree of attribution based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the document to be retrieved in the classified document library based on the target degree of attribution; Each sentence in the document to be retrieved is classified into the cluster to which the maximum similarity belongs among the clusters under the target category.

[0010] According to a document retrieval method provided by the present invention, the determining of the first vector based on the document to be retrieved and the clustering clusters includes: The first vector is determined based on the number of sentences in the document to be retrieved belonging to each cluster and the total number of sentences in the document to be retrieved.

[0011] The present invention also provides a document retrieval device, comprising the following modules: An acquisition unit is used to acquire documents to be retrieved and construct a cluster classification table of the classification document library; the horizontal rows of the cluster classification table are the classification libraries in the classification document library, the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classification library in the classification document library; each classification library corresponds to multiple cluster clusters, and the multiple cluster clusters are obtained based on sentence granularity clustering; A first determining unit, configured to determine, based on the cluster classification table, the classified document library to which the document to be retrieved belongs, and to determine the cluster to which each sentence in the document to be retrieved belongs; A second determining unit, configured to determine a first vector based on the document to be retrieved and the cluster, and to determine a second vector based on all documents in the classified document library and the cluster; The target document determination unit is used to determine the target document of the document to be retrieved based on the target similarity between the first vector and the second vector.

[0012] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the document retrieval methods described above is implemented.

[0013] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the document retrieval method described above is implemented.

[0014] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the document retrieval method described above is implemented.

[0015] The document retrieval method, device, electronic device and storage medium provided by the present invention determine the classified document library to which the document to be retrieved belongs based on the cluster classification table, and determine the cluster cluster to which each sentence in the document to be retrieved belongs, determine the first vector based on the document to be retrieved and the cluster cluster, and determine the second vector based on all documents and cluster clusters in the classified document library, and finally determine the target document of the document to be retrieved based on the target similarity between the first vector and the second vector. Thus, similarity matching of the document to be retrieved in each cluster direction is performed to obtain the document most relevant to the document to be retrieved, thereby improving the accuracy of document retrieval from a more fine-grained level, and the cluster classification table is used to reflect the corresponding relationship between each classification library and each cluster cluster in the multi-level corpus, so that the subsequent query based on the cluster classification table to the classified document library to which the document to be retrieved belongs and the cluster cluster to which each sentence in the document to be retrieved belongs is more accurate, meeting the needs of fast and accurate matching of long documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 It is a flowchart of the document retrieval method provided by the present invention.

[0018] Figure 2 It is a structural schematic diagram of the document retrieval device provided by the present invention.

[0019] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0022] In the related art, the term corpus in linguistics means a large amount of text, which is usually organized and has a certain format and markup. According to the characteristics of the corpus, it can be divided into monolingual corpus, bilingual corpus, parallel corpus, classified corpus, etc. According to the source of the corpus, it can be divided into written language corpus, spoken language corpus, composition corpus, learner corpus, ancient document corpus, etc. Corpus linguistics can perform grammatical and syntactic analysis on natural language, and can also study its relationship with other languages. The corpus was originally completed by hand, but now it is mainly completed automatically by electronic computers. At present, corpus linguistics mainly studies the collection, storage, retrieval, statistics, grammatical annotation, syntactic and semantic analysis of machine-readable natural language texts, as well as the application of corpora with the above functions in language teaching, language quantitative analysis, vocabulary research, collocation research, dictionary compilation, grammar research, language culture research, legal language research, work style analysis, natural language understanding and machine translation. Usually, document retrieval is based on corpus.

[0023] Figure 1 It is a flowchart of the document retrieval method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 and step 140 .

[0024] Step 110, obtain the document to be retrieved, and construct a cluster classification table of the classification document library; the horizontal rows of the cluster classification table are the classification libraries in the classification document library, the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to the classification libraries; each classification library corresponds to multiple cluster clusters, and the multiple cluster clusters are obtained based on sentence granularity clustering.

[0025] Specifically, a document to be retrieved can be obtained. The document to be retrieved here is a document that needs to be retrieved later. The document to be retrieved can be a document formed by text directly input by a user, or a document formed by text obtained by voice transcription of collected audio, or a document formed by text obtained by performing OCR (Optical Character Recognition) on an image obtained by an image acquisition device such as a scanner, mobile phone, or camera. The embodiment of the present invention does not make any specific limitations on this.

[0026] Then, a cluster classification table of the classified document library is constructed, wherein the horizontal rows of the cluster classification table are the classified libraries in the classified document library, the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classified library.

[0027] Among them, the classified document library is denoted as L, there are n categories in total, each classification library is denoted as Li, L={L1, L2, …,Ln}. Here, the document classification standard is implemented according to the major categories of the relevant industry classification standards and is divided into 97 industry categories.

[0028] Here, each classification library corresponds to multiple clusters, and the multiple clusters are obtained based on sentence granularity clustering, that is, for each classification document set Li of the classification document library L, clustering is performed according to sentence granularity to obtain Li's cluster cluster set Ci={Ci_1, Ci_2,…}.

[0029] It can be understood that a cluster is a collection of data objects with high similarity in a data set. These data objects have high similarity within the same cluster, but have great differences from data objects in other clusters.

[0030] Step 120: Based on the cluster classification table, determine the classified document library to which the document to be retrieved belongs, and determine the cluster to which each sentence in the document to be retrieved belongs.

[0031] Specifically, after the cluster classification table is constructed, the classification document library to which the document to be retrieved belongs can be queried based on the cluster classification table, and the cluster to which each sentence in the document to be retrieved belongs can be determined.

[0032] It can be understood that the horizontal rows of the clustering classification table are the classification libraries in the classification document library, the vertical columns of the clustering classification table are the cluster clusters, and the values ​​of the clustering classification table are the first contribution of each cluster cluster to each classification library. Furthermore, for all sentences of the document to be retrieved, the degrees of belonging of the sentences belonging to the same classification document library are added to obtain M, and the classification document library with the largest corresponding M value is queried in the clustering classification table as the classification to which the document to be retrieved belongs.

[0033] Then, each sentence of the document to be retrieved is assigned to the cluster with the maximum similarity among the clusters under the classification.

[0034] Step 130: determine a first vector based on the document to be retrieved and the cluster, and determine a second vector based on all documents in the classified document library and the cluster.

[0035] Specifically, the first vector may be determined based on the document to be retrieved and the clusters, that is, the first vector V may be determined based on the number of sentences in the document to be retrieved belonging to each cluster and the total number of sentences in the document to be retrieved.

[0036] Then, the second vector can be determined based on all documents and clusters in the classified document library, that is, the second vector W is determined based on the number of sentences in all documents in the classified document library belonging to each cluster and the total number of sentences in each document in the classified document library.

[0037] Step 140: Determine a target document of the document to be retrieved based on the target similarity between the first vector and the second vector.

[0038] Specifically, after the first vector and the second vector are determined, the target document of the document to be retrieved may be determined based on the target similarity between the first vector and the second vector.

[0039] Here, the target similarity is the text similarity, and the target similarity may be calculated by using methods such as cosine similarity and Pearson correlation coefficient, which is not specifically limited in the embodiment of the present invention.

[0040] Furthermore, after obtaining the target similarity, the document with the largest target similarity can be taken as the target document of the document to be retrieved.

[0041] It can be understood that the embodiment of the present invention constructs a multi-level classification corpus based on an existing classified document library based on relevant industry classification standards, and on this basis, performs similarity matching of the documents to be retrieved in each clustering direction to obtain the documents most relevant to the documents to be retrieved.

[0042] The method provided by the embodiment of the present invention determines the classified document library to which the document to be retrieved belongs based on the cluster classification table, determines the cluster cluster to which each sentence in the document to be retrieved belongs, determines the first vector based on the document to be retrieved and the cluster cluster, and determines the second vector based on all documents and cluster clusters in the classified document library. Finally, based on the target similarity between the first vector and the second vector, the target document of the document to be retrieved is determined. Thus, similarity matching is performed in each cluster direction of the document to be retrieved to obtain the document most relevant to the document to be retrieved, thereby improving the accuracy of document retrieval from a more fine-grained level. In addition, the cluster classification table is used to reflect the corresponding relationship between each classification library and each cluster cluster in the multi-level corpus, so that the subsequent query based on the cluster classification table to the classified document library to which the document to be retrieved belongs and the cluster cluster to which each sentence in the document to be retrieved belongs is more accurate, meeting the needs of fast and accurate matching of long documents.

[0043] Based on the above embodiment, the step of determining the first contribution of each cluster to each classification library includes: Step 210, determining a first contribution value of each word in the classified document library to each category; Step 220, determining a first similarity between each cluster; Step 230, determining a second similarity between the classified document library and each cluster; Step 240: determine a second contribution value of each cluster to each classification library based on the first contribution value and / or the first similarity, and determine the first contribution based on the second contribution value and the second similarity.

[0044] Specifically, the first contribution value of each word in the classified document library to each category may be determined first. Here, the first contribution value may be represented by {c1, c2,…, cn}.

[0045] Then, a variety of models may be used to determine the first similarity between the clusters, wherein the various models may include hierarchy-based, partition-based, and density-based clustering methods, which are not specifically limited in the embodiment of the present invention.

[0046] Furthermore, a second similarity between the classified document library and each cluster may be determined, wherein the second similarity may be calculated by using methods such as cosine similarity and Pearson correlation coefficient, which is not specifically limited in the embodiment of the present invention.

[0047] It is understandable that there are multiple clusters, and therefore, two similarity matrices can be obtained based on the first similarity determined based on each cluster and the second similarity determined based on the classified document library and each cluster.

[0048] Therefore, the second contribution value of each cluster to each classification library can be determined based on the first contribution value and / or the first similarity, and the first contribution degree can be determined based on the second contribution value and the second similarity, as follows: For the i-th classification library Li, the first contribution of the k-th cluster Cj_k under the j-th classification library to Li is calculated: 1) If i = j: Count all words under cluster Cj_k (repeat words and calculate independently), add up the first contribution values ​​of these words to the classification library Li, and get the second contribution value of Cj_k to Li; 2) If i != j: Based on the first similarity between clusters, among all clusters of Li, take the cluster Ci_l that is most similar to Cj_k, and its contribution value to Ci as the second contribution value of Cj_k to Li; 3) The first contribution of cluster Cj_k to classification library Li is calculated by multiplying the second similarity between Cj_k and Li by the second contribution value of Cj_k to Li.

[0049] Based on the above embodiment, the step 210 includes: Step 211, determining the reverse document frequency of each word, and determining the discrimination degree of each word for each category; Step 212: determine the first contribution value based on the reverse document frequency and the discrimination.

[0050] Specifically, 1) for word X, calculate its inverse document frequency (idf); Inverse document frequency is an indicator of word importance in NLP (Natural Language Processing). It is calculated by dividing the total number of documents in the corpus by the number of documents in which word X appears, and then taking the logarithm.

[0051] 2) Calculate the discrimination of vocabulary X for all categories {b1, b2,…, bn}: (1) Count the frequency of occurrence of word X in n classification libraries {f1, f2, …, fn}. The frequency of occurrence is the number of documents in the classification library that contain the word. The total frequency of occurrence is f=f1+f2+…+fn. (2) The number of documents in each classification library is {d1, d2, …, dn}, and the total number of documents is d = d1+d2+…+dn; (3) For the i-th classification library, the calculation method of the discrimination bi is: bi = log((d-di+1) / (f-fi+1)) / log(di+1 / fi+1)

[0052] 3) The contribution value ci of vocabulary X to the i-th category is: ci = idf * bi

[0053] Based on the above embodiment, it also includes: Step 310, splitting the document to be retrieved into sentences to obtain a sentence set of the document to be retrieved; Step 320, determining a third similarity between each sentence in the sentence set and each cluster, and determining a first contribution of each cluster to each classification library; Step 330, based on the third similarity and the first contribution, determine the attribution of each sentence in the sentence set to the classified document library, and based on the attribution, determine the target category of the document to be retrieved in the classified document library.

[0054] Specifically, the document to be retrieved can be split into sentences to obtain a sentence set S = {s1, s2, ...} of the document to be retrieved.

[0055] Then, the third similarity between each sentence in the sentence set and each cluster is determined, and the first contribution of each cluster to each classification library is determined.

[0056] Here, the third similarity may be calculated by using methods such as cosine similarity and Pearson correlation coefficient, which is not specifically limited in the embodiment of the present invention.

[0057] After the third similarity and the first contribution are obtained, the degree of belonging of each sentence in the sentence set to the classified document library may be determined based on the third similarity and the first contribution.

[0058] For example, the product of the third similarity between the sentence and the cluster and the first contribution between the cluster and the classified document library is taken as the attribution degree of each sentence in the sentence set to the classified document library.

[0059] Then, the value with the largest degree of attribution may be recorded as the m value, and the classified document library generating the m value may be marked as the target category of the document to be retrieved in the classified document library.

[0060] Based on the above embodiment, determining the target category of the document to be retrieved in the classified document library based on the attribution degree in step 330 includes: Step 331, determining a target attribution degree based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the document to be retrieved in the classified document library based on the target attribution degree; Step 332: Classify each sentence in the document to be retrieved into the cluster to which the maximum similarity belongs among the clusters under the target category.

[0061] Specifically, the m values ​​of all sentences in the sentence set belonging to the same classified document library are added together to obtain the target attribution degree M, and based on the target attribution degree, the target category of the document to be retrieved in the classified document library is determined, that is, the classified document library with the largest M value is used as the target category of the document to be retrieved in the classified document library.

[0062] Then, each sentence in the document to be retrieved can be classified into the cluster to which the maximum similarity belongs among the clusters under the target category.

[0063] Based on the above embodiment, determining the first vector based on the to-be-retrieved document and the cluster in step 130 includes: The first vector is determined based on the number of sentences in the document to be retrieved belonging to each cluster and the total number of sentences in the document to be retrieved.

[0064] Specifically, the first vector can be determined based on the number of sentences in each cluster of the document to be retrieved and the total number of sentences in the document to be retrieved, that is, each element in the first vector V is: The number of sentences in the document to be retrieved that belong to each cluster / the total number of sentences in the document to be retrieved.

[0065] The document retrieval device provided by the present invention is described below. The document retrieval device described below and the document retrieval method described above can be referenced to each other.

[0066] Based on any of the above embodiments, the present invention provides a document retrieval device. Figure 2 is a schematic diagram of the structure of the document retrieval device provided by the present invention, such as Figure 2 As shown, the device comprises: The acquisition unit 201 is used to acquire the document to be retrieved and construct a cluster classification table of the classification document library; the horizontal rows of the cluster classification table are the classification libraries in the classification document library, the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classification library in the classification document library; each classification library corresponds to multiple cluster clusters, and the multiple cluster clusters are obtained based on sentence granularity clustering; A first determining unit 202 is used to determine the classified document library to which the document to be retrieved belongs, and to determine the cluster to which each sentence in the document to be retrieved belongs, based on the cluster classification table; A second determining unit 203 is used to determine a first vector based on the document to be retrieved and the cluster, and to determine a second vector based on all documents in the classified document library and the cluster; The target document determination unit 204 is configured to determine a target document of the document to be retrieved based on the target similarity between the first vector and the second vector.

[0067] The device provided by the embodiment of the present invention determines the classified document library to which the document to be retrieved belongs based on the cluster classification table, determines the cluster cluster to which each sentence in the document to be retrieved belongs, determines the first vector based on the document to be retrieved and the cluster cluster, and determines the second vector based on all documents and cluster clusters in the classified document library. Finally, based on the target similarity between the first vector and the second vector, the target document of the document to be retrieved is determined. Thus, similarity matching is performed in each cluster direction of the document to be retrieved to obtain the document most relevant to the document to be retrieved, thereby improving the accuracy of document retrieval from a more fine-grained level. In addition, the cluster classification table is used to reflect the corresponding relationship between each classification library and each cluster cluster in the multi-level corpus, so that the subsequent query based on the cluster classification table to the classified document library to which the document to be retrieved belongs and the cluster cluster to which each sentence in the document to be retrieved belongs is more accurate, meeting the needs of fast and accurate matching of long documents.

[0068] Based on any of the above embodiments, the invention further includes a first contribution determination unit, wherein the first contribution determination unit is specifically configured to: A first contribution value determination unit, used to determine a first contribution value of each word in the classified document library to each category; A first similarity determination unit, used to determine a first similarity between each cluster; A second similarity determination unit, used to determine a second similarity between the classified document library and each cluster; The first contribution determination subunit is used to determine the second contribution value of each cluster to each classification library based on the first contribution value and / or the first similarity, and determine the first contribution based on the second contribution value and the second similarity.

[0069] Based on any of the foregoing embodiments, the first contribution value determining unit is specifically configured to: Determine the reverse document frequency of each word and determine the discrimination of each word for each category; Based on the inverse document frequency and the discrimination, the first contribution value is determined.

[0070] Based on any of the above embodiments, the further comprising a target category determination unit, wherein the target category determination unit is specifically used for: A splitting unit, used for splitting the document to be retrieved by sentence to obtain a sentence set of the document to be retrieved; a determination subunit, configured to determine a third similarity between each sentence in the sentence set and each cluster, and determine a first contribution of each cluster to each classification library; A target category determination subunit is used to determine the degree of attribution of each sentence in the sentence set to the classified document library based on the third similarity and the first contribution, and to determine the target category of the document to be retrieved in the classified document library based on the degree of attribution.

[0071] Based on any of the above embodiments, the target category determination subunit is specifically used for: Determining a target degree of attribution based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the document to be retrieved in the classified document library based on the target degree of attribution; Each sentence in the document to be retrieved is classified into the cluster to which the maximum similarity belongs among the clusters under the target category.

[0072] Based on any of the foregoing embodiments, the second determining unit 203 is specifically configured to: The first vector is determined based on the number of sentences in the document to be retrieved belonging to each cluster and the total number of sentences in the document to be retrieved.

[0073] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3As shown, the electronic device may include: a processor 310 , a communications interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communications interface 320 , and the memory 330 communicate with each other via the communication bus 340 . The processor 310 can call the logic instructions in the memory 330 to execute the document retrieval method, which includes: obtaining the document to be retrieved and constructing a cluster classification table of the classified document library; the horizontal rows of the cluster classification table are the classified libraries in the classified document library, and the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to the classified libraries; each classified library corresponds to multiple cluster clusters, and the multiple cluster clusters are obtained based on sentence granularity clustering; based on the cluster classification table, determine the classified document library to which the document to be retrieved belongs, and determine the cluster cluster to which each sentence in the document to be retrieved belongs; based on the document to be retrieved and the cluster cluster, determine a first vector, and based on all documents in the classified document library and the cluster cluster, determine a second vector; based on the target similarity between the first vector and the second vector, determine the target document of the document to be retrieved.

[0074] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the document retrieval method provided by the above-mentioned methods, which includes: obtaining a document to be retrieved and constructing a cluster classification table of a classified document library; the horizontal rows of the cluster classification table are each classified library in the classified document library, the vertical columns of the cluster classification table are each cluster cluster, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classified library; each classified library corresponds to multiple cluster clusters, and the multiple cluster clusters are obtained based on sentence granularity clustering; based on the cluster classification table, determine the classified document library to which the document to be retrieved belongs, and determine the cluster cluster to which each sentence in the document to be retrieved belongs; based on the document to be retrieved and the cluster cluster, determine a first vector, and based on all documents in the classified document library and the cluster cluster, determine a second vector; based on the target similarity between the first vector and the second vector, determine the target document of the document to be retrieved.

[0076] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the document retrieval method provided by the above-mentioned methods, the method comprising: obtaining a document to be retrieved, and constructing a cluster classification table of a classified document library; the horizontal rows of the cluster classification table are each classified library in the classified document library, the vertical columns of the cluster classification table are each cluster cluster, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classified library; each classified library corresponds to a plurality of cluster clusters, and the plurality of cluster clusters are obtained based on sentence granularity clustering; based on the cluster classification table, determining the classified document library to which the document to be retrieved belongs, and determining the cluster cluster to which each sentence in the document to be retrieved belongs; determining a first vector based on the document to be retrieved and the cluster cluster, and determining a second vector based on all documents in the classified document library and the cluster cluster; determining a target document of the document to be retrieved based on the target similarity between the first vector and the second vector.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A document retrieval method, characterized in that: include: Obtaining documents to be retrieved, and constructing a cluster classification table of the classification document library; the horizontal rows of the cluster classification table are the classification libraries in the classification document library, the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classification library; Each classification library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; Based on the cluster classification table, determining the classification document library to which the document to be retrieved belongs, and determining the cluster to which each sentence in the document to be retrieved belongs; Determine a first vector based on the document to be retrieved and the cluster, and determine a second vector based on all documents in the classified document library and the cluster; Based on the target similarity between the first vector and the second vector, a target document of the document to be retrieved is determined.

2. The document retrieval method according to claim 1, characterized in that: The step of determining the first contribution of each cluster to each classification library includes: Determine a first contribution value of each word in the classified document library to each category; Determine the first similarity between each cluster; Determining a second similarity between the classified document library and each cluster; Based on the first contribution value and / or the first similarity, a second contribution value of each cluster to each classification library is determined, and based on the second contribution value and the second similarity, the first contribution is determined.

3. The document retrieval method according to claim 2, characterized in that: The determining of the first contribution value of each word in the classified document library to each category includes: Determine the reverse document frequency of each word and determine the discrimination of each word for each category; Based on the inverse document frequency and the discrimination, the first contribution value is determined.

4. The document retrieval method according to any one of claims 1 to 3, characterized in that: Also includes: Splitting the document to be retrieved by sentences to obtain a sentence set of the document to be retrieved; Determining a third similarity between each sentence in the sentence set and each cluster, and determining a first contribution of each cluster to each classification library; Based on the third similarity and the first contribution, the attribution degree of each sentence in the sentence set to the classified document library is determined, and based on the attribution degree, the target category of the document to be retrieved in the classified document library is determined.

5. The document retrieval method according to claim 4, characterized in that: The step of determining the target category of the document to be retrieved in the classified document library based on the degree of attribution includes: Determining a target degree of attribution based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the document to be retrieved in the classified document library based on the target degree of attribution; Each sentence in the document to be retrieved is classified into the cluster to which the maximum similarity belongs among the clusters under the target category.

6. The document retrieval method according to any one of claims 1 to 3, characterized in that: The determining of the first vector based on the to-be-retrieved document and the clustering clusters includes: The first vector is determined based on the number of sentences in the document to be retrieved belonging to each cluster and the total number of sentences in the document to be retrieved.

7. A document retrieval device, characterized in that: include: An acquisition unit is used to acquire documents to be retrieved and construct a cluster classification table of the classification document library; the horizontal rows of the cluster classification table are the classification libraries in the classification document library, the vertical columns of the cluster classification table are the cluster clusters, and the values ​​of the cluster classification table are the first contribution of each cluster cluster to each classification library in the classification document library; Each classification library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; A first determining unit, configured to determine, based on the cluster classification table, the classified document library to which the document to be retrieved belongs, and to determine the cluster to which each sentence in the document to be retrieved belongs; A second determining unit, configured to determine a first vector based on the document to be retrieved and the cluster, and to determine a second vector based on all documents in the classified document library and the cluster; The target document determination unit is used to determine the target document of the document to be retrieved based on the target similarity between the first vector and the second vector.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the document retrieval method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the document retrieval method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the document retrieval method according to any one of claims 1 to 6 is implemented.

Images