Corpus construction method and device, electronic equipment and storage medium

Abstract

The invention provides a corpus construction method and device, electronic equipment and a storage medium, and the method comprises the steps: splitting an original document according to sentences, and obtaining a sentence set of the original document; determining a first similarity between each sentence in the sentence set and each cluster, and determining a first contribution degree of each cluster to the classification library; each classification library of the classification document library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; determining the attribution degree of each sentence in the sentence set and the classification document library based on the first similarity and the first contribution degree; determining a target affiliation degree based on the affiliation degrees of all sentences in the sentence set belonging to the same classification document library, and determining a target category of the original document in the classification document library based on the target affiliation degree; and classifying the sentences in the original document into the cluster to which the maximum similarity belongs in the clusters under the target category. In the process, original documents are classified from a finer-grained level, and the accuracy of corpus construction is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of text data processing, and in particular to a method, device, electronic device and storage medium for constructing corpus. Background Art

[0002] In linguistics, the term corpus refers to a large amount of text, usually organized and with a set format and tags. According to the characteristics of the corpus, it can be divided into monolingual corpus, bilingual corpus, parallel corpus, classified corpus, etc. According to the source of the corpus, it can be divided into written language corpus, spoken language corpus, composition corpus, learner corpus, ancient document corpus, etc.

[0003] Corpus linguistics can analyze the grammar and syntax of natural language, and can also study its relationship with other languages. Corpus was originally completed by hand, but now it is mainly completed automatically by electronic computers. At present, corpus linguistics mainly studies the collection, storage, retrieval, statistics, grammatical annotation, syntactic and semantic analysis of machine-readable natural language texts, as well as the application of corpora with the above functions in language teaching, quantitative language analysis, vocabulary research, collocation research, dictionary compilation, grammar research, language culture research, legal language research, work style analysis, natural language understanding and machine translation.

[0004] For the construction of classified corpus, a common method is to build a text classification model based on existing classified documents. For new documents, the text classification model is used to classify them into a certain category. With the deepening application of large language models in various vertical industries, there is a demand for more fine-grained classified corpus. Summary of the invention

[0005] The present invention provides a corpus construction method, device, electronic device and storage medium, which are used to solve the defects in the prior art that the classification granularity of classifying new documents into a certain category through a text classification model is coarse and inaccurate.

[0006] The present invention provides a method for constructing a corpus, comprising the following steps: Acquire an original document and a classified document library, and split the original document into sentences to obtain a sentence set of the original document; Determine a first similarity between each sentence in the sentence set and each cluster, and determine a first contribution of each cluster to a classification library of the classified document library; each classification library of the classified document library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; Determining the degree of belonging of each sentence in the sentence set to the classified document library based on the first similarity and the first contribution; Determining a target degree of attribution based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the original document in the classified document library based on the target degree of attribution; Each sentence in the original document is classified into the cluster to which the maximum similarity belongs among the clusters under the target category.

[0007] According to a corpus construction method provided by the present invention, the determining of the first contribution of each cluster to the classification library of the classification document library includes: Determining a target contribution value of each cluster to the classification library of the classification document library; The first contribution is determined based on the second similarity between each cluster and the classification library, and the target contribution value.

[0008] According to a corpus construction method provided by the present invention, the step of determining the target contribution value includes: In the case where the first classification library in the classified document library is the same as the second classification library in the classified document library, determining the target contribution value based on a first contribution value of the vocabulary under the second classification library to the first classification library; When the first classification library is different from the second classification library, the cluster similarity between the clusters under the first classification library and the clusters under the second classification library is determined, and the second contribution value of the cluster with the largest cluster similarity to the first classification library is used as the target contribution value.

[0009] According to a corpus construction method provided by the present invention, the clusters are determined based on the third contribution value of each word in the classified document library to each classification.

[0010] According to a corpus construction method provided by the present invention, the step of determining the third contribution value of each word in the classified document library to each classification includes: Determine the reverse document frequency of each word and determine the discrimination of each word for each category; The third contribution value is determined based on the inverse document frequency and the discrimination.

[0011] According to a corpus construction method provided by the present invention, the discrimination degree is determined based on the frequency of occurrence of each word in each classification library in the classification document library and the number of documents in each classification library.

[0012] The present invention also provides a corpus construction device, comprising the following modules: An acquisition unit, used to acquire an original document and a classified document library, and split the original document into sentences to obtain a sentence set of the original document; a first determination unit, configured to determine a first similarity between each sentence in the sentence set and each cluster, and determine a first contribution of each cluster to a classification library of the classified document library; each classification library of the classified document library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; A second determining unit, configured to determine a degree of belonging of each sentence in the sentence set to the classified document library based on the first similarity and the first contribution; A third determining unit is used to determine a target attribution degree based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determine a target category of the original document in the classified document library based on the target attribution degree; The classification unit is used to classify each sentence in the original document into a cluster to which the maximum similarity belongs among the clusters under the target category.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for constructing a corpus as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for constructing a corpus as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned methods for constructing corpora.

[0016] The corpus construction method, device, electronic device and storage medium provided by the present invention split the original document by sentence to obtain the sentence set of the original document, then determine the first similarity between each sentence in the sentence set and each clustering cluster, and determine the first contribution of each clustering cluster to the classification library of the classification document library, then determine the attribution of each sentence in the sentence set to the classification document library based on the first similarity and the first contribution, determine the target attribution based on the attribution of all sentences in the sentence set belonging to the same classification document library, and determine the target category of the original document in the classification document library based on the target attribution, and finally, classify each sentence in the original document into the clustering cluster with the maximum similarity in the clustering cluster under the target category. This process not only classifies the original document into the target category in the document classification library, but also classifies each sentence in the original document into the clustering cluster with the maximum similarity in the clustering cluster under the target category, thereby performing classification at a more fine-grained level and improving the accuracy of corpus construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the method for constructing the corpus provided by the present invention.

[0019] Figure 2 It is a structural schematic diagram of the corpus construction device provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, rather than to describe a specific order or precedence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type.

[0023] The present invention provides a method for constructing a corpus. Figure 1 It is a flow chart of the method for constructing the corpus provided by the present invention, such as Figure 1 As shown, the method includes step 110, step 120, step 130, step 140 and step 150.

[0024] Step 110, obtaining an original document and a classified document library, and splitting the original document into sentences to obtain a sentence set of the original document.

[0025] Specifically, the original document and the classified document library can be obtained. The original document here is the document that needs to be used for corpus construction later. The original document can be a document formed by text directly input by a user, or a document formed by text obtained by voice transcription of collected audio, or a document formed by text obtained by performing OCR (Optical Character Recognition) on an image obtained by an image acquisition device such as a scanner, mobile phone, or camera. The embodiment of the present invention does not make any specific limitations on this.

[0026] Among them, the classified document library is denoted as L, there are n categories in total, each classification library is denoted as Li, L={L1, L2, …,Ln}. Here, the document classification standard is implemented according to the major categories of the relevant industry classification standards and is divided into 97 industry categories.

[0027] After obtaining the original document, the original document may be split into sentences to obtain a sentence set of the original document. Here, the sentence set of the original document may be represented by S = {s1, s2,…}.

[0028] Step 120, determining the first similarity between each sentence in the sentence set and each cluster, and determining the first contribution of each cluster to the classification library of the classified document library; each classification library of the classified document library corresponds to multiple clusters, and the multiple clusters are obtained based on sentence granularity clustering.

[0029] Specifically, the first similarity between each sentence in the sentence set and each cluster may be determined, and the first contribution of each cluster to the classification library of the classification document library may be determined.

[0030] Here, the first similarity may be calculated by using methods such as cosine similarity and Pearson correlation coefficient, which is not specifically limited in the embodiment of the present invention.

[0031] Here, each classification library corresponds to multiple clusters, and the multiple clusters are obtained based on sentence granularity clustering, that is, for each classification document set Li of the classification document library L, clustering is performed according to sentence granularity to obtain Li's cluster cluster set Ci={Ci_1, Ci_2,…}.

[0032] It can be understood that a cluster is a collection of data objects with high similarity in a data set. These data objects have high similarity within the same cluster, but have great differences from data objects in other clusters.

[0033] Step 130: Determine the degree of belonging of each sentence in the sentence set to the classified document library based on the first similarity and the first contribution.

[0034] Specifically, after obtaining the first similarity and the first contribution, the attribution of each sentence in the sentence set to the classified document library can be determined based on the first similarity and the first contribution, and then the value with the largest attribution can be recorded as the m value.

[0035] Step 140, determining a target degree of attribution based on the degrees of attribution of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the original document in the classified document library based on the target degree of attribution.

[0036] Specifically, the m values ​​of all sentences in the sentence set belonging to the same classified document library are added together to obtain the target attribution degree M, and based on the target attribution degree, the target category of the original document in the classified document library is determined, that is, the classified document library with the largest M value is used as the target category of the original document in the classified document library.

[0037] Step 150: Classify each sentence in the original document into the cluster to which the maximum similarity belongs among the clusters under the target category.

[0038] Furthermore, each sentence in the original document may be classified into the cluster to which the maximum similarity belongs among the clusters under the target category.

[0039] Therefore, a multi-level classification corpus is constructed based on the original document. Not only the original document is classified into the target category in the document classification library, but also each sentence in the original document is classified into the cluster cluster with the maximum similarity in the cluster cluster under the target category, thereby performing classification at a more fine-grained level and improving the accuracy of corpus construction. Furthermore, on this basis, similarity matching of the document to be retrieved in each clustering direction is performed to obtain the document most relevant to the document to be retrieved.

[0040] The method provided by the embodiment of the present invention splits the original document by sentence to obtain the sentence set of the original document, then determines the first similarity between each sentence in the sentence set and each clustering cluster, and determines the first contribution of each clustering cluster to the classification library of the classification document library, then determines the attribution of each sentence in the sentence set to the classification document library based on the first similarity and the first contribution, determines the target attribution based on the attribution of all sentences in the sentence set belonging to the same classification document library, and determines the target category of the original document in the classification document library based on the target attribution, and finally, classifies each sentence in the original document into the clustering cluster with the maximum similarity in the clustering cluster under the target category. This process not only classifies the original document into the target category in the document classification library, but also classifies each sentence in the original document into the clustering cluster with the maximum similarity in the clustering cluster under the target category, thereby performing classification at a more fine-grained level and improving the accuracy of corpus construction.

[0041] Based on the above embodiment, the step 120 of determining the first contribution of each cluster to the classification library of the classification document library includes: Step 121, determining a target contribution value of each cluster to the classification library of the classification document library; Step 122: determining the first contribution based on the second similarity between each cluster and the classification library and the target contribution value.

[0042] Specifically, a target contribution value of each cluster to the classification library of the classification document library may be determined, and then the first contribution may be determined based on the second similarity between each cluster and the classification library and the target contribution value.

[0043] The second similarity may be calculated by using methods such as cosine similarity and Pearson correlation coefficient, which is not specifically limited in the embodiment of the present invention.

[0044] For example, the calculation method of the first contribution of the cluster Cj_k to the classification library Li of the classification document library is: the second similarity between Cj_k and the classification library Li is multiplied by the target contribution value of Cj_k to the classification library Li.

[0045] Based on the above embodiment, the step of determining the target contribution value includes: In the case where the first classification library in the classified document library is the same as the second classification library in the classified document library, determining the target contribution value based on a first contribution value of the vocabulary under the second classification library to the first classification library; When the first classification library is different from the second classification library, the cluster similarity between the clusters under the first classification library and the clusters under the second classification library is determined, and the second contribution value of the cluster with the largest cluster similarity to the first classification library is used as the target contribution value.

[0046] Specifically, when the first classification library in the classified document library is the same as the second classification library in the classified document library, the target contribution value is determined based on the first contribution value of the vocabulary under the second classification library to the first classification library.

[0047] For example, if i = j: count all words under cluster Cj_k (repeat words and calculate independently), add up the first contribution values ​​of these words to the classification library Li, and get the target contribution value of Cj_k to the first classification library Li.

[0048] When the first classification library is different from the second classification library, the cluster similarity between the clusters under the first classification library and the clusters under the second classification library is determined, and the second contribution value of the cluster with the largest cluster similarity to the first classification library is taken as the target contribution value.

[0049] For example, if i != j: among all clusters of Li, take the cluster Ci_l that is most similar to Cj_k, and its second contribution value to Li as the target contribution value of Cj_k to the first classification library Li. Here, the cluster Ci_l that is most similar to Cj_k is determined based on the similarity between the clusters.

[0050] It can be understood that, since there are multiple clusters, two similarity matrices can be obtained based on the similarities between the clusters and based on the similarities between the classified document library and the clusters.

[0051] Based on the above embodiment, the cluster is determined based on the third contribution value of each word in the classified document library to each category.

[0052] Specifically, for each classified document set Li of the classified document library L, the contribution value of each word in the classified document library L to each classification is {c1, c2,…, cn}, and clustering is performed at sentence granularity to obtain Li’s cluster cluster set Ci={Ci_1, Ci_2,…}.

[0053] Based on the above embodiment, the step of determining the third contribution value of each word in the classified document library to each category includes: Step 310, determining the reverse document frequency of each word, and determining the discrimination of each word for each category; Step 320: Determine the third contribution value based on the reverse document frequency and the discrimination.

[0054] Specifically, 1) for word X, calculate its inverse document frequency (idf); Inverse document frequency is an indicator of word importance in NLP (Natural Language Processing). It is calculated by dividing the total number of documents in the corpus by the number of documents in which word X appears, and then taking the logarithm.

[0055] 2) Calculate the discrimination of vocabulary X for all categories {b1, b2,…, bn}: (1) Count the frequency of occurrence of word X in n classification libraries {f1, f2, …, fn}. The frequency of occurrence is the number of documents in the classification library that contain the word. The total frequency of occurrence is f=f1+f2+…+fn. (2) The number of documents in each classification library is {d1, d2, …, dn}, and the total number of documents is d = d1+d2+…+dn; (3) For the i-th classification library, the calculation method of the discrimination bi is: bi = log((d-di+1) / (f-fi+1)) / log(di+1 / fi+1)

[0056] 3) The third contribution value ci of vocabulary X to the i-th category is: ci = idf * bi

[0057] Here, the discrimination degree is determined based on the frequency of occurrence of each word in each classification library in the classification document library and the number of documents in each classification library.

[0058] The following is a description of a corpus construction device provided by the present invention. The corpus construction device described below and the corpus construction method described above can be referenced to each other.

[0059] Based on any of the above embodiments, the present invention provides a corpus construction device, Figure 2 is a schematic diagram of the structure of the corpus construction device provided by the present invention, such as Figure 2 As shown, the device comprises: The acquisition unit 201 is used to acquire an original document and a classified document library, and split the original document into sentences to obtain a sentence set of the original document; A first determination unit 202 is used to determine a first similarity between each sentence in the sentence set and each cluster, and determine a first contribution of each cluster to a classification library of the classified document library; each classification library of the classified document library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; A second determining unit 203 is used to determine the degree of belonging of each sentence in the sentence set to the classified document library based on the first similarity and the first contribution; The third determining unit 204 is used to determine a target attribution degree based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determine a target category of the original document in the classified document library based on the target attribution degree; The classification unit 205 is used to classify each sentence in the original document into a cluster to which the cluster with the maximum similarity belongs among the clusters under the target category.

[0060] The device provided by the embodiment of the present invention splits the original document by sentence to obtain the sentence set of the original document, then determines the first similarity between each sentence in the sentence set and each clustering cluster, and determines the first contribution of each clustering cluster to the classification library of the classification document library, then determines the attribution of each sentence in the sentence set to the classification document library based on the first similarity and the first contribution, determines the target attribution based on the attribution of all sentences in the sentence set belonging to the same classification document library, and determines the target category of the original document in the classification document library based on the target attribution, and finally, classifies each sentence in the original document into the clustering cluster with the maximum similarity in the clustering cluster under the target category. This process not only classifies the original document into the target category in the document classification library, but also classifies each sentence in the original document into the clustering cluster with the maximum similarity in the clustering cluster under the target category, thereby performing classification at a more fine-grained level and improving the accuracy of corpus construction.

[0061] Based on any of the foregoing embodiments, the first determining unit 202 is specifically configured to: A target contribution value determination unit, used to determine a target contribution value of each cluster to the classification library of the classification document library; The first contribution determination unit is configured to determine the first contribution based on the second similarity between each cluster and the classification library and the target contribution value.

[0062] Based on any of the above embodiments, the target contribution value determination unit is specifically used to: In the case where the first classification library in the classified document library is the same as the second classification library in the classified document library, determining the target contribution value based on a first contribution value of the vocabulary under the second classification library to the first classification library; When the first classification library is different from the second classification library, the cluster similarity between the clusters under the first classification library and the clusters under the second classification library is determined, and the second contribution value of the cluster with the largest cluster similarity to the first classification library is used as the target contribution value.

[0063] Based on any of the above embodiments, the cluster is determined based on the third contribution value of each word in the classified document library to each category.

[0064] Based on any of the above embodiments, the third contribution value determining unit is further included, and the third contribution value determining unit is specifically configured to: Determine the reverse document frequency of each word and determine the discrimination of each word for each category; The third contribution value is determined based on the inverse document frequency and the discrimination.

[0065] Based on any of the above embodiments, the discrimination degree is determined based on the frequency of occurrence of each word in each classification library in the classification document library and the number of documents in each classification library.

[0066] Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Figure 3 As shown, the electronic device may include: a processor 310 , a communications interface 320 , a memory 330 and a communication bus 340 , wherein the processor 310 , the communications interface 320 , and the memory 330 communicate with each other via the communication bus 340 . The processor 310 can call the logic instructions in the memory 330 to execute the corpus construction method, which includes: obtaining an original document and a classified document library, and splitting the original document by sentence to obtain a sentence set of the original document; determining a first similarity between each sentence in the sentence set and each clustering cluster, and determining a first contribution of each clustering cluster to the classification library of the classified document library; each classification library of the classified document library corresponds to multiple clustering clusters, and the multiple clustering clusters are obtained based on sentence granularity clustering; based on the first similarity and the first contribution, determining the degree of attribution of each sentence in the sentence set to the classified document library; based on the degree of attribution of all sentences in the sentence set belonging to the same classified document library, determining a target degree of attribution, and based on the target degree of attribution, determining a target category of the original document in the classified document library; classifying each sentence in the original document into the clustering cluster to which the maximum similarity belongs in the clustering clusters under the target category.

[0067] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the corpus construction method provided by the above-mentioned methods, which includes: obtaining an original document and a classified document library, and splitting the original document by sentence to obtain a sentence set of the original document; determining a first similarity between each sentence in the sentence set and each clustering cluster, and determining a first contribution of each clustering cluster to a classification library of the classified document library; each classification library of the classified document library corresponds to multiple clustering clusters, and the multiple clustering clusters are obtained based on sentence granularity clustering; based on the first similarity and the first contribution, determining the degree of attribution of each sentence in the sentence set to the classified document library; based on the degree of attribution of all sentences in the sentence set belonging to the same classified document library, determining a target degree of attribution, and based on the target degree of attribution, determining a target category of the original document in the classified document library; classifying each sentence in the original document into the cluster to which the maximum similarity belongs in the cluster under the target category.

[0069] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the corpus construction method provided by the above-mentioned methods, the method comprising: obtaining an original document and a classified document library, and splitting the original document by sentence to obtain a sentence set of the original document; determining a first similarity between each sentence in the sentence set and each clustering cluster, and determining a first contribution of each clustering cluster to a classification library of the classified document library; each classification library of the classified document library corresponds to a plurality of clustering clusters, and the plurality of clustering clusters are obtained based on sentence granularity clustering; determining the degree of attribution of each sentence in the sentence set to the classified document library based on the first similarity and the first contribution; determining a target degree of attribution based on the degrees of attribution of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the original document in the classified document library based on the target degree of attribution; and classifying each sentence in the original document into the clustering cluster to which the maximum similarity belongs among the clustering clusters under the target category.

[0070] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0071] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a corpus, characterized in that: include: Acquire an original document and a classified document library, and split the original document into sentences to obtain a sentence set of the original document; Determine a first similarity between each sentence in the sentence set and each cluster, and determine a first contribution of each cluster to a classification library of the classified document library; each classification library of the classified document library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; Determining the degree of belonging of each sentence in the sentence set to the classified document library based on the first similarity and the first contribution; Determining a target degree of attribution based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determining a target category of the original document in the classified document library based on the target degree of attribution; Each sentence in the original document is classified into the cluster to which the maximum similarity belongs among the clusters under the target category.

2. The method for constructing a corpus according to claim 1, characterized in that: The determining of the first contribution of each cluster to the classification library of the classification document library includes: Determining a target contribution value of each cluster to the classification library of the classification document library; The first contribution is determined based on the second similarity between each cluster and the classification library, and the target contribution value.

3. The method for constructing a corpus according to claim 2, characterized in that: The step of determining the target contribution value includes: In the case where the first classification library in the classified document library is the same as the second classification library in the classified document library, determining the target contribution value based on a first contribution value of the vocabulary under the second classification library to the first classification library; When the first classification library is different from the second classification library, the cluster similarity between the clusters under the first classification library and the clusters under the second classification library is determined, and the second contribution value of the cluster with the largest cluster similarity to the first classification library is used as the target contribution value.

4. The method for constructing a corpus according to claim 3, characterized in that: The clusters are determined based on the third contribution value of each word in the classified document library to each category.

5. The method for constructing a corpus according to claim 4, characterized in that: The step of determining the third contribution value of each word in the classified document library to each category includes: Determine the reverse document frequency of each word and determine the discrimination of each word for each category; The third contribution value is determined based on the inverse document frequency and the discrimination.

6. The method for constructing a corpus according to claim 5, characterized in that: The degree of distinction is determined based on the frequency of occurrence of each word in each classification library in the classification document library and the number of documents in each classification library.

7. A corpus construction device, characterized in that: include: An acquisition unit, used to acquire an original document and a classified document library, and split the original document into sentences to obtain a sentence set of the original document; a first determination unit, configured to determine a first similarity between each sentence in the sentence set and each cluster, and determine a first contribution of each cluster to a classification library of the classified document library; each classification library of the classified document library corresponds to a plurality of clusters, and the plurality of clusters are obtained based on sentence granularity clustering; A second determining unit, configured to determine a degree of belonging of each sentence in the sentence set to the classified document library based on the first similarity and the first contribution; A third determining unit is used to determine a target attribution degree based on the attribution degrees of all sentences in the sentence set belonging to the same classified document library, and determine a target category of the original document in the classified document library based on the target attribution degree; The classification unit is used to classify each sentence in the original document into a cluster to which the maximum similarity belongs among the clusters under the target category.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method for constructing the corpus according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a corpus as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for constructing a corpus as claimed in any one of claims 1 to 6 is implemented.

Images