Method and device for generating text summary

Abstract

This disclosure proposes a method and apparatus for generating a text summary, relating to the field of natural language processing. The method comprises: obtaining a target text; clustering candidate sentences in the target text to generate at least one cluster; extracting the core sentences of each cluster from the target sentences included in the cluster; and generating a text summary of the target text based on the core sentences of each cluster. Thus, by clustering the sentences in the target text, the theme of the target text can be obtained, and a text summary can be extracted from the theme, making the text summary more comprehensive and accurate.

Description

Technical Field

[0001] The present disclosure relates to the field of natural language processing, and in particular to a method and device for generating a text summary. Background Art

[0002] Text summarization is a method for extracting valuable information from a given text and presenting it to the user in a simple, concise, and condensed form while retaining the relevant content of the source document. A good text summarization algorithm requires high relevance, minimal redundancy, a suitable compression ratio, and high coverage. Currently, text summarization methods primarily rely on extractive and generative summarization. Summary of the Invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to a certain extent.

[0004] To this end, one purpose of the present disclosure is to propose a method for generating a text summary.

[0005] The second objective of the present disclosure is to provide a device for generating a text summary.

[0006] A third objective of the present disclosure is to provide an electronic device.

[0007] A fourth object of the present disclosure is to provide a non-transitory computer-readable storage medium.

[0008] A fifth object of the present disclosure is to provide a computer program product.

[0009] To achieve the above-mentioned objectives, the first embodiment of the present disclosure proposes a method for generating a text summary, comprising: obtaining a target text; clustering candidate sentences in the target text to generate at least one cluster; for each cluster, extracting the core sentence of the cluster from the target sentences included in the cluster; and generating a text summary of the target text based on the core sentence of each cluster.

[0010] The present disclosure clusters texts, extracts core sentences from the clustered texts, and generates a text summary of the target text. This improves the relevance of the text summary, and makes it more coherent and readable.

[0011] According to one embodiment of the present disclosure, the method for generating a text summary further includes: splitting the target text to generate multiple candidate sentences; and subject-clustering the candidate sentences to generate at least one cluster, wherein one subject corresponds to one cluster.

[0012] According to one embodiment of the present disclosure, the method for generating a text summary further includes: obtaining the probability of a candidate sentence appearing in a cluster; and selecting a cluster with the highest probability of appearance and determining it as the target cluster to which the candidate sentence belongs.

[0013] According to one embodiment of the present disclosure, the method for generating a text summary further includes: obtaining a completeness parameter of a cluster; obtaining a homogeneity parameter between the cluster and a candidate sentence; obtaining a semantic weight of co-occurring words between the cluster and the target sentence included therein; and obtaining an occurrence probability based on the completeness parameter, the homogeneity parameter, and the semantic weight.

[0014] According to one embodiment of the present disclosure, the method for generating a text summary further includes: obtaining a first number of target sentences currently included in the cluster; obtaining a second number of candidate sentences in the target text; and determining a completeness parameter based on the first number and the second number.

[0015] According to one embodiment of the present disclosure, the method for generating a text summary also includes: obtaining a first word frequency of any participle appearing in a candidate sentence; obtaining a second word frequency of any participle appearing in a cluster and the total number of words in the cluster; obtaining a first weight of any participle in each cluster; and determining a homogeneity parameter based on the first word frequency, the second word frequency, the total number of words, the first weight, and the dictionary size.

[0016] According to one embodiment of the present disclosure, the method for generating a text summary further includes: obtaining the second word frequency of the co-occurring words in the target sentence in each cluster, and determining the semantic weight according to the second word frequency of each co-occurring word.

[0017] According to one embodiment of the present disclosure, the method for generating a text summary further includes: obtaining a sub-semantic weight between any two co-occurring words based on the second word frequency of any two co-occurring words; and summing the sub-semantic weights between all any two co-occurring words to obtain a semantic weight.

[0018] According to one embodiment of the present disclosure, the method for generating a text summary further includes:

[0019]

[0020] Among them, w ij represents the sub-semantic weight; x′ represents the target sentence; Indicates the co-occurrence word w i the frequency of the second word appearing in the target sentence; Indicates the co-occurrence word w j The frequency of the second word appearing in the target sentence.

[0021] According to one embodiment of the present disclosure, the method for generating a text summary further includes: performing a topic clustering analysis on the candidate sentences using a topic clustering model to obtain the probability of the candidate sentences appearing in the cluster.

[0022] According to one embodiment of the present disclosure, the method for generating a text summary further includes: obtaining a second weight of a target sentence in a cluster; and selecting a core sentence of the cluster from the target sentences in the cluster according to the second weight.

[0023] According to one embodiment of the present disclosure, the method for generating a text summary also includes: obtaining a directed graph between target sentences in the cluster, wherein the nodes in the directed graph represent the target sentences and the edges between the nodes represent the contribution relationship between the target sentences; and obtaining a second weight of each target sentence in the cluster based on the directed graph and the TextRank algorithm.

[0024] To achieve the above-mentioned purpose, the second embodiment of the present disclosure proposes a device for generating a text summary, including: an acquisition module for acquiring a target text; a clustering module for clustering candidate sentences in the target text to generate at least one cluster; an extraction module for extracting the core sentences of each cluster from the target sentences included in the cluster; and a generation module for generating a text summary of the target text based on the core sentences of each cluster.

[0025] According to one embodiment of the present disclosure, the clustering module is further used to: split the target text to generate multiple candidate sentences; perform topic clustering on the candidate sentences to generate at least one cluster, wherein one topic corresponds to one cluster.

[0026] According to one embodiment of the present disclosure, the clustering module is further configured to: obtain the probability of a candidate sentence appearing in a cluster; and select the cluster with the largest probability of appearance and determine it as the target cluster to which the candidate sentence belongs.

[0027] According to one embodiment of the present disclosure, the clustering module is further used to: obtain the integrity parameter of the cluster cluster; obtain the homogeneity parameter of the cluster cluster and the candidate sentence; obtain the semantic weight of the co-occurring words between the cluster cluster and the target sentences included; and obtain the occurrence probability based on the integrity parameter, homogeneity parameter and semantic weight.

[0028] According to one embodiment of the present disclosure, the clustering module is further used to: obtain a first number of target sentences currently included in the cluster; obtain a second number of candidate sentences in the target text; and determine an integrity parameter based on the first number and the second number.

[0029] According to one embodiment of the present disclosure, the clustering module is further used to: obtain the first word frequency of any participle appearing in the candidate sentence; obtain the second word frequency of any participle appearing in the cluster and the total number of words in the cluster; obtain the first weight of any participle in each cluster; and determine the homogeneity parameter based on the first word frequency, the second word frequency, the total number of words, the first weight and the dictionary size.

[0030] According to one embodiment of the present disclosure, the clustering module is further configured to obtain a second word frequency of co-occurring words in a target sentence in each cluster, and determine a semantic weight based on the second word frequency of each co-occurring word.

[0031] According to one embodiment of the present disclosure, the clustering module is further used to: obtain the sub-semantic weight between any two co-occurring words based on the second word frequency of any two co-occurring words; and sum up the sub-semantic weights between all any two co-occurring words to obtain the semantic weight.

[0032] According to one embodiment of the present disclosure, the device for generating a text summary further includes:

[0033]

[0034] Among them, w ij represents the sub-semantic weight; x′ represents the target sentence; Indicates the co-occurrence word w i the frequency of the second word appearing in the target sentence; Indicates the co-occurrence word w j The frequency of the second word appearing in the target sentence.

[0035] To achieve the above-mentioned purpose, the third aspect embodiment of the present disclosure proposes an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to implement the method for generating a text summary as described in the first aspect embodiment of the present disclosure.

[0036] To achieve the above-mentioned purpose, the fourth embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to implement the method for generating a text summary as described in the first embodiment of the present disclosure.

[0037] To achieve the above-mentioned purpose, the fifth embodiment of the present disclosure proposes a computer program product, including a computer program, which, when executed by a processor, is used to implement the method for generating a text summary as described in the first embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of a method for generating a text summary according to one embodiment of the present disclosure;

[0039] Figure 2 is a schematic diagram of a method for generating a text summary according to another embodiment of the present disclosure;

[0040] Figure 3 is a schematic diagram of the construction process of the Dirichlet process mixture model provided by the present disclosure;

[0041] Figure 4 is a schematic diagram of the construction process of the Chinese restaurant process provided by the present disclosure;

[0042] Figure 5 is a schematic diagram of a method for generating a text summary according to another embodiment of the present disclosure;

[0043] Figure 6 is a schematic diagram of a method for generating a text summary according to another embodiment of the present disclosure;

[0044] Figure 7 is a schematic structural diagram of a target node of a text summary according to another embodiment of the present disclosure;

[0045] Figure 8 is a schematic diagram of a device for generating a text summary according to an embodiment of the present disclosure;

[0046] Figure 9 is a schematic diagram of an electronic device according to one embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] The following describes in detail embodiments of the present disclosure, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present disclosure, and should not be construed as limiting the present disclosure.

[0048] Figure 1 A schematic diagram of an exemplary embodiment of a method for generating a text summary proposed in the present disclosure is shown as follows: Figure 1 As shown, the method for generating a text summary includes the following steps:

[0049] S101, obtaining target text.

[0050] In the embodiments of the present disclosure, there are multiple methods for obtaining the target text.

[0051] Optionally, the target text is input into the electronic device through an external input device. It should be noted that the external input device can be an external keyboard, an external scanner, etc.

[0052] Optionally, the electronic device can also obtain the target text from the text library by connecting to the text library. It should be noted that the target text can be prepared in advance and pre-stored in the text library, which can be stored in the storage space of the electronic device or in a server.

[0053] Optionally, you can also download the target text file from a web page by accessing the web page.

[0054] S102: Cluster the candidate sentences in the target text to generate at least one cluster.

[0055] It is understandable that the target text is composed of one or more sentences, and different sentences may have different semantics. Correspondingly, different semantics can represent different topics. After processing the target text to generate candidate sentences, the candidate sentences can be thematically clustered to generate at least one cluster. It should be noted that the sentences in the cluster have a common theme, which is different from the themes of other clusters. For example, the target text includes N candidate sentences, where sentences A, B, and C have the same theme, but a different theme from sentence D. Then sentences A, B, and C can be included in the same cluster, and sentence D is located in another cluster.

[0056] There may be multiple methods for generating clusters. Optionally, candidate sentences may be clustered using a topic clustering analysis algorithm to generate at least one cluster.

[0057] Optionally, the candidate sentences may be processed by calling a topic clustering model to generate at least one cluster. It should be noted that the topic clustering model may be trained in advance and pre-stored in the storage space of the electronic device to be retrieved and used when needed.

[0058] S103 , for each cluster, extracting the core sentence of the cluster from the target sentences included in the cluster.

[0059] The target sentences are all the sentences included in the corresponding cluster. By processing the target sentences in the cluster and extracting the cluster, the core sentences of the cluster can be extracted. It can be understood that the core sentences contain the theme information of the cluster.

[0060] Optionally, the target sentence in the cluster can be matched with the sentences in the sentence library to generate a matching degree between the cluster and the sentences in the sentence library. The higher the matching degree, the greater the possibility that the sentence in the sentence library is the core sentence of the cluster. Accordingly, the sentence with the highest matching degree can be considered the core sentence of the cluster. It should be noted that the above sentence library is a database that stores all the subject sentences.

[0061] Optionally, the target sentence in the cluster may be processed by an extraction algorithm to generate a semantic weight of the target sentence, and a core sentence of the cluster may be generated according to the semantic weight of the target sentence.

[0062] S104: Generate a text summary of the target text based on the core sentences of each cluster.

[0063] After completing the topic clustering of the target text, at least one cluster and at least one core sentence corresponding to the cluster can be obtained. The core sentence is processed to generate a text summary of the target text.

[0064] Optionally, when the generated cluster is a single one, it can be considered that the core sentence corresponding to the cluster is the text summary of the target text.

[0065] Optionally, when multiple clusters are generated, the top m core sentences of the top k topics with the largest number of sentences can be selected as the final text summary. It should be noted that k and m are pre-set and can be modified according to actual conditions.

[0066] Optionally, the core sentences of multiple clusters may be concatenated according to semantics to generate a text summary of the target text.

[0067] Optionally, weights of the core sentences of the multiple clusters may be further extracted, and a text summary of the target text may be generated according to the weights of the core sentences.

[0068] In the disclosed embodiment, a target text is first acquired, candidate sentences in the target text are clustered to generate at least one cluster. For each cluster, the core sentences of the cluster are extracted from the target sentences included in the cluster. Finally, a text summary of the target text is generated based on the core sentences of each cluster. Thus, by clustering the sentences in the target text, the intent of the target text can be better explored, the efficiency and accuracy of obtaining text summaries can be improved, and a more comprehensive text summary can be obtained.

[0069] In the above embodiment, the sentences in the target text are clustered to generate at least one cluster, which can be combined with Figure 2 To explain further, Figure 2 This is a schematic diagram of another exemplary embodiment of a method for generating a text summary proposed in the present disclosure. As shown in the figure, the method includes:

[0070] S201, split the target text to generate multiple candidate sentences.

[0071] After obtaining the target text, the target text can be split according to the splitting conditions to obtain multiple candidate sentences. The splitting conditions can be a complete sentence, a complete semantic paragraph, etc., which can be set according to actual needs.

[0072] Optionally, before splitting the target text, the target text may be cleaned first. By cleaning the data, erroneous data and useless data in the target text may be filtered out, so that subsequent splitting is easier and the candidate sentences obtained are more accurate.

[0073] Optionally, the data cleaning process may include: (1) removing special characters and stop words from the target text and storing them in a dictionary; (2) converting the target text into an array of candidate sentences; and (3) extracting topics generated by each candidate sentence. It should be noted that the topics may be bi-grams or tri-grams. The dictionary may be understood as a collection of words in the target text, and the dictionary may be stored in a storage space of an electronic device.

[0074] It should be noted that stop words refer to certain characters or words that are automatically filtered out before or after processing natural language data (or text). The stop words in this embodiment can be set according to actual needs.

[0075] S202 , performing topic clustering on the candidate sentences to generate at least one cluster, wherein one topic corresponds to one cluster.

[0076] A cluster is a collection of data objects obtained through cluster analysis. These objects are similar to objects in the same cluster, that is, they are homogeneous, and different from objects in other clusters. The clusters obtained based on the topic clustering model are called topic-based clusters, with each topic corresponding to a cluster.

[0077] In the embodiment of the present disclosure, the topic clustering model may be a Latent Dirichlet Allocation. It should be noted that the Dirichlet Process Mixture Model (DPMM) is a non-parametric Bayesian model.

[0078] In the embodiment of the present disclosure, by extracting cluster themes, semantic information of the target text can be introduced by cluster themes, which can make it easier to extract a summary of the target text.

[0079] It should be noted that the Dirichlet process mixture model can be understood as a clustering method, but it does not need to specify the number of categories. It can shorten the number of clusters from the data.

[0080] In the Dirichlet process mixture model, the following definitions are included:

[0081] G~DP(α0, G0),

[0082] θ i |G~G,

[0083] x i |θ i ~F(θ i ).

[0084] Where i is a natural number, F(θ i ) is a mixed distribution that contains several individual distributions. i ) includes data x1, x2, ..., x n , where n > 2 and n is a natural number. These data can be multivariate, independent, and come from some unknown distribution. Their values ​​can be real or categorical.

[0085] G is the probability distribution on the measure space generated by the Dirichlet process, θ i is the parameter of a distribution in the mixed distribution, x i |θ i ~F(θ i ) is based on the parameter θ i The observed value of a distribution generated if x i and x j come from the same distribution, then θ i =θ j , where j is a natural number.

[0086] like Figure 3 As shown, when constructing the Dirichlet process mixture model, the probability distribution G on the measure space Θ is generated according to the Dirichlet process with parameters α0 and G0, and the parameter θ of a distribution in the mixture distribution is obtained according to the parameter G. i , according to the parameter θ i Generate observations x from a certain distribution i .

[0087] F(θ i ) in the parameter θ i It comes from the Dirichlet Process (DP) G, that is, the prior of the parameters of the distribution is a Dirichlet process. Its concentration parameter is α0 and the base distribution is G0.

[0088] For any finite partition of space, if the measure of a random probability distribution in space on each part of this partition obeys a Dirichlet distribution, then we say that the random probability distribution obeys a Dirichlet process.

[0089] The Dirichlet process is a random process of nonparametric Bayesian model, which is defined as follows: Assume that G0 is a random probability distribution on the measure space Θ, parameter α0 is a positive real number, and for any finite partition A1,…,A on the measure space r , where r is a natural number, if there exists (G(A1),…,G(A r ))~Dir(α0G0(A1),…,α0G0(A r )), then G is considered to obey the Dirichlet process composed of the base distribution G0 and the parameter α0, which can be expressed as G~DP(α0, G0).

[0090] Because the Dirichlet distribution and the multinomial distribution have conjugate properties, when the partition of the measure space satisfies the multinomial distribution, that is, θ1,…,θ n ~Mult, then the posterior distribution of the Dirichlet process for the finite partition of the measure space also obeys the Dirichlet process, that is, (G(A1),…,G(A r ))|θ1,…,θ n ~Dir(α0G0(A1)+n1,…,α0G0(A r )+n r ), where n1,…,n r Indicates that the observation belongs to A k The number of.

[0091] A multinomial distribution refers to a situation where the value of a random variable in a single experiment is no longer 0-1, but rather has multiple discrete values ​​(1, 2, 3, ..., k). For example, in an experiment involving rolling a six-sided dice, the results of n experiments follow a multinomial distribution with k = 6. Here, n and k are both natural numbers.

[0092] In Bayesian probability theory, if the posterior probability and the prior probability follow the same distribution law, then the prior and posterior distributions are called conjugate distributions, and the prior distribution is called the conjugate prior distribution of the likelihood function. When the observed data conforms to a multinomial distribution, and both the prior and posterior distributions of the parameters follow the Dirichlet distribution, this is called a Dirichlet-multinomial conjugate distribution.

[0093] The data of the Dirichlet process mixture model comes from a mixed distribution, that is, n data come from k distributions, where k is a natural number and k≤n. Therefore, for observations from the same distribution, the distribution θ iThe parameters should be the same. However, if our prior G is a continuous distribution, then the sampling results cannot have the same value. Due to the characteristics of the Dirichlet process, we can extract the same parameter θ and make it similar to the base distribution. Furthermore, because the data growth rate of this model is much faster than the growth of the number of mixture models, for a given sample data, we can obtain a number of categories far smaller than the sample size, so this model can be used as a nonparametric clustering method.

[0094] The sampling method in the definition of the Dirichlet process can be modeled using the Chinese Restaurant Process (CRP). The modeling process is as follows:

[0095] like Figure 4 As shown, suppose there is a restaurant that can accommodate infinite tables, and the parameters of each table are Assign a table to each customer z1,…,z n , z i Denotes the table assigned to the i-th customer. Each new customer can open a new table with probability Or sit at a table that already has customers, with a probability of n k is the number of customers already at the kth table.

[0096] The CRP process can only partition identical data into blocks, but cannot cluster similar data. The clustering problem of similar data can be solved using the Dirichlet process mixture model. Based on the CRP process, the number of clusters in the Dirichlet process mixture model can be obtained.

[0097] In a hypothetical scenario, a Chinese restaurant has an infinite number of circular tables, each with infinite capacity. A customer is seated at the first table. The next customer can choose to sit at the same table as the previous one or at the next table down. This process continues indefinitely, with each customer having only two options: sitting at a table already occupied or at an unoccupied table. The probability of choosing an occupied table is proportional to the number of customers already at that table; that is, customers are more likely to sit at a table with more customers than at a table with fewer. The result of this process is commutative, meaning that the order in which customers are seated does not affect the probability of the final assignment. This property of the Chinese restaurant process greatly simplifies many problems in topic mining in text. For example, the number of topics obtained from a Dirichlet process mixture model can be determined using the Chinese restaurant process.

[0098] In the above embodiment, the candidate sentences are clustered by subject to generate at least one cluster, which can be combined with Figure 5 To explain further, Figure 5 This is a schematic diagram of another exemplary embodiment of a method for generating a text summary proposed in the present disclosure. As shown in the figure, the method includes:

[0099] S501, obtaining the probability of a candidate sentence appearing in a cluster.

[0100] In the embodiment of the present disclosure, first, the completeness parameter of the cluster is obtained, the homogeneity parameter of the cluster and the candidate sentence is obtained, the semantic weight of the co-occurring words between the cluster and the target sentence included is obtained, and then the occurrence probability is obtained based on the completeness parameter, homogeneity parameter and semantic weight.

[0101] The completeness parameter of the cluster can be obtained by the following steps: obtaining a first number of target sentences currently included in the cluster, obtaining a second number of candidate sentences in the target text, and determining the completeness parameter based on the first number and the second number. Specifically, the completeness parameter can be calculated using the following formula:

[0102]

[0103] Among them, α0 is the set parameter, m z represents the first number of candidate sentences included in the cluster, and D represents the second number of candidate sentences in all clusters.

[0104] Furthermore, obtaining the homogeneity parameter of the clusters and the candidate sentences may include the following steps: obtaining the first word frequency of any word in the candidate sentence, obtaining the second word frequency of any word in the cluster and the total number of words in the cluster, obtaining the first weight of any word in each cluster, and determining the homogeneity parameter based on the first word frequency, the second word frequency, the total number of words, the first weight, and the dictionary size. Specifically, the calculation formula of the homogeneity parameter may be:

[0105]

[0106] Among them, i and j are natural numbers, represents the first word frequency of word w in candidate sentence x, represents the second word frequency of word w in cluster z, N x Indicates the total number of words in the candidate sentence x, n z represents the total number of words in cluster z, V represents the dictionary size, and ICF w Indicates the importance of word w among various clusters. Specifically, ICF(w∈x) can be calculated by the following formula:

[0107]

[0108] Furthermore, obtaining the semantic weights of the co-occurring words between the clusters and the target candidate sentences included therein may include the following steps: obtaining the second word frequency of the co-occurring words in the target sentences in each cluster, and determining the semantic weight based on the second word frequency of each co-occurring word.

[0109] Furthermore, determining the semantic weight based on the second word frequency of each co-occurring word may include the following steps: obtaining the sub-semantic weight between any two co-occurring words based on the second word frequency of any two co-occurring words, and summing up the sub-semantic weights between all any two co-occurring words to obtain the semantic weight.

[0110] in, Indicates the co-occurrence word w j The frequency of the second word appearing in the target sentence, x′ represents the target sentence; Indicates the co-occurrence word w i The frequency of the second word appearing in the target sentence.

[0111] Furthermore, the sub-semantic weight cw ij The formula can be expressed as:

[0112]

[0113] According to the content described in the above embodiment, the probability of occurrence in a cluster can be expressed by the following formula:

[0114]

[0115] It can be seen that the more sentences a cluster contains, the higher the value of the integrity parameter, and the higher the integrity of the cluster. It can be understood that the larger the integrity parameter, the more representative the topic corresponding to the candidate sentence is of the semantics of the target document.

[0116] In the disclosed embodiments, the homogeneity function can be used to measure the homogeneity between clusters and candidates. The higher the value of the second term to the right of the equal sign, the higher the homogeneity between the clusters and the candidate sentences. The homogeneity of a cluster represents the degree of compactness of the cluster. The homogeneity function of a cluster ranges from [0 to 1], with larger values ​​indicating a higher degree of cluster compactness.

[0117] It can be understood that the higher the semantic weight of the co-occurring words between the cluster and the candidate sentence, the greater the possibility that the candidate sentence is the target sentence.

[0118] S502: Select the cluster with the highest probability of occurrence and determine it as the target cluster to which the candidate sentence belongs.

[0119] In the disclosed embodiment, the probability of occurrence of a candidate sentence in different clusters is different. The greater the probability of occurrence, the greater the possibility that the candidate sentence belongs to the cluster.

[0120] For example, the target text includes three clusters A, B, and C. The probability of the candidate sentence appearing in cluster A is 0.9, the probability of appearing in cluster B is 0.8, and the probability of appearing in cluster C is 0.7. Then we can consider cluster A as the target cluster to which the candidate sentence belongs.

[0121] Therefore, by matching candidate sentences with clusters, the semantics of the sentences can be judged by the matching probability, so that the semantics of the sentences can be obtained simply and efficiently.

[0122] It should be noted that the data involved in the above embodiments can all be obtained by processing the target text.

[0123] In the above embodiment, extracting the core sentence of the cluster from the target sentences included in the cluster may also include the following steps: Figure 6 As shown:

[0124] S601, obtaining a second weight of the target sentence in the cluster.

[0125] In an embodiment of the present disclosure, the clusters may be represented by a directed graph, wherein the nodes in the directed graph represent target sentences, and the edges between the nodes represent the contribution relationship between the sentences.

[0126] In implementation, the target sentence can be processed using the TextRank algorithm to generate a second weight for the target sentence. It should be noted that the TextRank algorithm is a graph-based ranking algorithm for keyword extraction and document summarization. It uses the co-occurrence information (semantics) between words within a document to extract keywords. It can extract keywords and keyword phrases from a given text and use an extractive automatic summarization method to extract key sentences from the text.

[0127] The TextRank algorithm calculation formula is as follows:

[0128]

[0129] Among them, WS(V i ) represents the second weight of the target sentence i, WS(V j ) represents the second weight of sentence j from the last iteration, d represents the damping coefficient, In(V i ) represents the set of nodes flowing into i, Out(V j ) represents the set of outgoing i nodes, W jiIt can be seen that the weight of the target node is related to the weight of the node flowing into i and the node flowing out of i. Figure 7 As shown, the target node i has common edges with the nodes flowing into i and the nodes flowing out of i. Specifically, except for isolated vertices, each node is associated with at least one edge and can be completely described by the edge set.

[0130] It should be noted that the damping coefficient described in this embodiment can be set according to the actual model, and is usually 0.85.

[0131] S602: Select a core sentence of the cluster from the target sentences in the cluster according to the second weight.

[0132] It is understandable that the greater the second weight of the target sentence, the closer the semantics of the target sentence is to the theme of the cluster. Thus, the core sentence of the cluster can be determined according to the proportion of the second weight.

[0133] Optionally, the target sentences in the cluster may be sorted according to the second weight value, and the first n target sentences may be selected to generate the core sentences of the cluster, where n is an integer greater than or equal to .

[0134] Optionally, the second weight may be processed by a core sentence generation algorithm to generate a core sentence of the cluster.

[0135] In the disclosed embodiment, the second weight of the target sentence in the cluster is first obtained, and then the core sentence of the cluster is selected from the target sentences in the cluster based on the second weight. Thus, by digitizing the semantics of the sentences in the cluster, the target sentence can be analyzed more conveniently, thereby obtaining an accurate core sentence.

[0136] Figure 8 A schematic diagram of a text summary generation device proposed in this disclosure, such as Figure 8 As shown, the text summary generation device 800 includes: an acquisition module 801, a clustering module 802, an extraction module 803 and a generation module 804, wherein:

[0137] An acquisition module 801 is used to acquire a target text;

[0138] A clustering module 802 is configured to cluster candidate sentences in the target text to generate at least one cluster;

[0139] An extraction module 803 is configured to extract, for each cluster, the core sentences of the cluster from the target sentences included in the cluster;

[0140] The generating module 804 is configured to generate a text summary of the target text according to the core sentences of each cluster.

[0141] Furthermore, the clustering module 802 is further configured to: split the target text to generate a plurality of candidate sentences; and perform topic clustering on the candidate sentences to generate at least one cluster, wherein one topic corresponds to one cluster.

[0142] Furthermore, the clustering module 802 is further configured to: obtain the probability of a candidate sentence appearing in a cluster; and select a cluster with the largest probability of appearing, and determine it as the target cluster to which the candidate sentence belongs.

[0143] Furthermore, the clustering module 802 is also used to: obtain the integrity parameter of the cluster; obtain the homogeneity parameter of the cluster and the candidate sentence; obtain the semantic weight of the co-occurring words between the cluster and the target sentence included; and obtain the occurrence probability based on the integrity parameter, homogeneity parameter and semantic weight.

[0144] Furthermore, the clustering module 802 is further configured to: obtain a first number of target sentences currently included in the cluster; obtain a second number of candidate sentences in the target text; and determine an integrity parameter based on the first number and the second number.

[0145] Furthermore, the clustering module 802 is also used to: obtain the first word frequency of any participle appearing in the candidate sentence; obtain the second word frequency of any participle appearing in the cluster and the total number of words in the cluster; obtain the first weight of any participle in each cluster; and determine the homogeneity parameter based on the first word frequency, the second word frequency, the total number of words, the first weight and the dictionary size.

[0146] Furthermore, the clustering module 802 is further configured to obtain a second word frequency of the co-occurring words in the target sentence in each cluster, and determine a semantic weight according to the second word frequency of each co-occurring word.

[0147] Furthermore, the clustering module 802 is further configured to: obtain a sub-semantic weight between any two co-occurring words according to the second word frequencies of any two co-occurring words; and sum up the sub-semantic weights between all any two co-occurring words to obtain a semantic weight.

[0148] Furthermore, the clustering module 802 uses the following formula to determine the sub-semantic weight:

[0149]

[0150] Among them, w ij represents the sub-semantic weight; x′ represents the target sentence; Indicates the co-occurrence word w i the frequency of the second word appearing in the target sentence; Indicates the co-occurrence word w j The frequency of the second word appearing in the target sentence.

[0151] In order to implement the above embodiment, the present disclosure also provides an electronic device 900, such as Figure 9 As shown, the electronic device 900 includes: a processor 901 and a memory 902 communicatively connected to the processor, the memory 902 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 901 to implement the method for generating a text summary as in the embodiment of the first aspect of the present disclosure.

[0152] In order to implement the above embodiment, the embodiment of the present disclosure further proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to implement the method for generating a text summary as in the embodiment of the first aspect of the present disclosure.

[0153] In order to implement the above embodiments, the embodiments of the present disclosure further provide a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for generating a text summary as described in the first embodiment of the present disclosure.

[0154] In the description of the present disclosure, it should be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present disclosure and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation to the present disclosure.

[0155] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0156] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and features of different embodiments or examples, unless they are mutually inconsistent.

[0157] Although the embodiments of the present disclosure have been shown and described above, it is understood that the above embodiments are illustrative and are not to be construed as limitations on the present disclosure. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present disclosure.

Claims

1. A method for generating a text summary, characterized in that: include: Get the target text; Splitting the target text to generate multiple candidate sentences; Performing topic clustering on the candidate sentences to generate at least one cluster, wherein one topic corresponds to one cluster; For each cluster, extracting a core sentence of the cluster from the target sentences included in the cluster; Generating a text summary of the target text according to the core sentences of each cluster; The subject clustering of the candidate sentences to generate the at least one cluster includes: Obtaining the integrity parameter of the cluster; Obtaining homogeneity parameters of the cluster and the candidate sentence; Obtaining semantic weights of co-occurring words between the cluster and the target sentence included therein; Obtaining, according to the completeness parameter, the homogeneity parameter, and the semantic weight, an occurrence probability of the candidate sentence in the cluster; The cluster with the highest occurrence probability is selected and determined as the target cluster to which the candidate sentence belongs.

2. The method according to claim 1, characterized in that The obtaining of the integrity parameter of the cluster includes: Obtaining a first number of target sentences currently included in the cluster; Obtaining a second number of the candidate sentences in the target text; The integrity parameter is determined based on the first number and the second number.

3. The method according to claim 1, characterized in that The obtaining of homogeneity parameters of the clusters and the candidate sentences includes: Obtain the first word frequency of any participle appearing in the candidate sentence; Obtaining the second word frequency of any one of the participles appearing in the cluster and the total number of words in the cluster; Obtaining the first weight of any word in each cluster; The homogeneity parameter is determined according to the first word frequency, the second word frequency, the total number of words, the first weight, and the size of the dictionary.

4. The method according to claim 1, wherein The obtaining of semantic weights of co-occurring words between the cluster and the target sentence includes: The second word frequency of the co-occurring words in the target sentence in each of the clusters is obtained, and the semantic weight is determined according to the second word frequency of each of the co-occurring words.

5. The method according to claim 4, characterized in that Determining the semantic weight according to the second word frequency of each of the co-occurring words includes: Obtaining a sub-semantic weight between any two co-occurring words according to the second word frequencies of the two co-occurring words; The sub-semantic weights between all the arbitrary two co-occurring words are summed to obtain the semantic weight.

6. The method according to claim 5, characterized in that The sub-semantic weight is determined using the following formula: Among them, the represents the sub-semantic weight; represents the target sentence; Indicates co-occurring words The second word frequency appearing in the target sentence; Indicates co-occurring words The second word frequency appearing in the target sentence, z represents the cluster.

7. The method according to any one of claims 1 to 6, characterized in that The subject clustering of the candidate sentences to generate the at least one cluster includes: The candidate sentences are subjected to a topic clustering analysis using a topic clustering model to obtain the probability of the candidate sentences appearing in the cluster.

8. The method according to any one of claims 1 to 6, characterized in that The step of extracting the core sentences of the cluster from the target sentences included in the cluster comprises: Obtaining a second weight of the target sentence in the cluster; According to the second weight, a core sentence of the cluster is selected from the target sentences in the cluster.

9. The method according to claim 8, characterized in that The obtaining of the second weight of the target sentence in the cluster includes: Obtaining a directed graph between target sentences in the clusters, wherein nodes in the directed graph represent the target sentences, and edges between the nodes represent contribution relationships between the target sentences; According to the directed graph and the TextRank algorithm, a second weight of each target sentence in the cluster is obtained.

10. A device for generating a text summary, characterized in that: include: Acquisition module, used to obtain target text; A clustering module, configured to split the target text and generate multiple candidate sentences; Performing topic clustering on the candidate sentences to generate at least one cluster, wherein one topic corresponds to one cluster; An extraction module, configured to extract, for each cluster, a core sentence of the cluster from the target sentences included in the cluster; A generation module, configured to generate a text summary of the target text based on the core sentences of each cluster; The clustering module is further configured to: Obtaining the integrity parameter of the cluster; Obtaining homogeneity parameters of the cluster and the candidate sentence; Obtaining semantic weights of co-occurring words between the cluster and the target sentence included therein; Obtaining, according to the completeness parameter, the homogeneity parameter, and the semantic weight, an occurrence probability of the candidate sentence in the cluster; The cluster with the highest occurrence probability is selected and determined as the target cluster to which the candidate sentence belongs.

11. The apparatus according to claim 10, wherein the clustering module is further configured to: Obtaining a first number of target sentences currently included in the cluster; Obtaining a second number of the candidate sentences in the target text; The integrity parameter is determined based on the first number and the second number.

12. The apparatus according to claim 10, wherein the clustering module is further configured to: Obtain the first word frequency of any participle appearing in the candidate sentence; Obtaining the second word frequency of any one of the participles appearing in the cluster and the total number of words in the cluster; Obtaining the first weight of any word in each cluster; The homogeneity parameter is determined according to the first word frequency, the second word frequency, the total number of words, the first weight, and the size of the dictionary.

13. The apparatus according to claim 10, wherein the clustering module is further configured to: The second word frequency of the co-occurring words in the target sentence in each of the clusters is obtained, and the semantic weight is determined according to the second word frequency of each of the co-occurring words.

14. The apparatus according to claim 13, wherein the clustering module is further configured to: Obtaining a sub-semantic weight between any two co-occurring words according to the second word frequencies of the two co-occurring words; The sub-semantic weights between all the arbitrary two co-occurring words are summed to obtain the semantic weight.

15. The device according to claim 14, characterized in that The following formula is also used to determine the sub-semantic weight: Among them, the represents the sub-semantic weight; represents the target sentence; Indicates co-occurring words The second word frequency appearing in the target sentence; Indicates co-occurring words The second word frequency that appears in the target sentence.

16. An electronic device, characterized in that: Including memory and processor; The processor reads the executable program code stored in the memory to run a program corresponding to the executable program code, so as to implement the method according to any one of claims 1 to 9.

17. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

18. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 9.

Images