A text abstract generation method and device, an electronic device, and a storage medium

Abstract

The application discloses a text abstract generation method and device, an electronic device and a storage medium. The method comprises the following steps: obtaining a plurality of target texts to be processed; performing word segmentation on each target text to be processed based on a latest word segmentation dictionary to obtain a vocabulary set corresponding to each target text to be processed; processing the vocabulary set corresponding to each target text to be processed by using a latest word vector model to obtain a word vector corresponding to each target text to be processed; calculating the similarity between each two target texts to be processed based on the word vector corresponding to each target text to be processed; aggregating the target texts to be processed based on the similarity between the target texts to be processed to obtain a plurality of text categories; extracting corresponding keywords from the word segmentation set corresponding to each text category; and determining the text abstract corresponding to each text category from each target text belonging to the text category based on the corresponding keywords of the text category.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of text data processing, in particular to a text summary generation method and device, an electronic device and a storage medium. BACKGROUND

[0002] In the process of business development, a large amount of text data is often collected, for example, when customer service personnel provide customer service, they need to create corresponding work orders in the customer center system. Since the amount of text data is large and the number of words in the text data may also be large, in order to facilitate subsequent processing, it is often necessary to extract the text summary of the text data, so that the subsequent processing can be directly carried out by using the concise text summary.

[0003] The current method for generating a text summary mainly adopts extraction and generation. The extraction mainly screens the text summary of the text data from each sentence by analyzing the closeness and importance between each sentence in the text data. The generation mainly generates a short sentence that can express the main meaning of the text data as a text summary by processing the text data through a model.

[0004] However, both methods need to process each text data respectively, so the overall processing efficiency is low. Moreover, the text summary obtained by the extraction is limited in expressing the meaning of the text data, and the text summary obtained by the generation does not come from the text data, so the readability is prone to problems. SUMMARY

[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a text summary generation method and device, an electronic device and a storage medium to solve the problems of low efficiency and limited expression of meaning or poor readability in the prior art.

[0006] In order to achieve the above-mentioned purpose, the present application provides the following technical solutions:

[0007] The first aspect of the present application provides a text summary generation method, comprising:

[0008] obtaining a plurality of target texts to be processed in a current batch;

[0009] performing word segmentation on each target text to be processed based on the latest word segmentation dictionary to obtain a vocabulary set corresponding to each target text to be processed;

[0010] processing the vocabulary set corresponding to each target text to be processed by using the latest trained word vector model to obtain a word vector corresponding to each target text to be processed;

[0011] calculate similarity between each two of the target texts to be processed based on the word vectors corresponding to the target texts to be processed;

[0012] aggregate each of the target texts to be processed based on the similarity between each two of the target texts to be processed, to obtain a plurality of text categories;

[0013] extract keywords corresponding to each of the text categories from a word segmentation set corresponding to the text category; wherein the word segmentation set corresponding to a text category includes a word set corresponding to each of the target texts to be processed belonging to the text category;

[0014] For each of the text categories, determine a text summary corresponding to the text category from each of the target texts to be processed belonging to the text category based on the keywords corresponding to the text category.

[0015] Optionally, in the method for generating a text summary, before the step of segmenting each of the target texts to be processed based on the latest segmentation dictionary to obtain a word set corresponding to each of the target texts to be processed, the method further comprises:

[0016] perform first segmentation on each of the target texts to be processed using a segmentation algorithm to obtain an initial word set corresponding to each of the target texts to be processed;

[0017] compare the initial word set corresponding to each of the target texts to be processed with a word set of a stock target text to obtain each of the current new words;

[0018] update each of the current new words with a frequency greater than a preset frequency to the segmentation dictionary.

[0019] Optionally, in the method for generating a text summary, the method for training the word vector model comprises:

[0020] construct a base of the word vector model using external corpus;

[0021] perform additional training on the base of the word vector model using historical stock target texts to obtain an initial word vector model;

[0022] whenever a batch of the target texts to be processed is processed, perform additional training on the word vector model using the target texts to be processed in the current batch and each of the current new words obtained in the current batch to obtain the latest word vector model.

[0023] Optionally, in the method for generating a text summary, the step of aggregating each of the target texts based on the similarity between each two of the target texts to obtain a plurality of text categories comprises:

[0024] taking each of the target texts as a current classification object;

[0025] combining each of the current classification objects and the current classification object having the greatest similarity with the current classification object to obtain a plurality of current text pairs;

[0026] sorting each of the current text pairs in descending order according to the similarity of the current text pairs to obtain a sorting result, wherein the similarity of the current text pairs is the similarity between the two current classification objects constituting the current text pairs;

[0027] according to the arrangement order in the sorting result, if the two current classification objects in each of the current text pairs have not been classified, classifying the two current classification objects in the current text pair into a new text category;

[0028] if the two current classification objects in the current text pair have been classified, maintaining the two current classification objects in the current text pair in the text categories to which the two current classification objects belong, respectively;

[0029] if only one of the current classification objects in the current text pair has been classified, classifying the current classification object in the current text pair that has not been classified into the text category to which the current classification object in the current text pair that has been classified belongs.

[0030] Optionally, in the method for generating a text summary, the method further comprises:

[0031] when there is no target text that has not been classified, screening the text categories from the current text categories, wherein the text categories contain a number of target texts greater than a preset number;

[0032] for each of the screened text categories, canceling the classification of each of the target texts in the text category, and taking each of the target texts as a current classification object, and returning to the step of combining each of the current classification objects and the current classification object having the greatest similarity with the current classification object to obtain a plurality of current text pairs.

[0033] Optionally, in the method for generating a text summary, the determining, for each text category, a text summary corresponding to the text category from each of the target texts belonging to the text category based on the keywords corresponding to the text category comprises:

[0034] The target punctuation is used to divide each of the target texts belonging to the text category into sentences to obtain a plurality of short sentences corresponding to the text category;

[0035] The number of keywords contained in each of the short sentences and the proportion of keywords are determined.

[0036] Each of the short sentences is sorted in descending order based on the number of keywords contained and the proportion of keywords.

[0037] Each of the short sentences ranked in the top N positions is determined as the text summary corresponding to the text category.

[0038] The second aspect of the present application provides a device for generating a text summary, comprising:

[0039] An acquisition unit is configured to acquire a plurality of target texts to be processed in a current batch.

[0040] A first tokenization unit is configured to tokenize each of the target texts to be processed based on a latest tokenization dictionary to obtain a vocabulary set corresponding to each of the target texts to be processed.

[0041] A feature processing unit is configured to process each of the vocabulary sets corresponding to the target texts to be processed using a latest trained word vector model to obtain a word vector corresponding to each of the target texts to be processed.

[0042] A calculation unit is configured to calculate the similarity between each two of the target texts to be processed based on the word vectors corresponding to each of the target texts to be processed.

[0043] A clustering unit is configured to aggregate each of the target texts to be processed based on the similarity between each two of the target texts to be processed to obtain a plurality of text categories.

[0044] An extraction unit is configured to extract, from each of the tokenization sets corresponding to the text categories, keywords corresponding to each of the text categories, wherein the tokenization set corresponding to one of the text categories comprises the vocabulary sets corresponding to each of the target texts belonging to the text category.

[0045] A generation unit is configured to determine, for each of the text categories, a text summary corresponding to the text category from each of the target texts belonging to the text category based on the keywords corresponding to the text category.

[0046] Optionally, in the text summary generation device, further comprising:

[0047] A second word segmentation unit configured to perform first word segmentation on each of the target texts to be processed by using a word segmentation algorithm to obtain an initial vocabulary set corresponding to each of the target texts to be processed;

[0048] A comparison unit configured to compare the initial vocabulary set corresponding to each of the target texts to be processed with a vocabulary set of the inventory target text to obtain each of the current new words;

[0049] An updating unit configured to update the current word segmentation of each of the current new words with a frequency greater than a preset frequency to the word segmentation dictionary.

[0050] Optionally, in the text summary generation device, further comprising:

[0051] A base construction unit configured to construct a base of the word vector model by using external corpus;

[0052] A first training unit configured to perform additional training on the base of the word vector model by using historical inventory target texts to obtain an initial word vector model;

[0053] A second training unit configured to perform additional training on the word vector model by using the target texts to be processed in a current batch and each of the current new words obtained in the current batch each time a batch of the target texts to be processed is processed to obtain a latest word vector model.

[0054] Optionally, in the text summary generation device, the clustering unit comprises:

[0055] A first object determination unit configured to determine each of the target texts to be processed as a current classification object;

[0056] A combination unit configured to combine each of the current classification objects and the current classification object with the greatest similarity therebetween respectively to obtain a plurality of groups of current text pairs;

[0057] A first sorting unit configured to sort each group of the current text pairs in descending order according to the similarity of each group of the current text pairs to obtain a sorting result, wherein the similarity of the current text pair is the similarity between the two current classification objects constituting the current text pair;

[0058] The dividing unit is configured to, according to the arrangement order in the sorting result, sequentially divide, for each current text pair, the two current classification objects in the current text pair into a new text category if neither of the two current classification objects in the current text pair has been classified, maintain the two current classification objects in the current text pair in the text categories to which the two current classification objects belong respectively if both of the two current classification objects in the current text pair have been classified, and divide the current classification object in the current text pair that has not been classified into the text category to which the current classification object in the current text pair that has been classified belongs.

[0059] Optionally, in the text summary generation device, the generation unit further includes:

[0060] The screening unit is configured to, when there is no unclassified target text, screen, from each current text category, the text category that contains a number of target texts greater than a preset number.

[0061] The second object determining unit is configured to, for each screened text category, cancel the classification of each target text in the text category, and return each target text as a current classification object to the combination unit to perform the combination of each current classification object and the current classification object with the greatest similarity to each current classification object to obtain a plurality of current text pairs.

[0062] Optionally, in the text summary generation device, the generation unit includes:

[0063] The sentence dividing unit is configured to, for each text category, divide each target text belonging to the text category into a plurality of short sentences with a target punctuation mark.

[0064] The index determining unit is configured to determine the number of keywords contained in each short sentence and the proportion of keywords.

[0065] The second sorting unit is configured to sort each short sentence in descending order based on the number of keywords contained and the proportion of keywords.

[0066] The selection unit is configured to determine each short sentence ranked in the top N positions as a text summary corresponding to the text category.

[0067] The third aspect of the present application provides an electronic device, including:

[0068] a memory and a processor;

[0069] The memory is configured to store a program.

[0070] The processor is configured to execute the program, and the program, when executed, is configured to implement the method for generating a text summary according to any one of the preceding embodiments.

[0071] The fourth aspect of the present application provides a computer storage medium configured to store a computer program, and the computer program, when executed, is configured to implement the method for generating a text summary according to any one of the preceding embodiments.

[0072] The method for generating a text summary provided by the embodiments of the present application comprises the following steps: obtaining a plurality of target texts to be processed in a current batch; performing word segmentation on each target text to be processed based on a latest word segmentation dictionary to obtain a vocabulary set corresponding to each target text to be processed; processing the vocabulary set corresponding to each target text to be processed by using a word vector model that is trained most recently to obtain a word vector corresponding to each target text to be processed; calculating a similarity between each two target texts to be processed based on the word vectors corresponding to the target texts to be processed; aggregating each target text to be processed based on the similarity between each two target texts to be processed to obtain a plurality of text categories; extracting a keyword corresponding to each text category from a word segmentation set corresponding to each text category, wherein the word segmentation set corresponding to each text category comprises the vocabulary set corresponding to each target text to be processed belonging to the text category; and determining a text summary corresponding to each text category from each target text to be processed belonging to the text category based on the keyword corresponding to the text category. In this way, by aggregating the texts into a plurality of text categories and then screening a text summary that accurately expresses the main meaning of the text data from the text data in the text category based on the keyword, the batch unified processing of the text data is realized, the processing efficiency is improved, and the readability of the text summary is ensured. BRIEF DESCRIPTION OF DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of the provided drawings.

[0074] Figure 1 A flowchart of the method for generating a text summary provided by the embodiments of the present application;

[0075] Figure 2 A flowchart of the method for updating a word segmentation dictionary provided by the embodiments of the present application;

[0076] Figure 3 A flowchart of a method for training a word vector model according to an embodiment of the present application is provided.

[0077] Figure 4 A flowchart of a method for aggregating target texts according to an embodiment of the present application is provided.

[0078] Figure 5 A flowchart of a method for determining a text summary from target texts according to an embodiment of the present application is provided.

[0079] Figure 6 An architecture diagram of a text summary generation device according to an embodiment of the present application is provided.

[0080] Figure 7 An architecture diagram of an electronic device according to an embodiment of the present application is provided. DETAILED DESCRIPTION

[0081] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0082] In the present application, the relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.

[0083] The present application provides a method for generating a text summary, as shown in the following steps: Figure 1

[0084] S101, obtaining a plurality of target texts to be processed in a current batch.

[0085] The target file to be processed refers to the current text to be processed of a specified type, for example, a work order constructed by a customer service personnel, etc.

[0086] ​S102, based on the current latest word segmentation dictionary, performing word segmentation on each target text to be processed to obtain a word set corresponding to each target text to be processed.

[0087] It should be noted that the constructed word segmentation dictionary may have limited data, so the obtained words are limited, and some new words may appear as the business and times develop, so in order to ensure the accuracy of word segmentation, the word segmentation dictionary is updated in the embodiment of the present application, so the latest updated word segmentation dictionary is used for word segmentation.

[0088] Optionally, in another embodiment of the present application, the word segmentation dictionary is updated before step S102. As shown in the figure, Figure 2 The embodiment of the present application provides a word segmentation dictionary updating method, which comprises:

[0089] S201, using a word segmentation algorithm to perform first word segmentation on each target text to be processed to obtain an initial word set corresponding to each target text to be processed.

[0090] It should be noted that the target text to be processed may include new words, and if the word segmentation dictionary is directly used for word segmentation, the accuracy of the obtained words will be affected, so in order to improve the accuracy of word segmentation and to facilitate more accurate word segmentation of subsequent processed texts, in the embodiment of the present application, the new words included in the current batch of target texts to be processed need to be determined, so the included words need to be first segmented to obtain an initial word set corresponding to each target text to be processed.

[0091] S202, comparing the initial word set corresponding to each target text to be processed with the word set of the inventory target text to obtain each current new word.

[0092] The inventory target text refers to a large amount of target text that has been previously processed or previously generated and has not been processed. The word segmentation dictionary is mainly obtained by processing the inventory target text, so in the embodiment of the present application, each current new word is obtained by comparing the initial word set corresponding to each target text to be processed with the word set of the inventory target text.

[0093] S203, updating each current word with a frequency greater than a preset frequency to the word segmentation dictionary.

[0094] It should be noted that for the new word with less frequency, since it is explained that it appears less in the target text, it is less used, and updating it to the segmentation dictionary will affect the processing amount in the segmentation process, therefore in the embodiment of the application, only the current segmentation with the frequency greater than the preset frequency is updated to the segmentation dictionary.

[0095] S103, using the current latest trained word vector model, respectively processing the word set corresponding to each target text to be processed, obtaining the word vector corresponding to each target text to be processed.

[0096] Similarly, since new words will continue to appear, in order to enable the word vector model to accurately process new words, in the embodiment of the application, the word vector model will be continuously trained by using new corpus, therefore when processing the current batch, the current latest trained word vector model is used for processing.

[0097] Optionally, the embodiment of the application provides a training method of a word vector model, as shown in Figure 3 , comprising:

[0098] S301, using external corpus to construct the base of the word vector model.

[0099] It should be noted that in order to ensure the accuracy of the output result of the word vector model, usually a word vector model is trained for a certain type of text generated in the business process. Therefore, the external corpus refers to the text not generated in the business process. Since the technology of constructing the word vector by using part of the external corpus is relatively mature, a reliable word vector model can be obtained.

[0100] Optionally, the word vector model can be constructed by word2vec.

[0101] S302, using the historical inventory target text to additionally train the base of the word vector model, obtaining the initial word vector model.

[0102] In order to enable the word vector model to accurately process the target text, it is necessary to additionally train the base of the word vector model obtained in the previous step by using the historical inventory text, thereby obtaining the initial word vector model. And for the word vector model, it will be continuously updated by executing step S303.

[0103] Among them, the historical inventory target text and the inventory target text in step S202 can be the same target text set, or can be different target text sets.

[0104] S303, when processing a batch of target texts to be processed, using the current batch of target texts to be processed and each current new word obtained in the current batch, performing additional training on the word vector model to obtain the latest word vector model.

[0105] It should be noted that after obtaining the trained word vector model in step S303, the subsequent will be continuously used to process each batch of target texts to be processed. Each time the target text to be processed may include a new word, so as to enable the word vector model to adapt to the new word, so as to obtain more accurate results. Therefore, in the embodiment of the application, each time a batch of target texts to be processed is processed, the word vector model is additionally trained using each new word obtained in the current batch and in combination with the current matching target text to be processed, thereby updating the word vector model.

[0106] Optionally, since it is mainly to enable the word vector model to adapt to the new word, the word vector model can be additionally trained only using the target text to be processed containing the new word in the current batch of target texts to be processed.

[0107] Therefore, it can be seen that in the embodiment of the application, the training of the word vector model adopts a pyramid mode for multi-level additional training.

[0108] Specifically, the word vector model can be adjusted according to the input result after each execution of step S103. Alternatively, the word vector model can be additionally trained using each obtained target text to be processed after execution of step S101.

[0109] S104, based on the word vector corresponding to each target text to be processed, calculating the similarity between each two target texts to be processed.

[0110] Since in the embodiment of the application, the target texts to be processed need to be clustered and then uniformly processed in the dimension of the category, the similarity between each two target texts to be processed needs to be calculated to enable the target texts to be processed to be clustered based on the similarity between each two target texts to be processed.

[0111] Optionally, the cosine similarity calculation method can be used to calculate the similarity between each two target texts to be processed. Therefore, compared with distance measurement, the cosine similarity pays more attention to the difference in direction between two vectors, and thus can better reflect the similarity between each two target texts to be processed.

[0112] S105, based on the similarity between each two target texts to be processed, aggregating each target text to be processed to obtain a plurality of text categories.

[0113] Specifically, the clustering algorithm can be used to aggregate the target texts based on the similarity between each two target texts, and other aggregation manners can also be used.

[0114] Optionally, in another embodiment of the present application, a specific implementation of step S105 includes the following steps, as shown in the following table. Figure 4

[0115] S401, taking each target text as a current classification object.

[0116] S402, combining each current classification object and the current classification object with the highest similarity to it, respectively, to obtain a plurality of groups of current text pairs.

[0117] That is, for each current classification object, another current classification object with the highest similarity to it is screened out, and then the two form a group of current text pairs. It should be noted that for the current classification object A, the object with the highest similarity to it is the current classification object B, but for the current classification object B, the object with the highest similarity to it is not necessarily the current classification object A, so in the embodiment of the present application, each current classification object is classified to form its corresponding current text pair, so one current classification object can exist in multiple groups of current text pairs.

[0118] S403, sorting each group of current text pairs in descending order according to the similarity of each group of current text pairs to obtain a sorting result.

[0119] The similarity of the current text pair is the similarity between the two current classification objects that form the current text pair.

[0120] S404, according to the arrangement order in the sorting result, judging whether the two current classification objects in each group of current text pairs are not classified.

[0121] Since the greater the similarity of the target text, the more it should be classified into the same class first, in the embodiment of the present application, each group of current text pairs is processed in turn according to the arrangement order in the sorting result.

[0122] If it is judged that the two current classification objects in the current text pair are not classified, step S405 is executed. If it is judged that the two current classification objects in the current text pair are not all not classified, step S406 is executed.

[0123] S405, classifying the two current classification objects in the current text pair into a new text class.

[0124] ​Since the two current classification objects in the current text pair are not classified, it indicates that the two current classification objects are not similar to other current classification objects, and the two current classification objects are the most similar, so a new text category is created, and the two current classification objects are classified into the text category.

[0125] S406, judging whether the two current classification objects in the current text pair are both classified.

[0126] If it is judged that the two current classification objects in the current text pair are both classified, it indicates that it is not necessary to re-divide them, so step S407 can be executed at this time. If it is judged that the two current classification objects in the current text pair are not both classified, and since it is judged in step S404 that the two current classification objects are not both not classified, it indicates that one of the two current classification objects in the current text pair is classified, and the other is not classified, so step S408 is executed at this time.

[0127] S407, maintaining the two current classification objects in the current text pair in the text categories to which they belong, respectively.

[0128] S408, classifying the current classification object in the current text pair that is not classified into the text category to which the current classification object that is classified in the current text pair belongs.

[0129] In order to make the classification more accurate, in an embodiment of the present application, a multi-level aggregation method is used, so in another embodiment of the present application, the method can further include:

[0130] When there is no unclassified target text to be processed, from the current text categories, a text category containing a number of target texts to be processed greater than a preset number is selected.

[0131] For each selected text category, the classification of each target text to be processed in the text category is cancelled, and each target text to be processed is taken as a current classification object, and then step S402 is returned to be executed.

[0132] That is, in an embodiment of the present application, after the classification is performed each time, a text category containing a number of target texts to be processed greater than a preset number is further selected. Since the number of these categories is relatively large, the error can also be relatively large, so they are selected out, and then for each selected text category, the target text to be processed in the text category is classified as a current classification object. Figure 4The method shown is repeated, and the text is divided again. If there are still text categories in the re-divided text categories where the number of unprocessed target texts is greater than the preset number, the text will be divided again until the number of unprocessed target texts in all text categories is no greater than the preset number.

[0133] S106. Extract the keywords corresponding to each text category from the word segmentation set corresponding to each text category.

[0134] Among them, the word segmentation set corresponding to a text category includes the vocabulary set corresponding to each target text to be processed belonging to the text category.

[0135] Since the content expressed by the target texts to be processed within the same text category is relatively related, all target texts to be processed within the same text category are processed uniformly in this embodiment. Therefore, the vocabulary sets corresponding to each target text to be processed belonging to the same text category are used to form a word segmentation set corresponding to that text category. Then, keywords are extracted from it, and the sentence that best expresses the meaning of all target texts to be processed within the text category is used as the text summary based on the keywords.

[0136] Alternatively, the TF-IDF algorithm can be used to extract keywords corresponding to each text category from the word segmentation set corresponding to each text category.

[0137] S107. For each text category, based on the keywords corresponding to the text category, determine the text summary corresponding to the text category from each target text to be processed belonging to the text category.

[0138] Since keywords are the most important words in expressing the main meaning of the target text, the target text is divided into multiple short sentences. Then, by using the keywords corresponding to the text category, the short sentence that best expresses the main meaning of the target text can be identified from each target text belonging to the text category, and this short sentence is identified as the text summary corresponding to the text category.

[0139] It should be noted that since the text categories are processed simultaneously, the resulting text summaries for each text category may include multiple texts.

[0140] Optionally, in another embodiment of this application, one specific implementation of step S107 is as follows: Figure 5 As shown, it includes:

[0141] S501. For each text category, segment each target text belonging to that text category into sentences using the target punctuation marks to obtain multiple short sentences corresponding to the text category.

[0142] The target punctuation mark can be a period, a comma, or the like.

[0143] S502, determine the number of keywords contained in each short sentence and the proportion of keywords.

[0144] Since the keywords are the most important words in the target text to be processed to express the main meaning, the more the number of keywords contained in a short sentence and the proportion of keywords, the more it can outline the main meaning expressed by the target text to be processed. Therefore, in the embodiment of the application, the number of keywords contained in each short sentence and the proportion of keywords are counted, and then step S503 is performed.

[0145] S503, sort each short sentence in descending order based on the number of keywords contained and the proportion of keywords.

[0146] Alternatively, it can be sorted in order from more to less in the number of keywords contained. If the number of keywords contained in the short sentence is the same, the proportion of keywords is considered.

[0147] S504, determine each short sentence ranked in the top N positions as the text summary corresponding to the text category.

[0148] The embodiment of the application provides a text summary generation method. A plurality of target texts to be processed in a current batch are obtained, and then each target text to be processed is segmented based on the current latest segmentation dictionary to obtain a vocabulary set corresponding to each target text to be processed. Each target text to be processed is processed by using a current latest trained word vector model to obtain a word vector corresponding to each target text to be processed. Then, the similarity between each two target texts to be processed is calculated based on the word vector corresponding to each target text to be processed, so that each target text to be processed can be aggregated based on the similarity between each two target texts to be processed to obtain a plurality of text categories. Then, the keywords corresponding to each text category are extracted from the segmentation set corresponding to each text category; wherein the segmentation set corresponding to a text category includes the vocabulary set corresponding to each target text to be processed belonging to the text category, and finally, for each text category, the text summary corresponding to the text category is determined from each target text to be processed belonging to the text category based on the keywords corresponding to the text category. Thus, by aggregating each text into a plurality of text categories, and then screening the text summary accurately expressing the main meaning of the text data from the text data in the text category through keywords, not only the batch unified processing of text data is realized, the processing efficiency is improved, but also the readability of the text summary is ensured.

[0149] Another embodiment of the application provides a text summary generation device, as shown in Figure 6 the following units:

[0150] The acquisition unit 601 is configured to acquire a plurality of target texts to be processed in a current batch.

[0151] The first segmentation unit 602 is configured to perform segmentation on each target text to be processed based on a latest segmentation dictionary to obtain a vocabulary set corresponding to each target text to be processed.

[0152] The feature processing unit 603 is configured to process the vocabulary set corresponding to each target text to be processed respectively by using a latest trained word vector model to obtain a word vector corresponding to each target text to be processed.

[0153] The calculation unit 604 is configured to calculate a similarity between each two target texts to be processed based on the word vector corresponding to each target text to be processed.

[0154] The clustering unit 605 is configured to aggregate each target text to be processed based on the similarity between each two target texts to be processed to obtain a plurality of text categories.

[0155] The extraction unit 606 is configured to extract a keyword corresponding to each text category from a segmentation set corresponding to each text category respectively. The segmentation set corresponding to a text category includes the vocabulary set corresponding to each target text to be processed belonging to the text category.

[0156] The generation unit 607 is configured to determine a text summary corresponding to each text category from each target text to be processed belonging to the text category based on the keyword corresponding to the text category for each text category.

[0157] Optionally, the text summary generation apparatus provided in another embodiment of the present application further includes:

[0158] The second segmentation unit is configured to perform first segmentation on each target text to be processed by using a segmentation algorithm to obtain an initial vocabulary set corresponding to each target text to be processed.

[0159] The comparison unit is configured to compare the initial vocabulary set corresponding to each target text to be processed with a vocabulary set of a stock target text to obtain each current new word.

[0160] The update unit is configured to update each current word with a frequency greater than a preset frequency in the segmentation dictionary to a current segmentation.

[0161] Optionally, the text summary generation apparatus provided in another embodiment of the present application further includes:

[0162] The base construction unit is configured to construct a base of the word vector model by using an external corpus.

[0163] The first training unit is configured to perform additional training on the base of the word vector model by using the historical inventory target text, so as to obtain an initial word vector model.

[0164] The second training unit is configured to perform additional training on the word vector model by using the current batch of target text to be processed and each current new word obtained in the current batch each time a batch of target text to be processed is processed, so as to obtain a latest word vector model.

[0165] Optionally, in the text summary generation apparatus provided by another embodiment of the present application, the clustering unit comprises:

[0166] The first object determination unit is configured to take each target text to be processed as a current classification object.

[0167] The combination unit is configured to combine each current classification object and the current classification object with the largest similarity therebetween respectively, so as to obtain a plurality of groups of current text pairs.

[0168] The first sorting unit is configured to sort each group of current text pairs in descending order of the similarity of each group of current text pairs, so as to obtain a sorting result.

[0169] The similarity of the current text pair is the similarity between the two current classification objects constituting the current text pair.

[0170] The division unit is configured to, according to the arrangement order in the sorting result, sequentially divide each group of current text pairs, if the two current classification objects in the current text pair have not been classified, the two current classification objects in the current text pair into a new text category. If the two current classification objects in the current text pair have been classified, the two current classification objects in the current text pair are maintained in the text categories to which they belong respectively. If only one of the current classification objects in the current text pair has been classified, the current classification object in the current text pair that has not been classified is divided into the text category to which the current classification object in the current text pair that has been classified belongs.

[0171] Optionally, in the text summary generation apparatus provided by another embodiment of the present application, the apparatus further comprises:

[0172] The screening unit is configured to, when there is no target text to be processed that has not been classified, screen out, from the current text categories, a text category containing a number of target texts to be processed greater than a preset number.

[0173] The second object determining unit is configured to cancel the category division of each to-be-processed target text in the text category and return each to-be-processed target text as a current classification object for each text category.

[0174] Optionally, in the text summary generation apparatus provided in another embodiment of the present application, the generation unit comprises:

[0175] The sentence dividing unit is configured to divide each to-be-processed target text belonging to the text category into multiple short sentences with the target punctuation mark for each text category, to obtain multiple short sentences corresponding to the text category.

[0176] The index determining unit is configured to determine the number of keywords contained in each short sentence and the proportion of keywords.

[0177] The second sorting unit is configured to sort each short sentence in descending order based on the number of keywords contained and the proportion of keywords.

[0178] The selection unit is configured to determine each short sentence ranked in the top N positions as the text summary corresponding to the text category.

[0179] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can be correspondingly referred to the corresponding steps in the above method embodiments, which will not be described here.

[0180] Another embodiment of the present application provides an electronic device, as shown in the figure, comprising: Figure 7 The memory 701 and the processor 702.

[0181] The memory 701 and the processor 702.

[0182] The memory 701 is configured to store a program.

[0183] The processor 702 is configured to execute the program stored in the memory 701, and the program is executed to specifically implement the text summary generation method provided in any one of the above embodiments.

[0184] Another embodiment of the present application provides a computer storage medium for storing a computer program, and the computer program is executed to implement the text summary generation method provided in any one of the above embodiments.

[0185] Computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disks (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. According to the definitions provided herein, computer readable media does not include transitory computer readable medium, such as a modulated data signal and a carrier wave.

[0186] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality, without reference to a specific

[0187] The above description of disclosed embodiments provides enabling disclosure sufficient for one of ordinary skill in the art to practice the application. Various modifications to those embodiments will be readily apparent to those of ordinary skill in the art, and generic principles defined herein can be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating text summaries, characterized in that, include: Retrieve multiple target texts to be processed in the current batch; Based on the latest word segmentation dictionary, each of the target texts to be processed is segmented into words to obtain the vocabulary set corresponding to each target text to be processed. Using the latest trained word vector model, the vocabulary set corresponding to each of the target texts to be processed is processed to obtain the word vectors corresponding to each of the target texts to be processed. Based on the word vectors corresponding to each of the target texts to be processed, the similarity between each pair of target texts to be processed is calculated. Based on the similarity between any two target texts to be processed, each target text to be processed is aggregated to obtain multiple text categories; Keywords corresponding to each text category are extracted from the word segmentation set corresponding to each text category; wherein, the word segmentation set corresponding to a text category includes the vocabulary set corresponding to each target text to be processed belonging to the text category; For each text category, based on the keywords corresponding to the text category, a text summary corresponding to the text category is determined from each of the target texts to be processed belonging to the text category; Specifically, determining the text summary corresponding to each text category from each target text belonging to that text category based on the keywords corresponding to that text category includes: For each of the text categories, the target texts belonging to the text category are segmented into sentences using the target punctuation marks to obtain multiple short sentences corresponding to the text category; Determine the number of keywords and the percentage of keywords contained in each of the short sentences; Based on the number and proportion of keywords contained, the short sentences are sorted in descending order; The top N short phrases are identified as the text summaries corresponding to the text category. The training method for the word vector model includes: The basis of the word vector model is constructed using external corpora; The word vector model is supplemented by training the basis of the word vector model using historical target text to obtain the initial word vector model. Whenever a batch of target texts to be processed is processed, the word vector model is further trained using the target texts to be processed in the current batch and each new word obtained in the current batch, so as to obtain the latest word vector model.

2. The method according to claim 1, characterized in that, Before segmenting each of the target texts to be processed based on the latest word segmentation dictionary to obtain the vocabulary set corresponding to each target text to be processed, the method further includes: The word segmentation algorithm is used to perform initial word segmentation on each of the target texts to be processed, thereby obtaining an initial vocabulary set corresponding to each of the target texts to be processed. The initial vocabulary set corresponding to each of the target texts to be processed is compared with the vocabulary set of the existing target texts to obtain each current new word; The current new words whose frequency of occurrence is greater than a preset frequency are updated in the word segmentation dictionary.

3. The method according to claim 1, characterized in that, The process involves aggregating the target texts based on the similarity between any two target texts to obtain multiple text categories, including: Each of the target texts to be processed is taken as the current classification object; Each current category object and the current category object with the highest similarity to it are combined to obtain multiple sets of current text pairs; The current text pairs in each group are sorted in descending order according to their similarity to obtain the sorting result; wherein, the similarity of the current text pair is the similarity between the two current classification objects that make up the current text pair; According to the sorting order in the sorting results, for each group of current text pairs, if neither of the two current classification objects in the current text pair has been classified into a category, the two current classification objects in the current text pair will be classified into a new text category. If both current category objects in the current text pair have been classified, then the two current category objects in the current text pair are kept in their respective text categories. If only one of the current category objects in the current text pair has been categorized, then the current category object that has not been categorized in the current text pair will be assigned to the current text pair, and the current category object that has been categorized will be assigned to the text category to which it belongs.

4. The method according to claim 3, characterized in that, Also includes: When there are no unclassified target texts to be processed, select text categories from the current text categories that contain more than a preset number of target texts to be processed; For each of the selected text categories, the category division of each target text to be processed in the text category is canceled, and each target text to be processed is taken as the current category object. Then, the process of combining each current category object with the current category object with the highest similarity to it is returned to obtain multiple sets of current text pairs.

5. A text summarization generating apparatus, characterized in that, include: The acquisition unit is used to acquire multiple target texts to be processed in the current batch; The first word segmentation unit is used to segment each of the target texts to be processed based on the latest word segmentation dictionary, so as to obtain the vocabulary set corresponding to each target text to be processed. The feature processing unit is used to process the vocabulary set corresponding to each of the target texts to be processed using the latest trained word vector model to obtain the word vectors corresponding to each target text to be processed. The calculation unit is used to calculate the similarity between every two target texts based on the word vectors corresponding to each of the target texts to be processed; Clustering unit is used to aggregate each of the target texts to be processed based on the similarity between each pair of target texts to be processed, so as to obtain multiple text categories; An extraction unit is used to extract keywords corresponding to each text category from the word segmentation set corresponding to each text category; wherein, the word segmentation set corresponding to a text category includes the vocabulary set corresponding to each target text to be processed belonging to the text category; The generation unit is configured to, for each text category, determine a text summary corresponding to the text category from each of the target texts to be processed belonging to the text category, based on the keywords corresponding to the text category; A basis construction unit is used to construct the basis of the word vector model using external corpus; The first training unit is used to supplement the base of the word vector model with historical target text to obtain the initial word vector model. The second training unit is used to further train the word vector model using the current batch of target texts and each new word obtained in the current batch whenever a batch of target texts to be processed is processed, so as to obtain the latest word vector model. The generation unit includes: The sentence segmentation unit is used to segment each of the target texts to be processed belonging to each of the text categories into sentences using target punctuation marks, thereby obtaining multiple short sentences corresponding to the text category. The indicator determination unit is used to determine the number of keywords and the percentage of keywords contained in each of the short sentences; The second sorting unit is used to sort the short sentences in descending order based on the number of keywords and the proportion of keywords contained therein. The selection unit is used to determine the top N short sentences as the text summary corresponding to the text category.

6. The apparatus according to claim 5, characterized in that, Also includes: The second word segmentation unit is used to perform initial word segmentation on each of the target texts to be processed using a word segmentation algorithm to obtain an initial vocabulary set corresponding to each of the target texts to be processed. The comparison unit is used to compare the initial vocabulary set corresponding to each of the target texts to be processed with the vocabulary set of the existing target texts to obtain each current new word; An update unit is used to update the word segmentation dictionary with the current new words whose frequency of occurrence is greater than a preset frequency.

7. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is used to execute the program, which, when executed, is specifically used to implement the text summary generation method as described in any one of claims 1 to 4.

8. A computer storage medium, characterized in that, Used to store a computer program, which, when executed, is used to implement the text digest generation method as described in any one of claims 1 to 4.

Images