Abstract generation method, device, equipment and medium based on natural language processing

Abstract

The present application relates to the field of artificial intelligence technology, and provides a method, device, equipment and medium for generating a summary based on natural language processing, wherein the method comprises: splitting the original text to obtain at least one sentence and the words in the sentence; determining the summary probability of the words; taking the word corresponding to the maximum value of the summary probability as the first word; taking the sentence in which the first word is located as the first sentence; taking the word after the first word in the first sentence as the second word, and taking the word in the sentence after the first sentence as the third word; based on the combined summary probability of the first word and the second word, selecting at least one fourth word from the second word; based on the combined summary probability of the first word, the fourth word and the third word, selecting at least one fifth word from the third word; obtaining a text sequence composed of the first word, the fourth word and the fifth word, and obtaining a summary of the original text. The present application can improve the accuracy of generating a summary.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and mainly to a method, device, equipment and medium for generating a summary based on natural language processing. Background Art

[0002] In the era of Internet big data with information explosion, text summarization is widely used in today's industry, such as news headline generation, scientific literature summary generation, search result fragment generation, product review summary, etc. Text summarization can be mainly divided into extractive summarization and generative summarization. Among them, extractive summarization directly extracts words, phrases or sentences from the original text, and then combines them into a complete document in a suitable way to form a summary. However, the sentences of extractive summarization are usually lengthy and the reading experience is poor. Generative summarization uses natural language processing and generation technology, and the computer tries to understand the general meaning of the original text, and then generates a summary in the way that humans write. The summary may contain words, phrases or sentences that never appear in the original text, resulting in different semantic information from the original text. Summary of the invention

[0003] The embodiments of the present application provide a method, apparatus, device and medium for generating a summary based on natural language processing, which can improve the accuracy of generating a summary.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a summary based on natural language processing, wherein:

[0005] Splitting the original text to obtain at least one sentence and words in the sentence;

[0006] determining a summary probability of the word;

[0007] The word corresponding to the maximum value of the summary probability is used as the first word;

[0008] The sentence containing the first word is taken as the first sentence;

[0009] using the word after the first word in the first sentence as the second word, and using the word in the sentence after the first sentence as the third word;

[0010] Selecting at least one fourth character from the second character based on the combined summary probability of the first character and the second character;

[0011] Based on the combined summary probability of the first word, the fourth word, and the third word, selecting at least one fifth word from the third word;

[0012] A text sequence consisting of the first character, the fourth character, and the fifth character is obtained to obtain a summary of the original text.

[0013] In a second aspect, an embodiment of the present application provides a summary generation device based on natural language processing, wherein:

[0014] A splitting unit, used for splitting the original text to obtain at least one sentence and words in the sentence;

[0015] a determination unit, configured to determine the summary probability of the word; use the word corresponding to the maximum value of the summary probability as the first word; use the sentence in which the first word is located as the first sentence; use the word after the first word in the first sentence as the second word, and use the word in the sentence after the first sentence as the third word;

[0016] a selection unit, configured to select at least one fourth word from the second word based on the combined summary probability of the first word and the second word; and select at least one fifth word from the third word based on the combined summary probability of the first word, the fourth word, and the third word;

[0017] The composition unit is used to obtain a text sequence composed of the first character, the fourth character, and the fifth character to obtain a summary of the original text.

[0018] In a third aspect, an embodiment of the present application provides a computer device, comprising a processor, a memory, a communication interface, and a computer program, wherein the computer program is stored in the memory, the computer program is configured to be executed by the processor, and the computer program includes instructions for some or all of the steps described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program enables a computer to execute to implement some or all of the steps described in the first aspect.

[0020] Implementing the embodiments of the present application will have the following beneficial effects:

[0021] After adopting the above-mentioned summary generation method, device, equipment and medium based on natural language processing, the original text is first split to obtain at least one sentence and the words in each sentence. Then determine the summary probability of each word. Then the word corresponding to the maximum value of the summary probability is used as the first word, and the sentence where the first word is located is used as the first sentence, the word after the first word in the first sentence is used as the second word, and the word in the sentence after the first sentence is used as the third word. Then, based on the combined summary probability of the first word and the second word, at least one fourth word is selected from the second word. Based on the combined summary probability of the first word and the fourth word, at least one fifth word is selected from the third word. Finally, a text sequence consisting of the first word, the second word and the third word is obtained to obtain a summary of the original text. In this way, a summary is generated based on the combined summary probability of each word in the original text and the words selected in the previous text, which can improve the accuracy of generating the summary. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] in:

[0024] Figure 1 A flowchart of a method for generating a summary based on natural language processing provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of a summary generation device based on natural language processing provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. According to the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0028] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally includes steps or units that are not listed, or optionally includes other steps or units inherent to these processes, methods, products or devices.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] The network architecture used in the embodiment of the present application includes a server and an electronic device. The embodiment of the present application does not limit the number of electronic devices and servers, and the server can provide services for multiple electronic devices at the same time. The server can be an independent server, or it can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The server may be implemented by a server cluster consisting of multiple servers.

[0031] The electronic device may be a personal computer (PC), a laptop or a smart phone, or an all-in-one machine, a handheld computer, a tablet computer (pad), a smart TV player terminal, a vehicle-mounted terminal or a portable device. The operating system of the electronic device on the PC side, such as an all-in-one machine, may include but is not limited to Linux system, Unix system, Windows series system (such as Windows XP, Windows 7, etc.), Mac OS X system (Apple computer operating system) and other operating systems. The operating system of the electronic device on the mobile side, such as a smart phone, may include but is not limited to Android system, IOS (Apple mobile phone operating system), Window system and other operating systems.

[0032] The electronic device can install and run applications, and the server can be the server corresponding to the application installed in the electronic device, providing application services for the application. Among them, the application can be a separately integrated application software, or a small program embedded in other applications, or a system on a web page, etc., which is not limited here. In addition, this application does not limit the application scenario for generating summaries. The original text to be generated can be news, web pages, storage files, etc., and the summary can be a news title, web page search terms, storage tags, etc., which are not limited here.

[0033] For example, a user can upload a file to be stored through an application in an electronic device and send a summary generation request. The application or a server corresponding to the application can obtain a storage tag of the file as a summary of the file and store the file through the summary of the file.

[0034] The embodiment of the present application proposes a method for generating a summary based on natural language processing, which can be performed by a device for generating a summary based on natural language processing. The device can be implemented by software and / or hardware, and can generally be integrated in an electronic device or a server, and can improve the accuracy of generating a summary.

[0035] Please refer to Figure 1 , Figure 1 This is a flowchart of a method for generating a summary based on natural language processing provided by the present application. Taking the method applied to a server as an example, the method includes the following steps S101 to S108, wherein:

[0036] S101, splitting the original text to obtain at least one sentence and words in the sentence.

[0037] In an embodiment of the present application, the original text may include at least one sentence. The sentence refers to a text sequence with a complete meaning, which is usually separated by a terminator, such as a period, a question mark, etc., which is not limited here. The sentence can be composed of individual characters, and the character, as an independent individual, has its own meaning. Usually, it is described with characters in Chinese, and other languages ​​may be described with other descriptions, for example, it is described with words in English. Optionally, step S101 specifically includes the following steps: splitting the original text based on the language of the original text to obtain at least one sentence and the characters in the sentence. In this way, splitting sentences and characters based on the grammar of different languages ​​can improve the accuracy of splitting.

[0038] When splitting characters, the order of characters in the original text can also be recorded. For example, the larger the character number, the later the character is. In other words, the last character in the original text has the largest number, and the last character in a sentence is the character with the largest number in the sentence. Conversely, the smaller the character number, the earlier the character is.

[0039] S102: Determine the summary probability of the word.

[0040] In the embodiments of the present application, the summary probability of a word refers to the probability that the word is part of the summary. The summary probability can be understood as the importance of the word, for example, the impact of the word on the change of the meaning of the sentence, whether the word belongs to a modifying word such as an adjective. In other words, the greater the summary probability, the higher the accuracy of the word in generating a summary. The present application does not limit the method for determining the summary probability of a word, which can be determined based on one or more of the word's part of speech, word meaning or word frequency, or can be obtained by evaluating the word based on a preset algorithm, for example, using a term frequency (TF)-inverse document frequency (IDF) algorithm.

[0041] In a possible example, step S102 may include the following steps A1 to A4, wherein:

[0042] A1. Determine the word in the sentence where the character is located.

[0043] A sentence can also be composed of words (or phrases) composed of characters. The present application does not limit the method for determining words. It can be determined whether the vocabulary composed of a character in a sentence and at least one character adjacent to the character after the character is a preset vocabulary. If so, the vocabulary is determined to be a word in the sentence where the character is located. Otherwise, it is determined that there is no word in the sentence where the character is located. Alternatively, a word segmentation model composed of one or more of word segmenters, Jieba segmenters, Ansj segmenters, or HanLP segmenters can be used to divide the original text. While obtaining the words, the part of speech of each word can be obtained, for example, the two major categories of nouns and verbs, as well as names of people, places, and institutions, or adverbs, noun verbs, etc. While obtaining the words, the meaning of each word can also be obtained to determine the meaning of the word in the sentence.

[0044] A2. Get the word frequency and inverse text frequency.

[0045] A3. Get word frequency and inverse text frequency.

[0046] In the present application, frequency is used to describe the frequency of occurrence of a character (or word). For example, the proportion of a character in a sentence in the original text. The calculation of frequency tf can refer to the following formula (1), which is equal to the number of times n the i-th character (or word) appears in the sentence in the original text. i / The ratio between the total number of characters (or total number of words) N in the sentence.

[0047] tf=n i / N (1)

[0048] The inverse text frequency is used to describe the ratio of the number of occurrences of a word (or term) in the entire document. The calculation of the inverse text frequency idf can refer to the following formula (2):

[0049] idf=log(M / m i +1) (2)

[0050] Among them, M is the total number of summaries in the preset text library, m i is the number of abstracts containing the i-th character (or word). From formula (2), it can be seen that the more common a character (or word) is, the larger the denominator is, and the closer the inverse text frequency (idf) is to 0. The denominator +1 can avoid the situation where the denominator is 0, so that the inverse text frequency (idf) of the character (or word) can be calculated even if the character (or word) does not appear in all abstracts.

[0051] The present application does not limit the abstracts in the preset text library, which may be pre-generated abstracts, or abstracts obtained from web pages through crawler technology, etc. In a possible example, the document type of the original text is determined; and the preset text library is searched based on the document type.

[0052] The document types may be classified into entertainment, life, health, food, travel, automobiles, etc., or may be classified into characters, animals, plants, etc., or may be classified into news, notifications, chat records, etc., without limitation. It can be understood that searching for a preset text library based on the original document is conducive to improving the accuracy of the acquisition frequency and the inverse document frequency.

[0053] It should be noted that step A2 may be performed before step A3, or step A2 may be performed after step A3, or step A2 may be performed simultaneously with step A3, which is not limited here.

[0054] A4. Calculate the summary probability of a word based on the frequency and inverse text frequency of the word, as well as the frequency and inverse text frequency of the word.

[0055] In a possible example, step A4 may include the following steps A41 to A43, wherein:

[0056] A41. Calculate the product between the frequency of the word and the inverse text frequency to obtain a first value.

[0057] The first value is equal to the product of the frequency of the word and the inverse document frequency. The first value is used to describe the recall rate of the word. The calculation of the first value f1 can refer to the following formula (3):

[0058] f1=tf1*idf1 (3)

[0059] Among them, tf1 is the frequency of the word, and idf1 is the inverse text frequency of the word.

[0060] A42. Calculate the word frequency and the inverse text frequency based on the preset adjustment parameters to obtain a second value.

[0061] In an embodiment of the present application, the preset adjustment parameters may include a reconciliation parameter b, a document length ratio L and a constant k. Among them, the reconciliation parameter b is used to limit the influence of the document length on the calculation result. The document length ratio L is equal to the ratio between the current document length and the average document length, that is, the ratio between the character length of the sentence and the average character length of the summary sentence. The constant k is used to limit the growth limit of the frequency tf2. It can be understood that when tf2 increases, the product between tf2 and idf2 increases, so the constant k is used to limit the growth limit of the tf2 value. The present application does not limit the value of the constant k, for example, k = 1.2.

[0062] The second value is used to describe the recall rate of the word. The calculation of the second value f2 of the word can refer to the following formula (4):

[0063] f2=(idf2*(k+1)*tf2 / k*(1.0-b+b*L)+tf2 (4)

[0064] Among them, tf2 is the frequency of the word, and idf2 is the inverse text frequency of the word.

[0065] It should be noted that the present application does not limit the preset adjustment parameters, and other parameters may also be included in practical applications. In a possible example, the second value of the word may be multiplied by the preset weight of the word based on the above formula (4).

[0066] The preset weight of a word is used to describe the importance of the word. That is, the larger the preset weight of a word is, the more important the word is in the sentence. The present application does not limit the preset weight of a word. In a possible example, the following steps are also included: determining the preset weight of the word based on the part of speech and / or meaning of the word. In this way, the accuracy of setting the preset weight can be improved, which is conducive to improving the accuracy of calculating the second value.

[0067] For example, the preset weight of a verb may be 1.3; the preset weight of an adverb may be 1.2; the preset weight of a place noun may be 1.4; the preset weight of a demonstrative pronoun may be 0.7; the preset weight of a stop word may be 0.1, and so on.

[0068] It should be noted that step A41 can be executed before step A42, or step A41 can be executed after step A42, or step A41 can be executed together with step A42, which is not limited here.

[0069] A43. Calculate the summary probability of the word based on the first numerical value and the second numerical value.

[0070] The present application does not limit the method for calculating the summary probability of a word, and the product, maximum value, minimum value, or weighted average value between the first value and the second value may be calculated.

[0071] It can be understood that in steps A41 to A43, the summary probability of the word is obtained by calculating the first value obtained by the frequency of the word and the inverse document frequency, and the second value obtained by the frequency of the word in the sentence where the word is located and the inverse document frequency, which can improve the accuracy of selecting the word as the summary and is conducive to improving the accuracy of generating the summary.

[0072] In steps A1 to A4, the summary probability of a word is calculated based on the frequency and inverse document frequency of the word, and the frequency and inverse document frequency of the word in the sentence where the word is located. In other words, determining the summary probability of a word from both the word and the word in the sentence where the word is located can improve the accuracy of selecting the word as the summary, which is conducive to improving the accuracy of generating the summary.

[0073] S103: taking the word corresponding to the maximum value of the summary probability as the first word.

[0074] S104, taking the sentence containing the first word as the first sentence.

[0075] S105. Use the word after the first word in the first sentence as the second word, and use the word in the sentence after the first sentence as the third word.

[0076] In the embodiment of the present application, the first word refers to the word corresponding to the maximum value of the summary probability, and the first sentence is the sentence where the first word is located. The second word is the word after the first word in the first sentence, and the third word is the word in the sentence after the first sentence. The fourth word is the word selected from the second word, and the fifth word is the word selected from the fourth word. The combined summary probability refers to the probability that the word and the word combination are part of the summary. In other words, the combined summary probability of the first word and the second word refers to the probability that the first word and the second word are combined as part of the summary. The combined summary probability of the first word and the fourth word, and the third word refers to the probability that the first word and the fourth word, and the third word are combined as part of the summary. It can be understood that the greater the combined summary probability of the first word and the second word, the greater the probability that the second word is combined with the first word to generate the summary of the original text. The combined summary probability of the first word and the fourth word, and the third word, indicates that the third word, together with the first word and the fourth word, has a greater probability of generating the summary of the original text.

[0077] It should be noted that in step S105, the second character may be determined first and then the third character, or the third character may be determined first and then the second character, or the second character and the third character may be determined simultaneously, which is not limited here.

[0078] S106. Select at least one fourth word from the second word based on the combined summary probability of the first word and the second word.

[0079] In a possible example, step S106 may include the following steps B1 to B6, wherein:

[0080] B1. Take the first word as the previous text.

[0081] B2. Obtain the probability of the combined summary of the previous text and the second word.

[0082] Before executing step B2, it is first determined that the selected word, that is, the currently selected word is the first word, as the preceding text. In a possible example, step B2 may include the following steps B21 to B24, wherein:

[0083] B21. Get the reference summary sentence corresponding to the previous text.

[0084] The reference summary sentence refers to a pre-stored summary sentence containing the previous text. The reference summary sentence can be found from the aforementioned preset text library, or crawled from a web page or a pre-generated summary, etc., which is not limited here.

[0085] B22. Obtain a reference text sequence consisting of the preceding text and the second word.

[0086] The reference text sequence is a text sequence obtained by combining the previous text and the second word, and the number of the reference text sequences is equal to the number of the second words.

[0087] B23. Obtain the similarity value between the reference summary sentence and the reference text sequence.

[0088] The similarity value between the reference summary sentence and the reference text sequence can be determined based on the repetition rate between the characters or words in the two texts and the meaning described by the texts, or can be determined based on the Jaccard similarity coefficient and edit distance between the two texts, etc., which is not limited here.

[0089] The Jaccard similarity coefficient is used to calculate the similarity between samples of symbolic or Boolean metrics. The Jaccard similarity coefficient is equal to the ratio of the number of sample set intersections to the number of sample set unions. The calculation formula (5) of the Jaccard similarity coefficient S1 is as follows.

[0090] S1=N(A∩B) / N(A∪B) (5)

[0091] Where A is the sample set of words in the reference summary sentence, and B is the sample set of words in the reference text sequence. N(A∩B) represents the number of samples in the union of sample set A and sample set B, and N(A∪B) represents the number of samples in the union of sample set A and sample set B.

[0092] The edit distance refers to the number of single-character editing operations required to change from one string to another. Single-character operations include insertion, deletion, and replacement. The number of operations is c, and the edit distance S2 can be calculated by referring to formula (6).

[0093] S2=(Max(lenth(doc1),lenth(doc2))-c) / (Max(lenth(doc1),lenth(doc2))) (6)

[0094] Where, lenth(doc1) is the text length of the reference summary sentence, and lenth(doc2) is the text length of the reference text sequence. Since the number of operations may be greater than the length of the text, the final S2 may be a negative value, and the larger value between this value and 0 needs to be taken, that is, S2 = Max(0, S2).

[0095] In a possible example, step B23 may include the following steps: obtaining the Jaccard similarity coefficient and the edit distance between the reference summary sentence and the reference text sequence; and calculating the similarity value between the reference summary sentence and the reference text sequence based on the Jaccard similarity coefficient and the edit distance.

[0096] The similarity value may be equal to the product of the Jaccard similarity coefficient and the edit distance, the maximum value, the minimum value, or the weighted average value, etc., which is not limited here.

[0097] It can be understood that in this example, the similarity between the reference summary sentence and the reference text sequence is calculated by using the Jaccard similarity coefficient and the edit distance between the reference summary sentence and the reference text sequence. In this way, two methods for calculating text similarity values ​​are used to improve the accuracy of obtaining similarity values, which is conducive to improving the accuracy of selecting words as summaries.

[0098] B24. Based on the similarity value and the summary probability of the second word, calculate the combined summary probability of the previous text and the second word.

[0099] Among them, the combined summary probability of the preceding text and the second word can be equal to the product of the similarity value and the summary probability of the second word, the maximum value, the minimum value or the weighted average value, etc., which is not limited here. It can be understood that if the combined summary probability of the preceding text and the second word is equal to the product of the similarity value and the summary probability of the second word, then the combined summary probability of the preceding text and the second word can be understood as the summary probability obtained by updating the summary probability of the second word based on the similarity value, which is conducive to improving the accuracy of the word selected as the summary.

[0100] It can be understood that in steps B21 to B24, based on the similarity between the reference summary sentence corresponding to the previous text and the reference text sequence composed of the previous text and the second word, the probability of the combined summary of the previous text and the second word is calculated, which is conducive to improving the accuracy of generating the summary.

[0101] In a possible example, after step B23, the following steps B241 to B243 may be further included, wherein:

[0102] B241. Get the character interval between the fourth character currently selected and the second character after the fourth character.

[0103] B242. Determine a weight value of the second character based on the relationship between the character interval and the preset length.

[0104] B243. Calculate the product of the similarity value, the summary probability of the second word and the weighted value to obtain the combined summary probability of the previous text and the second word.

[0105] The character interval between the fourth character currently selected and the second character after the fourth character refers to the character interval between the last fourth character currently selected and the second character after the fourth character that has not been selected. The character interval should be a positive number, and can be equal to the difference between the sequence number of the second character after the fourth character currently selected and the sequence number of the fourth character.

[0106] The present application does not limit the preset length, and can be determined based on the position of the first word and the character length of the first sentence. The weighted value of the second word can be 0 when the character interval is greater than the preset length, and 1 when the character interval is less than or equal to the preset length.

[0107] In a possible example, step B242 may include: determining a weight value of the second word based on a ratio between a character interval and a preset length. The smaller the ratio, the greater the weight value, and when the ratio is greater than 1, the weight value is 0. In this way, it is possible to avoid selecting words with a long interval as part of the summary, and the probability of selecting adjacent words can be increased, which is conducive to improving the accuracy of the semantics of the summary.

[0108] It can be understood that in steps B241 to B243, the character interval between the fourth character currently selected and the second character after the fourth character is first obtained. The weighted value of the second character is then determined based on the size relationship between the character interval and the preset length. Then the product between the similarity value, the summary probability of the second character and the weighted value is calculated to obtain the combined summary probability of the previous text and the second character. In this way, it is possible to avoid selecting characters with longer intervals as part of the summary, improve the probability of selecting adjacent characters, improve the accuracy of selecting characters as the summary, and help improve the accuracy of the semantics of the summary.

[0109] B3. The second word corresponding to the maximum value of the combined summary probability is used as the fourth word.

[0110] B4. Determine whether the fourth character is the last character in the first sentence, and whether the fourth character is the termination character of the first sentence.

[0111] If the fourth character is not the last character in the first sentence, it means that there is an unselected second character after the fourth character in the first sentence, and step B5 and step B6 can be performed, wherein:

[0112] B5. Obtain a text sequence of the previous text and the fourth character combination to obtain a new previous text.

[0113] B6. Take the second character after the fourth character as the new second character.

[0114] It should be noted that step B5 can be performed before step B6, or step B5 can be performed after step B6, or step B5 can be performed simultaneously with step B6. After performing step B5 and step B6, the process returns to step B2, and when the fourth character is the last character in the first sentence, or is the terminating character of the first sentence, the process of selecting the fourth character is terminated, and step S107 can be performed.

[0115] If the fourth character is the last character in the first sentence, it means that the fourth character in the first sentence has been selected, and the character used as the summary in the next sentence can be searched, for example, and step S107 is executed. If the fourth character is the terminal character of the first sentence, it means that the text sequence corresponding to the selected character in the first sentence is complete, and the character used as the summary in the next sentence can be searched, and step S107 is executed. Otherwise, if the fourth character is not the last character in the first sentence and the fourth character is not the terminal character of the first sentence, steps B5 and B6 can be executed.

[0116] The present application does not limit the method for determining whether the fourth character is the termination character of the first sentence, and the judgment can be made by determining the complete value of the sentence composed of the preceding text and the fourth character. If the complete value is equal to 1, the fourth character is determined to be the termination character of the first sentence. Otherwise, it is determined that the fourth character is not the termination character of the first sentence.

[0117] It can be understood that in steps B1 to B6, the first word selected is first used as the previous text, and then the combined summary probability of the previous text and each second word is obtained. The second word corresponding to the maximum value of the combined summary probability is used as the fourth word. If the fourth word is not the last word in the first sentence and is not the terminating word of the first sentence, the previous text is updated by the text sequence obtained by combining the previous text and the fourth word, and the second word is updated to the second word after the fourth word. Then the combined summary probability of the new previous text and the new second word is obtained until the selected fourth word is the last word in the first sentence, or is the terminating word of the first sentence. In this way, the fourth word with a greater probability of combination with the previous text can be selected from the second word after the first word in the first sentence, which can improve the accuracy of selecting the fourth word.

[0118] S107. Select at least one fifth character from the third character based on the combined summary probability of the first character, the fourth character and the third character.

[0119] In the embodiment of the present application, the third word is a word in the sentence after the first sentence, and the fifth word is a word selected from the third word. It should be noted that the sentence where the fifth word is located may include the sentence or part of the sentence after the first sentence. In other words, some sentences are not used to generate the summary.

[0120] In a possible example, step S107 may refer to the description of step S106, and include the following steps C1 to C6, wherein:

[0121] C1. Get the text sequence consisting of the first word and the fourth word to get the new previous text.

[0122] C2. Obtain the probability of the combined summary of the previous text and the third word.

[0123] C3. The third word corresponding to the maximum value of the combined summary probability is used as the fifth word.

[0124] C4. Determine whether the fifth character is the last character in the original text, and determine whether the fifth character is the termination character of the previous text.

[0125] Before executing step C2, the selected characters, that is, the currently selected characters are the first character and the fourth character selected in the first sentence, are combined to obtain a new preceding text. Then, the combined summary probability of the preceding text and the characters after the first sentence is obtained. The method for obtaining the combined summary probability can refer to the description of step B2, which will not be repeated here.

[0126] If the fifth character is not the last character in the original text, it means that there is an unselected fifth character after the selected fifth character, and step C5 and step C6 can be performed, wherein:

[0127] C5. Obtain a text sequence of the previous text and the fifth word combination to obtain a new previous text.

[0128] C6. Take the sentence containing the fifth character as the second sentence, and take the character after the fifth character in the second sentence as the new third character.

[0129] It should be noted that step C5 can be performed before step C6, or step C5 can be performed after step C6, or step C5 can be performed simultaneously with step C6. After performing step C5 and step C6, the process returns to step C2, and when the fifth character is the last character in the original text, or is the end character of the previous text, the step of selecting the fifth character is terminated, and step S108 can be performed.

[0130] If the fifth character is the last character in the original text, it means that the characters in the original text have been selected, and step S108 can be executed, that is, a summary of the original text is formed based on the currently selected characters. If the fifth character is the terminal character of the previous text, it means that the text sequence corresponding to the previous text and the fifth character is a complete summary, and the step of selecting the fifth character can be ended, and step S108 can be executed. There is no limitation on the method of determining whether the fifth character is the terminal character of the previous text, and the description of whether the fourth character is the terminal character of the first sentence can be referred to, which will not be repeated here.

[0131] It can be understood that in steps C1 to C6, the selected first word and the fourth word are first combined to obtain a text sequence as the previous text. Then the combined summary probability of the previous text and each third word is obtained. The third word corresponding to the maximum value of the combined summary probability is used as the fifth word. If the fifth word is not the last word in the original text and is not the terminal word of the previous text, the previous text is updated with the text sequence obtained by combining the previous text and the fifth word, and the third word is updated to the third word after the fifth word. Then the combined summary probability of the new previous text and the new third word is obtained until the selected fifth word is the last word in the original text, or is the terminal word of the previous text. In this way, the fifth word with a greater probability of combination with the previous text can be selected from the third words after the first sentence, which can improve the accuracy of selecting the fifth word.

[0132] S108, obtaining a text sequence consisting of the first character, the fourth character, and the fifth character to obtain a summary of the original text.

[0133] The summary of the original text can be a sentence arranged by arranging the first, fourth and fifth characters in the order in the original text, or can be a sentence arranged based on the meaning and part of speech of the first, fourth and fifth characters, etc., without limitation here.

[0134] exist Figure 1In the method shown, the original text is first split to obtain at least one sentence and the characters in each sentence. Then the summary probability of each character is determined. Then the character corresponding to the maximum summary probability is used as the first character, and the sentence where the first character is located is used as the first sentence, the character after the first character in the first sentence is used as the second character, and the character in the sentence after the first sentence is used as the third character. Then, based on the combined summary probability of the first character and the second character, at least one fourth character is selected from the second character. Based on the combined summary probability of the first character and the fourth character, at least one fifth character is selected from the third character. Finally, a text sequence consisting of the first character, the second character and the third character is obtained to obtain a summary of the original text. In this way, a summary is generated based on the combined summary probability of each character in the original text and the characters selected in the previous text, which can improve the accuracy of generating the summary.

[0135] The method of the embodiment of the present application is described in detail above, and the device of the embodiment of the present application is provided below.

[0136] and Figure 1 The embodiment shown is consistent with that shown in FIG. Figure 2 , Figure 2 Schematic diagram of a summary generation device based on natural language processing proposed in this application. Figure 2 As shown, the above-mentioned summary generation device 200 based on natural language processing includes:

[0137] The splitting unit 201 is used to split the original text to obtain at least one sentence and the words in the sentence;

[0138] The determination unit 202 is used to determine the summary probability of the word; take the word corresponding to the maximum summary probability as the first word; take the sentence where the first word is located as the first sentence; take the word after the first word in the first sentence as the second word, and take the word in the sentence after the first sentence as the third word;

[0139] The selection unit 203 is used to select at least one fourth word from the second word based on the combined summary probability of the first word and the second word; and select at least one fifth word from the third word based on the combined summary probability of the first word, the fourth word, and the third word;

[0140] The composition unit 204 is used to obtain a text sequence composed of the first character, the fourth character, and the fifth character to obtain a summary of the original text.

[0141] In a possible example, the determination unit 202 is specifically used to determine the word in the sentence where the word is located; obtain the frequency and inverse text frequency of the word; obtain the frequency and inverse text frequency of the word; and calculate the summary probability of the word based on the frequency and inverse text frequency of the word, and the frequency and inverse text frequency of the word.

[0142] In a possible example, the determination unit 202 is specifically used to calculate the product between the frequency of the word and the inverse text frequency to obtain a first value; calculate the frequency and the inverse text frequency of the word based on a preset adjustment parameter to obtain a second value; and calculate the summary probability of the word based on the first value and the second value.

[0143] In a possible example, the selection unit 203 is specifically used to take the first character as the previous text; obtain the combined summary probability of the previous text and the second character; take the second character corresponding to the maximum value of the combined summary probability as the fourth character; if the fourth character is not the last character in the first sentence, and the fourth character is not the termination character of the first sentence, then obtain a text sequence of the combination of the previous text and the fourth character to obtain a new previous text, take the second character after the fourth character as a new second character, execute the step of obtaining the combined summary probability of the previous text and the second character, and when the fourth character is the last character in the first sentence, or the fourth character is the termination character of the first sentence, end the step of selecting the fourth character.

[0144] In a possible example, the selection unit 203 is specifically used to obtain a reference summary sentence corresponding to the previous text; obtain a reference text sequence composed of the previous text and the second word; obtain a similarity value between the reference summary sentence and the reference text sequence; and calculate a combined summary probability of the previous text and the second word based on the similarity value and the summary probability of the second word.

[0145] In a possible example, the selection unit 203 is also used to obtain the character interval between the fourth character currently selected and the second character after the fourth character; determine the weighted value of the second character based on the size relationship between the character interval and a preset length; calculate the product of the similarity value, the summary probability of the second character and the weighted value to obtain the combined summary probability of the previous text and the second character.

[0146] In a possible example, the selection unit 203 is specifically used to obtain a text sequence composed of the first character and the fourth character to obtain a new previous text; obtain a combined summary probability of the previous text and the third character; use the third character corresponding to the maximum value of the combined summary probability as the fifth character; if the fifth character is not the last character in the original text, and the fifth character is not the termination character of the previous text, then obtain a text sequence composed of the previous text and the fifth character to obtain a new previous text, use the sentence where the fifth character is located as the second sentence, use the character after the fifth character in the second sentence as the new third character, perform the step of obtaining the combined summary probability of the previous text and the third character, and when the fifth character is the last character in the original text, or the fifth character is the termination character of the previous text, end the step of selecting the fifth character.

[0147] The detailed execution process of each unit in the abstract generation device 200 based on natural language processing can refer to the execution steps in the aforementioned method embodiment, which will not be repeated here.

[0148] and Figure 1 For details, please refer to Figure 3 , Figure 3 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Figure 3 As shown, the computer device 300 includes a processor 310, a memory 320 and a communication interface 330. The processor 310, the memory 320 and the communication interface 330 are connected to each other via a bus 350. Figure 2 The related functions implemented by the splitting unit 201 , the determining unit 202 , the selecting unit 203 and the composing unit 204 shown can be implemented by the processor 310 .

[0149] The memory 320 stores a computer program 340, which is configured to be executed by the processor 310. The computer program 340 includes instructions for executing the following steps:

[0150] Splitting the original text to obtain at least one sentence and words in the sentence;

[0151] determining a summary probability of the word;

[0152] The word corresponding to the maximum value of the summary probability is used as the first word;

[0153] The sentence containing the first word is taken as the first sentence;

[0154] using the word after the first word in the first sentence as the second word, and using the word in the sentence after the first sentence as the third word;

[0155] Selecting at least one fourth character from the second character based on the combined summary probability of the first character and the second character;

[0156] Based on the combined summary probability of the first word, the fourth word, and the third word, selecting at least one fifth word from the third word;

[0157] A text sequence consisting of the first character, the fourth character, and the fifth character is obtained to obtain a summary of the original text.

[0158] In one possible example, in determining the summary probability of the word, the computer program 340 includes instructions specifically for performing the following steps:

[0159] determining the word in the sentence in which the character is located;

[0160] Obtain the frequency and inverse text frequency of the word;

[0161] Obtain the frequency and inverse text frequency of the word;

[0162] The summary probability of the word is calculated based on the frequency and inverse text frequency of the word, and the frequency and inverse text frequency of the word.

[0163] In one possible example, in the aspect of calculating the summary probability of the word based on the frequency and inverse text frequency of the word, and the frequency and inverse text frequency of the word, the computer program 340 includes instructions specifically for performing the following steps:

[0164] Calculate the product between the frequency of the word and the inverse text frequency to obtain a first value;

[0165] Calculate the frequency and inverse text frequency of the word based on the preset adjustment parameter to obtain a second value;

[0166] A digest probability of the word is calculated based on the first value and the second value.

[0167] In one possible example, in the aspect of selecting at least one fourth word from the second word based on the combined summary probability of the first word and the second word, the computer program 340 includes instructions specifically for performing the following steps:

[0168] Take the first word as the preceding text;

[0169] Obtaining a probability of a combined summary of the preceding text and the second word;

[0170] The second word corresponding to the maximum value of the combined summary probability is used as the fourth word;

[0171] If the fourth character is not the last character in the first sentence and the fourth character is not the terminating character of the first sentence, then obtaining a text sequence of the combination of the previous text and the fourth character to obtain a new previous text; taking the second character after the fourth character as the new second character; executing the step of obtaining the probability of a combined summary of the previous text and the second character, and ending the step of selecting the fourth character when the fourth character is the last character in the first sentence or the terminating character of the first sentence.

[0172] In a possible example, in the aspect of obtaining the probability of a combined summary of the preceding text and the second word, the computer program 340 includes instructions for executing the following steps:

[0173] Obtain a reference summary sentence corresponding to the previous text;

[0174] Obtaining a reference text sequence consisting of the preceding text and the second character;

[0175] Obtaining a similarity value between the reference summary sentence and the reference text sequence;

[0176] Based on the similarity value and the summary probability of the second word, a combined summary probability of the preceding text and the second word is calculated.

[0177] In one possible example, the computer program 340 includes instructions for further performing the following steps:

[0178] Obtain the character interval between the fourth character currently selected and the second character after the fourth character;

[0179] Determining a weight value of the second character based on a magnitude relationship between the character interval and a preset length;

[0180] The product of the similarity value, the summary probability of the second word and the weighted value is calculated to obtain a combined summary probability of the previous text and the second word.

[0181] In one possible example, in the aspect of selecting at least one fifth word from the third word based on the combined summary probability of the first word, the fourth word, and the third word, the computer program 340 includes instructions specifically for performing the following steps:

[0182] Obtaining a text sequence consisting of the first character and the fourth character to obtain a new preceding text;

[0183] Obtaining a probability of a combined summary of the preceding text and the third word;

[0184] The third word corresponding to the maximum value of the combined summary probability is used as the fifth word;

[0185] If the fifth character is not the last character in the original text, and the fifth character is not the terminating character of the previous text, then a text sequence of the previous text and the fifth character is obtained to obtain a new previous text; the sentence where the fifth character is located is taken as the second sentence, and the character after the fifth character in the second sentence is taken as the new third character; the step of obtaining the probability of a combined summary of the previous text and the third character is performed, and when the fifth character is the last character in the original text, or the fifth character is the terminating character of the previous text, the step of selecting the fifth character is terminated.

[0186] An embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, which enables a computer to execute part or all of the steps of any method recorded in the method embodiment, and the computer includes an electronic device or a server.

[0187] The present application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to execute to implement some or all steps of any method described in the method embodiment. The computer program product can be a software installation package, and the computer includes an electronic device or a server.

[0188] In the above embodiment, the computer-readable storage medium may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function, etc.; the data storage area may store data created according to the use of the blockchain node, etc. For example, the blockchain may store preset algorithms, preset vocabularies, preset text libraries, preset adjustment parameters, preset weights, preset lengths, etc., which are not limited here.

[0189] The blockchain referred to in the embodiments of the present application is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods, each of which contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0190] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modes involved are not necessarily required by the present application.

[0191] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0192] In the several embodiments provided in the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic, such as the division of units, which is only a logical function division. There may be other division methods in actual implementation, such as at least one unit or component can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0193] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be in one place, or they may be distributed on at least one network unit. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0194] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software program mode.

[0195] If the integrated unit is implemented in the form of a software program model and sold or used as an independent product, it can be stored in a computer-readable memory. According to this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM), random access memory (RAM), mobile hard disk, magnetic disk or optical disk, etc., and other media that can store program codes.

[0196] The embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A summary generation method based on natural language processing, characterized in that: include: Splitting the original text to obtain at least one sentence and words in the sentence; determining a summary probability of the word; The word corresponding to the maximum value of the summary probability is used as the first word; The sentence containing the first word is taken as the first sentence; using the word after the first word in the first sentence as the second word, and using the word in the sentence after the first sentence as the third word; Selecting at least one fourth character from the second character based on the combined summary probability of the first character and the second character; Based on the combined summary probability of the first word, the fourth word, and the third word, selecting at least one fifth word from the third word; Obtaining a text sequence consisting of the first character, the fourth character, and the fifth character to obtain a summary of the original text; Wherein, determining the summary probability of the word includes: determining the word in the sentence in which the character is located; Obtain the frequency and inverse text frequency of the word; Obtain the frequency and inverse text frequency of the word; Calculating a summary probability of the word based on the frequency and inverse text frequency of the word, and the frequency and inverse text frequency of the word; The selecting at least one fourth word from the second word based on the combined summary probability of the first word and the second word comprises: Take the first word as the preceding text; Obtaining a probability of a combined summary of the preceding text and the second word; The second word corresponding to the maximum value of the combined summary probability is used as the fourth word; If the fourth character is not the last character in the first sentence and the fourth character is not the terminating character of the first sentence, then obtaining a text sequence of the combination of the previous text and the fourth character to obtain a new previous text; taking the second character after the fourth character as the new second character; executing the step of obtaining the probability of a combined summary of the previous text and the second character, and ending the step of selecting the fourth character when the fourth character is the last character in the first sentence or the terminating character of the first sentence.

2. The method according to claim 1, characterized in that The calculating of the summary probability of the word based on the frequency and inverse text frequency of the word, and the frequency and inverse text frequency of the word, comprises: Calculate the product between the frequency of the word and the inverse text frequency to obtain a first value; Calculate the frequency and inverse text frequency of the word based on the preset adjustment parameter to obtain a second value; A digest probability of the word is calculated based on the first value and the second value.

3. The method according to claim 1, characterized in that The obtaining a probability of a combined summary of the preceding text and the second word includes: Obtain a reference summary sentence corresponding to the previous text; Obtaining a reference text sequence consisting of the preceding text and the second character; Obtaining a similarity value between the reference summary sentence and the reference text sequence; Based on the similarity value and the summary probability of the second word, a combined summary probability of the preceding text and the second word is calculated.

4. The method according to claim 3, characterized in that: The method further comprises: Obtain the character interval between the fourth character currently selected and the second character after the fourth character; Determining a weight value of the second character based on a magnitude relationship between the character interval and a preset length; The product of the similarity value, the summary probability of the second word and the weighted value is calculated to obtain a combined summary probability of the previous text and the second word.

5. The method according to any one of claims 1 to 4, characterized in that The selecting at least one fifth word from the third word based on the combined summary probability of the first word, the fourth word, and the third word comprises: Obtaining a text sequence consisting of the first character and the fourth character to obtain a new preceding text; Obtaining a probability of a combined summary of the preceding text and the third word; The third word corresponding to the maximum value of the combined summary probability is used as the fifth word; If the fifth character is not the last character in the original text, and the fifth character is not the terminating character of the previous text, then a text sequence of the previous text and the fifth character is obtained to obtain a new previous text; the sentence where the fifth character is located is taken as the second sentence, and the character after the fifth character in the second sentence is taken as the new third character; the step of obtaining the probability of a combined summary of the previous text and the third character is performed, and when the fifth character is the last character in the original text, or the fifth character is the terminating character of the previous text, the step of selecting the fifth character is terminated.

6. A summary generation device based on natural language processing, characterized in that: Comprising means for indicating the method as claimed in any one of claims 1-5.

7. A computer device, characterized in that: The method comprises a processor, a memory and a communication interface, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the steps in any one of the methods of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which enables a computer to execute to implement the method according to any one of claims 1 to 5.

Images