Text abstract generation method and device, computer equipment and storage medium
By introducing attention mechanism and dynamic expansion convolutional neural network in text summary generation, the problems of insufficient information coverage, poor language fluency, insufficient context understanding and model bias in the existing technology are solved, and the generation is more accurate, concise and objective. text summary.
Patent Information
- Application Number
- CN202411376118.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-27
- Publication Date
- 2025-02-18
AI Technical Summary
The existing text summary generation technology has problems such as insufficient information coverage, poor language fluency, insufficient context understanding and model bias, which leads to inaccurate and objective enough in the generated summary.
The text summary generation method based on attention mechanism and dynamic expansion convolution neural network is adopted. The embedded text is parsed through the attention module, and the text feature information is output. The dynamic expansion convolution neural network module is used to perform multiple convolutions to generate text information corresponding text summary.
Improves the accuracy and fluency of text summary, enhances the understanding of the context, reduces model bias, and produces summary more concise and informative.
Smart Images
Figure CN119474357A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, which is applied in the financial field, and in particular to a text summary generation method, device, computer equipment and storage medium. Background Art
[0002] In text summarization tasks in the financial field, such as summarizing long documents such as insurance contracts and financial reports, it can help users and related practitioners quickly understand the overall content of the text and save them from reading too much text content.
[0003] Existing text summary generation technologies mainly rely on statistical and machine learning methods, such as TF-IDF, TextRank algorithms, and deep learning-based RNN and Transformer models (such as BERT). These methods can generate effective summaries to a certain extent, but there are still at least the following four problems: 1. Insufficient information coverage: Existing models sometimes miss important information, and the generated summaries cannot fully cover the key information of the original text. 2. Poor language fluency: The generated summaries lack language fluency and naturalness, and may have grammatical errors or incoherent sentences. 3. Insufficient contextual understanding: When dealing with complex sentence structures or long-distance dependent information, the existing models perform poorly, and the generated summaries may lack coherence and logic. 4. Model bias: Due to the bias of training data, the model may be biased towards certain topics or viewpoints when generating summaries, and cannot present the original content fairly and objectively.
[0004] Therefore, there is an urgent need for a text summary generation method to improve the quality of text summary generation in the financial field. Summary of the invention
[0005] The present application provides a text summary generation method, device, computer equipment and storage medium, aiming to solve the existing text summary generation technology which mainly relies on statistics and machine learning methods, and has problems such as insufficient information coverage, poor language fluency, insufficient context understanding and model bias. The provided method proposes a text summary generation method based on attention mechanism and dynamically expanded convolutional neural network. The method provided by the present application can be applied in the financial system to generate more accurate, concise and information-rich contract / text summaries, help users quickly obtain key information of documents, and improve user experience and work efficiency of business personnel.
[0006] In a first aspect, the present application provides a text summary generation method, comprising:
[0007] Get the text information to be generated;
[0008] Embedding the text information into words to obtain embedded text;
[0009] Inputting the embedded text into a preset attention module, the attention module parses the embedded text and outputs text feature information;
[0010] Inputting the text feature information into a preset dynamic expansion convolutional neural network module, the dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and position distribution information corresponding to each of the text words;
[0011] A text summary corresponding to the text information is generated according to each of the text words and the corresponding position distribution information.
[0012] In some embodiments, the dynamically expanding convolutional neural network module includes multiple convolutional layers, fully connected layers and an output layer; the text feature information is input into a preset dynamically expanding convolutional neural network module, including: obtaining the dynamic expansion rate corresponding to each convolutional layer according to the text feature information; the text feature information is input into each of the convolutional layers respectively, and each of the convolutional layers performs a hole convolution on the text feature information according to the corresponding dynamic expansion rate, and outputs text convolution information; each of the text convolution information is input into a fully connected layer, and the fully connected layer performs a full connection on multiple text convolution information and outputs a first fully connected information; the text fully connected information is input into the output layer, and the output layer normalizes the text fully connected information to obtain multiple text words and the position distribution information corresponding to each of the text words.
[0013] Exemplarily, the method of obtaining the dynamic expansion rate corresponding to each convolutional layer according to the text feature information includes: obtaining the text sequence length corresponding to the text information, and determining the average pooling dimension according to the text sequence length; performing average pooling on the text feature information according to the average pooling dimension; performing full connection on the text information after average pooling, and outputting second fully connected information; and parsing the second fully connected information to obtain the dynamic expansion rate corresponding to each convolutional layer.
[0014] Exemplarily, the dynamically expanded convolutional neural network also includes a residual layer; before inputting each of the text convolution information into the fully connected layer, it also includes: inputting each of the text convolution information and the text feature information into the residual layer respectively, and the residual layer is used to perform a residual connection between each of the text convolution information and the text feature information, so as to input the text convolution information after the residual connection into the fully connected layer.
[0015] In some embodiments, before generating a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information, it also includes: obtaining label information corresponding to the text word; the label information includes the real position distribution information corresponding to the text word; calculating the cross entropy loss function value of each of the real position distribution information and the position distribution information; and completing the optimization of the attention module and the dynamically expanding convolutional neural network module according to the cross entropy loss function value.
[0016] In some embodiments, the position distribution information includes the end symbol probability information corresponding to the text word; generating the text summary corresponding to the text information based on each of the text words and the corresponding position distribution information includes: obtaining a truncation threshold; removing the text words whose end symbol probability information is less than the truncation threshold from the multiple text words to obtain the target text word; and generating the text summary according to the target text word.
[0017] In some embodiments, embedding the text information into words to obtain embedded text includes: inputting the text information into a preset language representation model, the language representation model converting the text information into a dense vector form, and outputting the embedded text.
[0018] In a second aspect, the present application provides a text summary generation device, comprising:
[0019] A text acquisition unit, used to acquire text information to be generated;
[0020] A text embedding unit, used to embed the text information into words to obtain embedded text;
[0021] A feature output unit, used to input the embedded text into a preset attention module, and the attention module parses the embedded text and outputs text feature information;
[0022] A convolution output unit, used to input the text feature information into a preset dynamic expansion convolutional neural network module, the dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and position distribution information corresponding to each of the text words;
[0023] The summary generating unit is used to generate a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information.
[0024] In a third aspect, the present application further provides a computer device, including:
[0025] Memory and processor;
[0026] The memory is used to store computer programs;
[0027] The processor is used to execute the computer program and implement the steps of the text summary generation method described in the first aspect when executing the computer program.
[0028] In a fourth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the steps of the text summary generation method as described in the first aspect above.
[0029] The present application discloses a method, device, computer equipment and storage medium for generating a text summary. First, the method obtains the text information to be generated. Then, the text information is embedded in words to obtain the embedded text. Then, the embedded text is input into a preset attention module, and the attention module parses the embedded text and outputs text feature information. Further, the text feature information is input into a preset dynamic expansion convolutional neural network module, and the dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and the position distribution information corresponding to each text word. Finally, a text summary corresponding to the text information is generated according to each text word and the corresponding position distribution information. A method for generating a text summary based on an attention mechanism and a dynamic expansion convolutional neural network is proposed.
[0030] At the same time, the method provided in this application can be applied in finance or other systems where there is a large amount of text that needs to generate summaries (such as medical, education, etc.) to generate more accurate, concise and information-rich contract / text summaries, helping users to quickly obtain key information of the document, thereby improving user experience and the work efficiency of business personnel.
[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0033] Figure 1 This is a flowchart showing the steps of a text summary generation method provided in an embodiment of the present application;
[0034] Figure 2 is a schematic flow chart of the steps of a method for obtaining location distribution information provided by an embodiment of the present application;
[0035] Figure 3 It is a schematic flow chart of the steps of an optimization method for an attention module and a dynamically expanded convolutional neural network module provided in one embodiment of the present application;
[0036] Figure 4 It is a structural diagram of a text summary generating device provided in an embodiment of the present application;
[0037] Figure 5 It is a schematic block diagram of the structure of a computer device provided in one embodiment of the present application.
[0038] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0040] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions.
[0041] It should be understood that, in order to facilitate the clear description of the technical solutions of the embodiments of the present invention, in the embodiments of the present invention, the words "first", "second", etc. are used to distinguish the same items or similar items with substantially the same functions and effects. Those skilled in the art can understand that the words "first", "second", etc. do not limit the quantity and execution order, and the words "first", "second", etc. do not necessarily limit the difference.
[0042] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.
[0043] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0044] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0045] In text summarization tasks in the financial field, such as summarizing long documents such as insurance contracts and financial reports, it can help users and related practitioners quickly understand the overall content of the text and save them from reading too much text content.
[0046] Existing text summary generation technologies mainly rely on statistical and machine learning methods, such as TF-IDF, TextRank algorithms, and deep learning-based RNN and Transformer models (such as BERT). These methods can generate effective summaries to a certain extent, but there are still at least the following four problems: 1. Insufficient information coverage: Existing models sometimes miss important information, and the generated summaries cannot fully cover the key information of the original text. 2. Poor language fluency: The generated summaries lack language fluency and naturalness, and may have grammatical errors or incoherent sentences. 3. Insufficient contextual understanding: When dealing with complex sentence structures or long-distance dependent information, the existing models perform poorly, and the generated summaries may lack coherence and logic. 4. Model bias: Due to the bias of training data, the model may be biased towards certain topics or viewpoints when generating summaries, and cannot present the original content fairly and objectively.
[0047] Therefore, there is an urgent need for a text summary generation method to improve the quality of text summary generation in the financial field.
[0048] To solve the above problems, please refer to Figure 1 , Figure 1 This is a schematic flow chart of a text summary generation method provided by an embodiment of the present application. The text summary generation method can be implemented by a computer device, and the computer device can be deployed on a single server or a server cluster. It can also be deployed on a handheld terminal, a laptop computer, a wearable device or a robot, etc.
[0049] It should be noted that the provided method can be applied not only to the financial field to generate summaries of contracts and other texts, but also to other systems where there are a large number of texts that need to generate summaries (such as medical, education, etc.) to accurately and quickly generate summaries of massive texts. Therefore, the embodiments of this application do not limit the field of the provided method.
[0050] To solve the above problems, please refer to Figure 1 Specifically, Figure 1 As shown, the provided text summary generation method includes steps S101 to S105. The details are as follows:
[0051] Step S101: Obtain text information to be generated.
[0052] Specifically, the text acquired by the computer device can be any form of text in the application scenario, such as articles, reports, email content, posts on related platforms and social media, etc. In the provided method, the computer device can obtain the text information uploaded by the user that needs to generate a text summary, or it can read text information from multiple sources such as files, web pages, and databases to generate a summary in accordance with relevant regulations. At the same time, it is necessary to ensure the preprocessing of the input text, which may include removing invalid characters, standardizing punctuation marks, counting words, etc. The text acquisition step is the basis for all subsequent processing. Ensuring that the text information is high-quality and standardized original text data can improve the accuracy and efficiency of subsequent processing.
[0053] Step S102: embed the text information into words to obtain embedded text.
[0054] Specifically, word embedding converts each word in a text into a high-dimensional vector. Pre-trained models (such as Word2Vec, GloVe, FastText, BERT, etc.) are used in this process. The vector corresponding to the embedded text captures the semantics and contextual relationship of the word, making the semantic features of the text clearer. At the same time, in the process of word embedding, text preprocessing operations such as stem extraction, word form restoration, and removal of stop words need to be considered to improve the effect of word embedding. In turn, word embedding simplifies the text representation, making semantically similar words closer in the vector space, so that subsequent machine learning models can better understand the text content and context.
[0055] In some embodiments, text information is word-embedded to obtain embedded text, including: inputting the text information into a preset language representation model, the language representation model converting the text information into a dense vector form, and outputting the embedded text.
[0056] Language representation models refer to pre-trained natural language processing models that can convert text into dense vectors. Common language representation models include:
[0057] Word2Vec: Embeds words into a low-dimensional continuous vector space through unsupervised learning.
[0058] GloVe (Global Vectors for Word Representation): Based on the frequency information of the word co-occurrence matrix, it can capture semantic similarity.
[0059] FastText: Character-level vector representation of words, which can better handle unknown and rare words.
[0060] BERT (Bidirectional Encoder Representations from Transformers): Utilizes the bidirectional Transformer architecture to better capture contextual information.
[0061] By passing the input text information to the preset language representation model. The model converts each word or phrase into a multi-dimensional dense vector. These vectors are mathematical vector representations, usually with dimensions ranging from dozens to hundreds, depending on the design of the representation model. The model takes context into account during the conversion process, so that the generated vector contains not only the semantic information of the word, but also its position and function in the sentence. Finally, the language representation model outputs a dense vector for each word, which is combined to form embedded text. Embedded text is the representation of the original text in the vector space, which is used for subsequent machine learning and deep learning model processing. This has the following beneficial effects:
[0062] 1. Improve semantic understanding capabilities: Through the language representation model, each word of the text is converted into a dense vector that can capture the semantics and contextual relationships of the words, thereby improving the system's ability to understand the text.
[0063] 2. Handling unknown words and semantically similar words: Models like FastText in particular can handle unknown words and misspelled words, making the embedding representation more robust. Models such as GloVe and BERT can capture semantically similar words well, making semantically similar words closer in the vector space.
[0064] 3. Rich contextual information: Bidirectional models like BERT can take into account the previous and next information in the context, so that the generated embedding vector contains richer contextual information.
[0065] 4. Improve the performance of subsequent models: High-quality word embeddings can provide better feature representation for subsequent attention mechanisms and convolutional neural networks, improving the overall text processing and summary generation effects.
[0066] 5. Reduce computational complexity: Compared with sparse vectors (such as the bag-of-words model), dense vectors greatly reduce dimensions and reduce storage and computational complexity.
[0067] In summary, the provided embodiment converts text information into dense vectors by using a preset language representation model, which not only enhances the system's understanding of text semantics and context, but also provides high-quality input for subsequent processing, making the entire summary generation process more efficient and accurate.
[0068] Step S103: Input the embedded text into a preset attention module, and the attention module parses the embedded text and outputs text feature information.
[0069] Specifically, the attention module assigns weights by calculating the importance of each word or phrase in the context. Common attention mechanisms include Bahdanau Attention, Luong Attention, and the Self-Attention mechanism in Transformers. The embodiment of the present application does not limit the type of attention mechanism. At the same time, the text feature information can be an attention matrix, which determines the contribution of each word to the overall text features at different positions. The attention mechanism can identify which parts are the key content of the text and perform focused analysis, thereby improving the accuracy and effect of text feature extraction.
[0070] Step S104. Input the text feature information into a preset dynamic expansion convolutional neural network module. The dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and position distribution information corresponding to each text word.
[0071] Specifically, the Dilated Convolutional Neural Network (DCNN) is a special convolutional neural network whose convolution kernel uses dilation technology to capture contextual information of different ranges. The dynamic dilation rate adjusts the sampling interval of the convolution kernel on the feature map, thereby extracting features at multiple scales. Multiple convolution operations will deeply understand the structure and semantics of the text layer by layer, and output feature maps containing text words and their position information. Dynamic dilation convolution can expand the receptive field and obtain a wider range of contextual dependencies, thereby enriching the feature representation and improving the quality and accuracy of summary generation.
[0072] In some embodiments, the dynamically expanded convolutional neural network module includes multiple convolutional layers, fully connected layers, and output layers; see Figure 2 , inputting the text feature information into a preset dynamically dilated convolutional neural network module, including steps S104a to S104d:
[0073] Step S104a. Obtain the dynamic expansion rate corresponding to each convolutional layer according to the text feature information.
[0074] Step S104b. Input the text feature information into each convolution layer respectively. Each convolution layer performs hole convolution on the text feature information according to the corresponding dynamic expansion rate and outputs the text convolution information.
[0075] Step S104c. Input each text convolution information into the fully connected layer. The fully connected layer fully connects multiple text convolution information and outputs the first fully connected information.
[0076] Step S104d. Input the text full connection information into the output layer, and the output layer normalizes the text full connection information to obtain multiple text words and position distribution information corresponding to each text word.
[0077] The above embodiment first inputs the text feature information into the module, and assigns a dynamic dilation rate to each convolution layer according to a specific algorithm or rule. Each convolution layer uses the assigned dynamic dilation rate to perform dilated convolution to extract information from the feature map. The dynamic dilation rate determines the sampling interval of the convolution kernel and affects the receptive field of the convolution operation. The fully connected layer aggregates the text convolution information output by all convolution layers, performs a fully connected operation, and outputs the integrated first fully connected information. The output layer normalizes the fully connected information and finally generates multiple text words and their corresponding position distribution information.
[0078] The embodiments provided have at least the following beneficial effects:
[0079] 1. Multi-level feature extraction: Through the dynamic expansion operation of multiple convolutional layers, text features can be extracted from different scales and long-distance dependencies can be captured.
[0080] 5. Rich contextual information: Dynamically expanded convolution expands the receptive field, allowing the model to utilize a wider range of contextual information.
[0081] 3. Feature integration: The fully connected layer integrates the information of multiple convolutional layers, improving the completeness and accuracy of feature expression.
[0082] 4. Normalization: The normalization of the output layer ensures the consistency and usability of the generated results.
[0083] In summary, the provided embodiments further optimize the function of the dynamically expanded convolutional neural network, making the system more comprehensive, efficient and accurate when processing text feature information.
[0084] Exemplarily, the dynamic expansion rate corresponding to each convolutional layer is obtained according to the text feature information, including: obtaining the text sequence length corresponding to the text information, and determining the average pooling dimension according to the text sequence length; average pooling the text feature information according to the average pooling dimension; fully connecting the average pooled text information and outputting the second fully connected information; parsing the second fully connected information to obtain the dynamic expansion rate corresponding to each convolutional layer.
[0085] The example provided first calculates the length of the text sequence based on the input text feature information, and then determines the average pooling dimension required for each convolutional layer. According to the determined average pooling dimension, the text feature information is averaged and pooled to reduce the feature information scale for subsequent processing. The feature information after average pooling is input to the fully connected layer for further integration and output of the second fully connected information. The information output by the fully connected layer is parsed to determine the dynamic expansion rate that should be used for each convolutional layer. Then, by dynamically adjusting the expansion rate of the convolutional layer, the adaptability of the model to texts of different lengths is improved. At the same time, average pooling reduces the scale of feature information and improves computational efficiency.
[0086] And the fully connected layer integrates the feature information after average pooling to enhance the feature expression ability.
[0087] Exemplarily, the dynamically expanded convolutional neural network also includes a residual layer; before each text convolution information is input into the fully connected layer, it also includes: inputting each text convolution information and text feature information into the residual layer respectively, and the residual layer is used to perform residual connection on each text convolution information and the text feature information, so as to input the text convolution information after residual connection into the fully connected layer.
[0088] The example provided first introduces a residual connection layer into the network and provides a residual path. Then the residual layer adds the original text feature information and the text convolution information after convolution to form the information after residual connection. The residual connection can effectively alleviate the gradient vanishing or gradient exploding problems in deep networks, improve the training effect and convergence speed. At the same time, the residual connection can retain the original feature information, avoid information loss, and enhance the representation ability of the model. And the residual path enables the feature information to be transmitted at a deeper level, improving the depth and expression ability of the network.
[0089] Step S105: Generate a text summary corresponding to the text information according to each text word and the corresponding position distribution information.
[0090] Specifically, based on the feature information output from the dynamically dilated convolutional network, words or sentences with higher weights are selected through scoring and sorting. Common methods include rule-based summary generation, sequence-to-sequence (Seq2Seq) models, generative adversarial networks (GANs), etc. The final summary may also be optimized for grammar and coherence through a language model to ensure that the generated text is easy to read. Then, all the previously extracted feature information is integrated, and a concise and accurate text summary is generated by selecting the most important content. This not only improves the efficiency of information acquisition, but also allows users to quickly understand the core content of the text. Through the provided method, text information can be effectively extracted and processed, and finally a high-quality text summary can be generated. While improving the summary generation effect, the result is ensured to have high practicality and accuracy.
[0091] In some embodiments, Figure 3 As shown, before generating a text summary corresponding to the text information according to each text word and the corresponding position distribution information, steps S106 to S108 are also included.
[0092] Step S106. Obtain label information corresponding to the text words; the label information includes real position distribution information corresponding to the text words.
[0093] Step S107: Calculate the cross entropy loss function value of each real position distribution information and position distribution information.
[0094] Step S108. Complete the optimization of the attention module and the dynamically expanded convolutional neural network module according to the cross entropy loss function value.
[0095] By obtaining the true label information of each text word from the training data set. The label information includes the position distribution information of the text word in the real data, which is used to evaluate the accuracy of the model prediction. The text word position distribution information predicted by the model is compared with the true label information, and the cross entropy loss function value is calculated. The cross entropy loss function is often used in classification problems. It guides the model to optimize by evaluating the difference between the predicted distribution and the true distribution. Using the calculated cross entropy loss value as the loss function, the back propagation algorithm is applied to update the model parameters, thereby optimizing the attention module and the dynamically expanded convolutional neural network module to improve the performance of the model.
[0096] The embodiments provided have the following beneficial effects:
[0097] 1. Enhance the generalization ability of the model: By using real label information for training, the generalization ability of the model for unseen data can be effectively improved.
[0098] 2. Improve prediction accuracy: Calculating the cross entropy loss and performing parameter optimization can reduce the difference between the predicted distribution and the true distribution, thereby improving the model's prediction accuracy for text position distribution.
[0099] 3. Optimize module performance: By jointly optimizing the attention module and the dynamically expanded convolutional neural network module, the overall model performance is improved, making summary generation more accurate and efficient.
[0100] Furthermore, by introducing true label information and cross-entropy loss function, the attention module and dynamically expanded convolutional neural network module are optimized, which significantly improves the prediction accuracy and generalization ability of the model.
[0101] Exemplarily, the cross-extraction loss function is where y i is the real location distribution information, is the location distribution information. At the same time, the Adam optimizer can be used for gradient descent during the optimization process.
[0102] In some embodiments, the position distribution information includes the end symbol probability information corresponding to the text word; generating a text summary corresponding to the text information based on each text word and the corresponding position distribution information includes: obtaining a truncation threshold; removing text words whose end symbol probability information is less than the truncation threshold from multiple text words to obtain a target text word; and generating the shown text summary based on the target text word.
[0103] By setting a truncation threshold, it is used to filter the terminal symbol probability information. For example, if the truncation threshold is set to 0.5, the text words with terminal symbol probability less than 0.5 will be ignored. Then, according to the set truncation threshold, all text words are traversed, and those text words with terminal symbol probability information lower than the threshold are removed, and the target text words that meet the conditions are retained. The removed target text words are combined to generate the final text summary. This step ensures that the generated summary only contains relevant text words that may be the end, which enhances the coherence and completeness of the summary.
[0104] The provided embodiment can remove irrelevant text words by using the terminator probability information and the truncation threshold, making the generated summary more compact and coherent, and improving the overall quality of the summary. At the same time, the text words with low terminator probability are automatically screened, which reduces the burden of manual adjustment and screening, and improves the automation and efficiency of summary generation. By removing the text words with low terminator probability, the redundant information in the summary can be effectively reduced, making the summary content more refined and easy to understand.
[0105] Then, by using the terminator probability information in the position distribution information and combining it with the truncation threshold, relevant text words are automatically screened to generate a text summary with higher quality, more coherent and accurate content.
[0106] To facilitate understanding of the specific use process of the method provided in this application, the complete steps of the method provided in this application are now described through an example:
[0107] 1. The text is embedded in the word through the bert model, and the embedded text is represented as X∈R L×d , where L is the sequence length and d is the word embedding dimension.
[0108] 2. The input text sequence is represented as X∈R L×d Enter the attention module, the input passes through the attention layer of the attention module, and the output C∈R L×d , where L is the text length and d is the vector dimension.
[0109] 3. Input C to the Dynamic Dilated Convolutional Neural Network (DDCNN) module: The output C from the attention module based on the dot product attention mechanism. The input C is subjected to n layers of convolution operations to obtain the output H. The convolution formula for each layer is: H i+1 =ReLU(W i*di *H i +b i ). In the formula, H i+1 H represents the output after convolution, and its dimension is L*d, where L represents the text length and d represents the vector length. i represents the output of the previous layer before convolution, b i is the bias term, dimension is L*1, It means to perform convolution operation on H, and its convolution kernel is fixed to 1*2, where 1 corresponds to the dimension of L and 2 corresponds to the dimension of d. And perform dilated convolution in the dimension d. Its expansion rate is di, which is greater than 1 and less than d / 2.
[0110] di is obtained by average pooling the output C (L*1 dimension) and then passing through two fully connected layers to obtain the output D(n*1), where n represents the number of convolutional layers, and each value represents the expansion rate di of the i-th layer. After rounding up, the final di is obtained.
[0111] 4. After each convolution layer, there will be a residual connection H i+1 =H i + C. By connecting the output of each layer to C through a residual connection, the gradient disappearance problem is effectively alleviated.
[0112] After the residual connection, it will pass through a fully connected layer Hi = ReLU (H i-1 W i +b i ).
[0113] 5. The output of the fully connected layer is input as z to the output layer, and the function of the output layer is y = softmax (z). The softmax function is normalized to obtain the word probability distribution at each position, which includes the end symbol [SEP]. By setting the threshold threshold, the leftmost token whose probability of the end symbol [SEP] is greater than the threshold is truncated to obtain the final predicted summary text.
[0114] This application provides a method for generating text summaries based on an attention mechanism and a dynamically expanded convolutional neural network. At the same time, the method provided by this application can be applied in finance or other systems where a large amount of text needs to be summarised (such as medical, education, etc.) to generate more accurate, concise and informative contract / text summaries, helping users quickly obtain key information of documents, and improving user experience and the work efficiency of business personnel.
[0115] See also Figure 4 As shown, Figure 4 1 is a schematic diagram of the structure of a text summary generating device 200 provided in an embodiment of the present application. The text summary generating device 200 is used to execute the steps of the text summary generating method shown in the above embodiments. The text summary generating device 200 can be a single server or a server cluster, or the text summary generating device 200 can be a terminal, which can be a handheld terminal, a laptop computer, a wearable device or a robot, etc.
[0116] like Figure 4 As shown, the text summary generating device 200 includes:
[0117] A text acquisition unit 201, used to acquire text information to be generated;
[0118] A text embedding unit 202, used to embed the text information into words to obtain embedded text;
[0119] A feature output unit 203, configured to input the embedded text into a preset attention module, wherein the attention module parses the embedded text and outputs text feature information;
[0120] A convolution output unit 204 is used to input the text feature information into a preset dynamic expansion convolutional neural network module, and the dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and position distribution information corresponding to each of the text words;
[0121] The summary generating unit 205 is used to generate a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information.
[0122] In some embodiments, the dynamically expanding convolutional neural network module includes multiple convolutional layers, fully connected layers and output layers; the convolution output unit 204 is also used to: obtain the dynamic expansion rate corresponding to each convolution layer according to the text feature information; input the text feature information into each convolution layer respectively, and each convolution layer performs hole convolution on the text feature information according to the corresponding dynamic expansion rate, and outputs text convolution information; input each text convolution information into the fully connected layer, and the fully connected layer fully connects multiple text convolution information and outputs first fully connected information; input the text fully connected information into the output layer, and the output layer normalizes the text fully connected information to obtain multiple text words and the position distribution information corresponding to each of the text words.
[0123] Exemplarily, the convolution output unit 204 is also used to: obtain the text sequence length corresponding to the text information, and determine the average pooling dimension according to the text sequence length; perform average pooling on the text feature information according to the average pooling dimension; fully connect the text information after average pooling, and output second fully connected information; parse the second fully connected information to obtain the dynamic expansion rate corresponding to each of the convolutional layers.
[0124] Exemplarily, the dynamically expanded convolutional neural network also includes a residual layer; before inputting each of the text convolution information into the fully connected layer, the convolution output unit 204 is also used to: input each of the text convolution information and the text feature information into the residual layer respectively, and the residual layer is used to perform a residual connection between each of the text convolution information and the text feature information, so as to input the text convolution information after the residual connection into the fully connected layer.
[0125] In some embodiments, before generating a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information, it also includes: a module optimization unit, used to obtain label information corresponding to the text word; the label information includes the real position distribution information corresponding to the text word; calculate the cross entropy loss function value of each of the real position distribution information and the position distribution information; and optimize the attention module and the dynamically expanded convolutional neural network module according to the cross entropy loss function value.
[0126] In some embodiments, the position distribution information includes the end symbol probability information corresponding to the text word; the summary generation unit 205 is also used to obtain a truncation threshold; remove the text words whose end symbol probability information is less than the truncation threshold from the multiple text words to obtain the target text word; and generate the text summary shown according to the target text word.
[0127] In some embodiments, the text embedding unit 202 is further used to: input the text information into a preset language representation model, the language representation model converts the text information into a dense vector form, and outputs the embedded text.
[0128] It should be noted that those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the text summary generation device and each module described above can refer to the corresponding process in the text summary generation method embodiments described in the above embodiments, and will not be repeated here.
[0129] The above-mentioned text summary generation method can be implemented in the form of a computer program. The computer program can be used in Figure 4 Run on the device shown.
[0130] See also Figure 5 , Figure 5 1 is a schematic block diagram of the structure of a computer device provided in an embodiment of the present application. The computer device includes a processor, a memory and a network interface connected via a device bus, wherein the memory may include a storage medium and an internal memory.
[0131] The storage medium can store an operating device and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any text summary generation method.
[0132] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.
[0133] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any text summary generation method.
[0134] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will appreciate that Figure 5 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal to which the scheme of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0135] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.
[0136] In one embodiment, the processor is used to run a computer program stored in the memory to implement the following steps:
[0137] Get the text information to be generated;
[0138] Embedding the text information into words to obtain embedded text;
[0139] Inputting the embedded text into a preset attention module, the attention module parses the embedded text and outputs text feature information;
[0140] Inputting the text feature information into a preset dynamic expansion convolutional neural network module, the dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and position distribution information corresponding to each of the text words;
[0141] A text summary corresponding to the text information is generated according to each of the text words and the corresponding position distribution information.
[0142] In some embodiments, the dynamically expanding convolutional neural network module includes multiple convolutional layers, fully connected layers and an output layer; the text feature information is input into a preset dynamically expanding convolutional neural network module, including: obtaining the dynamic expansion rate corresponding to each convolutional layer according to the text feature information; the text feature information is input into each of the convolutional layers respectively, and each of the convolutional layers performs a hole convolution on the text feature information according to the corresponding dynamic expansion rate, and outputs text convolution information; each of the text convolution information is input into a fully connected layer, and the fully connected layer performs a full connection on multiple text convolution information and outputs a first fully connected information; the text fully connected information is input into the output layer, and the output layer normalizes the text fully connected information to obtain multiple text words and the position distribution information corresponding to each of the text words.
[0143] Exemplarily, the method of obtaining the dynamic expansion rate corresponding to each convolutional layer according to the text feature information includes: obtaining the text sequence length corresponding to the text information, and determining the average pooling dimension according to the text sequence length; performing average pooling on the text feature information according to the average pooling dimension; performing full connection on the text information after average pooling, and outputting second fully connected information; and parsing the second fully connected information to obtain the dynamic expansion rate corresponding to each convolutional layer.
[0144] Exemplarily, the dynamically expanded convolutional neural network also includes a residual layer; before inputting each of the text convolution information into the fully connected layer, it also includes: inputting each of the text convolution information and the text feature information into the residual layer respectively, and the residual layer is used to perform a residual connection between each of the text convolution information and the text feature information, so as to input the text convolution information after the residual connection into the fully connected layer.
[0145] In some embodiments, before generating a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information, it also includes: obtaining label information corresponding to the text word; the label information includes the real position distribution information corresponding to the text word; calculating the cross entropy loss function value of each of the real position distribution information and the position distribution information; and completing the optimization of the attention module and the dynamically expanding convolutional neural network module according to the cross entropy loss function value.
[0146] In some embodiments, the position distribution information includes the end symbol probability information corresponding to the text word; generating the text summary corresponding to the text information based on each of the text words and the corresponding position distribution information includes: obtaining a truncation threshold; removing the text words whose end symbol probability information is less than the truncation threshold from the multiple text words to obtain the target text word; and generating the text summary according to the target text word.
[0147] In some embodiments, embedding the text information into words to obtain embedded text includes: inputting the text information into a preset language representation model, the language representation model converting the text information into a dense vector form, and outputting the embedded text.
[0148] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a computer program. The computer program includes program instructions. The processor executes the program instructions to implement the steps of the text summary generation method provided in the above embodiments of the present application.
[0149] The computer-readable storage medium may be an internal storage unit of the computer device described in the foregoing embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the computer device.
[0150] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A text summary generation method, characterized in that: include: Get the text information to be generated; Embedding the text information into words to obtain embedded text; Inputting the embedded text into a preset attention module, the attention module parses the embedded text and outputs text feature information; Inputting the text feature information into a preset dynamic expansion convolutional neural network module, the dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and position distribution information corresponding to each of the text words; A text summary corresponding to the text information is generated according to each of the text words and the corresponding position distribution information.
2. The method according to claim 1, characterized in that The dynamically expanding convolutional neural network module includes a plurality of convolutional layers, a fully connected layer and an output layer; the step of inputting the text feature information into a preset dynamically expanding convolutional neural network module includes: Obtaining a dynamic expansion rate corresponding to each convolutional layer according to the text feature information; Inputting the text feature information into each of the convolutional layers respectively, each of the convolutional layers performs a hole convolution on the text feature information according to the corresponding dynamic expansion rate, and outputs the text convolution information; Input each of the text convolution information into a fully connected layer, the fully connected layer fully connects the multiple text convolution information and outputs the first fully connected information; The text full connection information is input into the output layer, and the output layer normalizes the text full connection information to obtain multiple text words and position distribution information corresponding to each of the text words.
3. The method according to claim 2, characterized in that The obtaining the dynamic expansion rate corresponding to each convolutional layer according to the text feature information includes: Obtaining a text sequence length corresponding to the text information, and determining an average pooling dimension according to the text sequence length; Performing average pooling on the text feature information according to the average pooling dimension; Performing full connection on the text information after average pooling, and outputting second fully connected information; Parse the second fully connected information to obtain a dynamic expansion rate corresponding to each of the convolutional layers.
4. The method according to claim 2, characterized in that: The dynamically expanded convolutional neural network also includes a residual layer; before inputting each of the text convolution information into the fully connected layer, it also includes: Each of the text convolution information and the text feature information is input into the residual layer respectively, and the residual layer is used to perform residual connection on each of the text convolution information and the text feature information, so as to input the text convolution information after residual connection into the fully connected layer.
5. The method according to claim 1, characterized in that Before generating a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information, the method further includes: Acquire label information corresponding to the text word; the label information includes real position distribution information corresponding to the text word; Calculate the cross entropy loss function value of each of the true position distribution information and the position distribution information; The optimization of the attention module and the dynamically expanded convolutional neural network module is completed according to the cross entropy loss function value.
6. The method according to claim 1, characterized in that The position distribution information includes the terminator probability information corresponding to the text word; and generating a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information includes: Get the cutoff threshold; Removing text words whose end symbol probability information is less than the cutoff threshold from the plurality of text words to obtain a target text word; Generates a summary of the text based on the target text words.
7. The method according to claim 1, characterized in that The word embedding of the text information to obtain the embedded text includes: The text information is input into a preset language representation model, and the language representation model converts the text information into a dense vector form and outputs the embedded text.
8. A text summary generation device, characterized in that: include: A text acquisition unit, used to acquire text information to be generated; A text embedding unit, used to embed the text information into words to obtain embedded text; A feature output unit, used to input the embedded text into a preset attention module, and the attention module parses the embedded text and outputs text feature information; A convolution output unit, used to input the text feature information into a preset dynamic expansion convolutional neural network module, the dynamic expansion convolutional neural network module performs multiple convolutions on the text feature information according to a preset dynamic expansion rate, and outputs multiple text words and position distribution information corresponding to each of the text words; The summary generating unit is used to generate a text summary corresponding to the text information according to each of the text words and the corresponding position distribution information.
9. A computer device, characterized in that: The computer device includes a memory and a processor; The memory is used to store computer programs; The processor is configured to execute the computer program and implement the method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the method according to any one of claims 1 to 7.