An automatic text summarization method and device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- 太保科技有限公司
- Filing Date
- 2022-11-30
- Publication Date
- 2026-06-26
Smart Images
Figure CN115730061B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to an automatic text summarization method and apparatus. Background Technology
[0002] Internet technology has accelerated the collection and dissemination of information, ushering in an era of information explosion. While the abundance and diversity of information resources bring great convenience to people's lives, the sheer volume of information also presents significant challenges. Quickly extracting the desired information from trillions of data points on the internet has become a difficult task. Generally, unprocessed original texts contain excessive redundant information, requiring lengthy reading times, while the truly needed information is often obscured by irrelevant details, making information retrieval difficult for users. In this era of rapid information growth on the internet, researching an automatic text summarization method to extract key information from text can improve users' information retrieval and reading efficiency, facilitating their work and daily lives.
[0003] Current automatic document summarization solutions are all based on deep learning models. First, the extremely long text is segmented according to rules, and then the summary text is selected. This method requires designing a large number of manually generated features for model training, which has low training efficiency and poor generalization. Secondly, this method easily splits semantically continuous segments into two parts, resulting in poor semantic integrity, that is, the text is not fluent. In the end, the resulting text summary cannot accurately extract important and effective information from extremely long texts. Summary of the Invention
[0004] Based on this, this application proposes an automatic text summarization method and apparatus, which aims to improve the semantic integrity of the final text summary.
[0005] Compared with the prior art, this application has the following beneficial effects:
[0006] This application first obtains the target document and extracts the main text from the target document; then, it segments the main text into paragraphs and treats each segment as an independent information point; next, it performs semantic recognition on each information point to obtain the semantics corresponding to each information point; then, it compares the semantics corresponding to each information point with the target semantics to obtain a semantic comparison result; finally, it splices the information points with consistent semantic comparison results in a preset order to obtain a text summary of the target document.
[0007] This application obtains a text summary by first acquiring the main text of the target document and segmenting it, then performing semantic analysis on the segmented text, and finally comparing it with the target semantics. The resulting text summary has higher semantic consistency with the target semantics and improves the semantic integrity of the text summary. Attached Figure Description
[0008] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0009] Figure 1 A flowchart of one method of the automatic text summarization method provided in the embodiments of this application;
[0010] Figure 2 A flowchart of one method of the information point splicing method provided in the embodiments of this application;
[0011] Figure 3 This is a schematic diagram of an automatic text summarization device provided in an embodiment of this application. Detailed Implementation
[0012] In existing technologies, automatic document summarization solutions are all based on deep learning models. First, the extremely long text is segmented according to rules, and then the summary text is selected. The aforementioned rules are generally based on the position of the text in the paragraph or the length of the preset text segment, and then the extremely long text is segmented.
[0013] Research shows that the existing technology requires a large number of manually designed features for model training, resulting in low training efficiency and poor generalization. Furthermore, this method easily splits semantically continuous segments into two parts, leading to poor semantic integrity and incoherent text. Consequently, the resulting text summary cannot accurately extract important and effective information from extremely long texts.
[0014] This application embodiment first obtains a target document and extracts the main text from the target document; then, the main text is segmented into paragraphs, and each segment is treated as an independent information point; then, semantic recognition is performed on each information point to obtain the semantics corresponding to each information point; then, the semantics corresponding to each information point are compared with the target semantics to obtain a semantic comparison result; finally, the information points with consistent semantic comparison results are spliced together in a preset order to obtain a text summary of the target document.
[0015] This application embodiment obtains the main text of the target document and segments it, then performs semantic analysis on the segmented main text, compares it with the target semantics, and finally obtains a text summary. The resulting text summary has higher semantic consistency with the target semantics and improves the semantic integrity of the text summary.
[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.
[0017] See Figure 1 , Figure 1 A flowchart of an automatic text summarization method provided in this application embodiment includes:
[0018] S101: Obtain the target document.
[0019] The target document refers to the document for which text summarization is required. This document can be a long document or a short document. Text summarization refers to extracting important text segments from the document content. Important text segments are text segments that summarize the central idea of the entire document. They can be a sentence, a paragraph, or a summary of other lengths.
[0020] This step is primarily to obtain the objectives of the text summarization task.
[0021] S102: Obtain the main text from the target document.
[0022] The main text refers to the useful document information after removing useless information such as watermarks, titles, headers and footers. The watermark is a unique identifier added to the background of the published document by the newspaper to prevent plagiarism.
[0023] In one possible implementation, retrieving the body text from the target document includes:
[0024] Retrieve the text segments contained in the target document;
[0025] Obtain the content features of the text segment;
[0026] When the content features meet the preset conditions, the text segment is determined as the main text.
[0027] The content features include any one or more of the following: relative position coordinates, average pixel area of the text contained in the text segment, offset of the center point of the text segment, and text features contained in the text segment.
[0028] Relative position coordinates refer to establishing a rectangular coordinate system with any point in the document as the origin, thereby using coordinates to describe the position of each text.
[0029] The average pixel area of the text contained in a text segment refers to the size of the text.
[0030] The text segment center point offset is mainly used to detect whether the text is fully recognized;
[0031] The text features contained in a text segment refer to whether the text is purely text or purely numeric, for example, a phone number is purely numeric, while an email address may contain special characters.
[0032] In one possible implementation, when the content features meet preset conditions, determining the text segment as body text includes:
[0033] When the relative position coordinates of a text segment are within the target coordinate range, the text segment is determined to be the main text.
[0034] When the average pixel area of the characters contained in a text segment is less than the pixel area threshold, the text segment is determined to be the main text.
[0035] When the center point offset of a text segment is less than the center point offset threshold, the text segment is determined to be the main text.
[0036] When the text features contained in a text segment match the target text features, the text segment is determined to be the main text.
[0037] When the relative position coordinates of a text segment are not within the target coordinate range, such as when the coordinates are too large or too small, it can be identified as the header or footer of the document, which is useless information.
[0038] When the average pixel area of the text contained in a text segment is greater than the pixel area threshold, it can be identified as a document title, which is useless information;
[0039] Since watermarks are usually placed in the background of a document as slanted text, when the offset of the center point of a text segment is greater than the center point offset threshold, it can be identified as a watermark, which is useless information.
[0040] The target text feature refers to a continuous mixture of text and numbers. If a single line of pure numbers or special characters, such as the "@" character in an email address, appears, it can be considered useless information.
[0041] By using the above preset conditions, the text to be included in the abstract can be determined.
[0042] S103: Divide the main text into paragraphs and treat each paragraph as an independent information point.
[0043] Since each document contains at least one paragraph of body text, segmenting it by paragraph can divide the entire document into multiple parts, reducing the computational load for subsequent semantic recognition and semantic comparison steps.
[0044] In one possible implementation, the segmented text can be further segmented, and a symbol recognition model can be used to identify the period in the segment as the segmentation basis to divide the segment into at least one sentence before proceeding with the subsequent semantic recognition steps.
[0045] S104: Perform semantic recognition on each information point to obtain the semantics corresponding to each information point.
[0046] Semantic recognition refers to identifying the meaning or theme contained in a sentence or paragraph. In this embodiment, the Roberta matrix is used to convert the text into a vector representation, and then deep learning operations are performed to ultimately achieve the purpose of semantic recognition.
[0047] S105: Compare the semantics corresponding to each information point with the target semantics to obtain the semantic comparison results.
[0048] Target semantics refers to the theme that needs to be extracted from a document. For example, for an economic research report, the target semantics should be set to economic research data, etc.
[0049] In one possible implementation, the semantics corresponding to each information point are compared with the target semantics to obtain semantic comparison results, including:
[0050] Determine whether the semantics of each information point are consistent with the target semantics. If they are, mark the information point as the first identifier; otherwise, mark the information point as the second identifier.
[0051] The semantic comparison results are obtained based on the first identifier and the second identifier.
[0052] The first identifier and the second identifier can be assigned values, such as the first identifier being assigned the value 1 and the second identifier being assigned the value 0, or any other form that can identify information points.
[0053] S106: Concatenate the information points with consistent semantic comparison results in a preset order to obtain the text summary of the target document.
[0054] The preset order can be the paragraph order before the document is cut, the order of splicing according to semantic coherence, or other splicing methods.
[0055] This application embodiment obtains the main text of the target document and segments it, then performs semantic analysis on the segmented main text, compares it with the target semantics, and finally obtains a text summary. The resulting text summary has higher semantic consistency with the target semantics and improves the semantic integrity of the text summary.
[0056] In the embodiments of this application, the above Figure 1There are several possible implementations of step S106, which will be described below. It should be noted that the implementations given below are merely illustrative examples and do not represent all implementations of the embodiments of this application.
[0057] See Figure 2 The diagram shows a flowchart of an information point splicing method, which includes:
[0058] S201: Encode the text containing all information points with consistent semantic comparison results using a text concatenation processing model.
[0059] In this embodiment of the application, the text concatenation processing model can be implemented using the T5 (Transfer Text-to-TextTransformer) model. The T5 model can implement various methods for text processing, including defining and scoring the semantic relevance between different sentences. Of course, other models capable of text processing can also be used for text processing.
[0060] The purpose of encoding is to convert natural language into machine language that can be recognized by machines.
[0061] S202: Semantically extract the text contained in each encoded information point, and determine the semantic relevance between each text through calculation.
[0062] Among them, relevance characterizes the degree of semantic continuity of each text. For example, if two sentences A and B have similar semantics, and sentence B begins with a conjunction, then A and B can be combined in the order of AB and their semantics can be re-identified to determine whether the final semantics are the same as those of A and B. If the semantics are the same, it means that A and B have a high degree of continuity, that is, high relevance. Of course, relevance and continuity can also be negatively correlated, that is, high continuity and low relevance, which is also within the scope of protection of this application.
[0063] Since semantic recognition has already been performed on the information point text, this step only requires extracting the recognized semantics.
[0064] S203: Concatenate the texts according to their relevance to obtain an undecoded text summary.
[0065] The purpose of splicing texts according to relevance is to improve the semantic coherence and completeness of the text summary.
[0066] S204: Decode the undecoded text summary to obtain the decoded text summary of the target document.
[0067] The purpose of this decoding step is to convert machine language into natural language for output.
[0068] This application embodiment uses a text splicing processing model to extract the semantics contained in the information points in the previous embodiment, then evaluates the relevance between the semantics of each information point, and finally splices the text of each information point according to the relevance to obtain a text summary, thereby improving the semantic coherence and completeness of the final text summary.
[0069] The above are some specific implementations of the automatic text summarization method provided in the embodiments of this application. Based on this, this application also provides a corresponding apparatus. The apparatus provided in the embodiments of this application will be described below from the perspective of functional modularity.
[0070] See Figure 3 The diagram shown illustrates the structure of an automatic text summarization device, which includes:
[0071] The target document acquisition module 301 is used to acquire the target document;
[0072] The main text acquisition module 302 is used to acquire the main text from the target document;
[0073] The text segmentation module 303 is used to segment the main text into paragraphs and treat each segment as an independent information point.
[0074] The semantic recognition module 304 is used to perform semantic recognition on each information point to obtain the semantics corresponding to each information point;
[0075] The semantic comparison module 305 is used to compare the semantics corresponding to each information point with the target semantics to obtain the semantic comparison result;
[0076] The information point splicing module 306 is used to splice information points with consistent semantic comparison results in a preset order to obtain a text summary of the target document.
[0077] In one possible implementation, the text acquisition module 302 includes:
[0078] A text segment acquisition unit is used to acquire text segments contained in the target document;
[0079] The content feature acquisition unit is used to acquire the content features of the text segment, wherein the content features include any one or more of the following: relative position coordinates, average pixel area of the characters contained in the text segment, offset of the center point of the text segment, and character features contained in the text segment.
[0080] The main text determination unit is used to determine the text segment as main text when the content features meet preset conditions.
[0081] In one possible implementation, the text determination unit includes:
[0082] The first determining subunit is used to determine that the text segment is the main text when the relative position coordinates of the text segment are within the target coordinate range;
[0083] The second determining subunit is used to determine that the text segment is body text when the average pixel area of the characters contained in the text segment is less than the pixel area threshold.
[0084] The third determining subunit is used to determine that the text segment is the main text when the center point offset of the text segment is less than the center point offset threshold.
[0085] The fourth determining subunit is used to determine that the text segment is the main text when the text features contained in the text segment contain the target text features.
[0086] In one possible implementation, the semantic comparison module 305 includes:
[0087] The judgment unit is used to determine whether the semantics of each information point is consistent with the target semantics. If it is, the information point is marked as the first identifier; otherwise, the information point is marked as the second identifier.
[0088] The comparison result acquisition unit is used to obtain the semantic comparison result based on the first identifier and the second identifier.
[0089] In one possible implementation, the information point splicing module 306 includes:
[0090] The encoding unit is used to encode the text contained in all information points with consistent semantic comparison results using a text concatenation processing model;
[0091] The relevance determination unit is used to extract the semantics of the text contained in each encoded information point and determine the relevance between the semantics of each text through calculation. The relevance represents the degree of continuity of the semantics of each text.
[0092] A text splicing unit is used to splice the texts according to the relevance to obtain an undecoded text summary;
[0093] The decoding unit is used to decode the undecoded text digest to obtain the decoded text digest of the target document.
[0094] This application embodiment obtains the main text of the target document and segments it, then performs semantic analysis on the segmented main text, compares it with the target semantics, and finally obtains a text summary. The resulting text summary has higher semantic consistency with the target semantics and improves the semantic integrity of the text summary.
[0095] This application also provides corresponding devices and computer storage media for implementing the solutions provided in this application.
[0096] The device includes a memory and a processor. The memory stores instructions or code, and the processor executes the instructions or code to enable the device to perform the automatic text summarization method described in any embodiment of this application.
[0097] The computer storage medium stores code, and when the code is run, the device running the code implements the method described in any embodiment of this application.
[0098] In the embodiments of this application, the terms "first" and "second" (if they exist) are used only as name identifiers and do not represent the order of first and second.
[0099] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that all or part of the steps in the methods of the above embodiments can be implemented by means of software plus a general-purpose hardware platform. Based on this understanding, the technical solution of this application can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as a read-only memory (ROM) / RAM, magnetic disk, optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, a server, or a network communication device such as a router) to execute the methods described in various embodiments or some parts of the embodiments of this application.
[0100] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on its differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.
[0101] The above description is merely an exemplary implementation of this application and is not intended to limit the scope of protection of this application.
Claims
1. An automatic text summarization method, characterized in that, The method includes: Obtain the target document; Obtain the text segments contained in the target document; Obtain the content features of the text segment; wherein, the content features include any one or more of the following: relative position coordinates, average pixel area of the characters contained in the text segment, offset of the center point of the text segment, and character features contained in the text segment; When the content features meet the preset conditions, the text segment is determined as the main text. The main text is divided into paragraphs, and each paragraph is treated as an independent information point. Semantic recognition is performed on each information point to obtain the semantic meaning corresponding to each information point; The semantics corresponding to each information point are compared with the target semantics to obtain the semantic comparison results; The text is encoded using a text concatenation processing model to represent all information points that have the same semantic comparison results. Semantic extraction is performed on the text contained in each encoded information point, and the correlation between the semantics of each text is determined by calculation. The correlation represents the degree of continuity of the semantics of each text. The texts are concatenated according to their relevance to obtain an undecoded text summary; The undecoded text digest is decoded to obtain the decoded text digest of the target document.
2. The method according to claim 1, characterized in that, The step of determining the text segment as body text when the content features meet preset conditions includes: When the relative position coordinates of the text segment are within the target coordinate range, the text segment is determined to be the main text. When the average pixel area of the characters contained in the text segment is less than the pixel area threshold, the text segment is determined to be body text. When the center point offset of the text segment is less than the center point offset threshold, the text segment is determined to be the main text. When the text segment contains text features that match the target text features, the text segment is determined to be the main text.
3. The method according to claim 1, characterized in that, The step of comparing the semantics corresponding to each information point with the target semantics to obtain the semantic comparison result includes: Determine whether the semantics of each information point are consistent with the target semantics. If they are, mark the information point as the first identifier; otherwise, mark the information point as the second identifier. The semantic comparison results are obtained based on the first identifier and the second identifier.
4. An automatic text summarization device, characterized in that, The device includes: The target document acquisition module is used to acquire the target document. The main text acquisition module is used to acquire text segments contained in the target document; acquire content features of the text segments; and determine the text segment as main text when the content features meet preset conditions; wherein, the content features include any one or more of the following: relative position coordinates, average pixel area of the characters contained in the text segment, offset of the center point of the text segment, and character features contained in the text segment; The text segmentation module is used to segment the main text into paragraphs and treat each segment as an independent information point. A semantic recognition module is used to perform semantic recognition on each information point to obtain the semantics corresponding to each information point; The semantic comparison module is used to compare the semantics corresponding to each information point with the target semantics to obtain the semantic comparison result; The information point concatenation module is used to encode the text contained in all information points with consistent semantic comparison results using a text concatenation processing model; extract the semantics of the encoded text contained in each information point, and determine the relevance between the semantics of each text through calculation, wherein the relevance characterizes the degree of continuity of the semantics of each text; concatenate the texts according to the relevance to obtain an undecoded text summary; and decode the undecoded text summary to obtain the decoded text summary of the target document.
5. An automatic text summarization device, characterized in that, The device includes: A memory for storing the instructions or code for the automatic text summarization; A processor for executing the instructions or code for the automatic text summarization to implement the automatic text summarization method according to any one of claims 1-3.
6. A computer storage medium, characterized in that, The computer storage medium stores code, and when the code is executed, the device running the code implements the automatic text summarization method according to any one of claims 1-3.