A method and system for processing PDF announcement documents

Abstract

The present invention discloses a method and system for processing an announcement document. Based on the format and content of an acquired PDF announcement document, the PDF announcement document is judged to be a standardized PDF document or a non-standardized PDF document. When the PDF announcement document is a standardized PDF document, the PDF announcement document is converted from PDF format to a temporary Word announcement document in docx format. A preset coordinate algorithm is used to segment the announcement domain information of the PDF announcement document and the Word announcement document, and after the segmentation, the formatted announcement plain text corresponding to the PDF announcement document is obtained. The present invention utilizes the body line object and table object in the Word announcement document, and effectively distinguishes the body line object and table object in the PDF announcement document when the preset coordinate algorithm is used, so that the body line object and the table object after the PDF announcement document conversion are distinguished and isolated from each other, meeting the requirements of natural language processing technology.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and more specifically, to a method and system for processing PDF announcement documents. Background Art

[0002] In the actual securities lending business in the securities market, various unstructured PDF announcement documents have a greater impact on the securities market than structured financial statement data. Therefore, using natural language processing technology to study the impact of PDF announcement documents on the securities market can provide strong support for the development of securities lending business and risk management.

[0003] At present, various algorithms and system inputs in natural language processing technology are required to be in the form of strings of plain text. However, the PDF announcement documents released by major stock exchanges in the securities market are all in PDF format. For PDF announcement documents in PDF format, there are many free or paid tools on the market that can convert PDF announcement documents in PDF format into plain text without character loss, such as TXT files or Word files. However, during the format conversion process, some of the original content or format (such as blank lines, headers, footers, page numbers, line breaks, etc.) in the PDF announcement document in PDF format are also retained, making the text content obtained after the conversion present a chaotic format, resulting in the plain text obtained after the format conversion still unable to meet the input information required by natural language processing technology.

[0004] In summary, how to convert a PDF announcement document from PDF format into the plain text (or formatted announcement plain text) required by natural language processing technology has become a technical problem that technical personnel in this field urgently need to solve. Summary of the invention

[0005] In view of this, the present invention discloses a method and system for processing announcement documents, so as to realize that the announcement plain text obtained after the PDF announcement document is converted meets the requirements of natural language processing technology.

[0006] A method for processing an announcement document, comprising:

[0007] Get the PDF announcement document;

[0008] Based on the format and content of the PDF announcement document, determining whether the PDF announcement document is a standardized PDF document or a non-standardized PDF document;

[0009] When the PDF announcement document is the standardized PDF document, converting the PDF announcement document from PDF format to a temporary Word announcement document in docx format, wherein the Word announcement document includes a text line object and a table object;

[0010] For the PDF announcement document and the Word announcement document, a preset coordinate algorithm is used to segment the announcement domain information, and after the segmentation is completed, the formatted announcement plain text corresponding to the PDF announcement document is obtained, wherein the announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text.

[0011] Optionally, when the PDF announcement document is the standardized PDF document, converting the PDF announcement document from PDF format to a temporary Word document in docx format includes:

[0012] When the PDF announcement document is the standardized PDF document, the open source module pdf2docx is used to extract the non-table content in the PDF announcement document to obtain the text line object, and the table content in the PDF announcement document is extracted to obtain the table object;

[0013] The Word document is obtained based on the text row object and the table object.

[0014] Optionally, the PDF announcement document and the Word announcement document are segmented using a preset coordinate algorithm, and after the segmentation is completed, a formatted announcement plain text corresponding to the PDF announcement document is obtained, including:

[0015] Scan the PDF announcement document, and extract text block elements of the PDF announcement document page by page and line by line, wherein the text block elements include: text characters, upper left corner coordinate values ​​and lower right corner coordinate values ​​of a text block rectangle;

[0016] The text block elements extracted page by page and line by line are stored in a pre-established ordered dictionary according to the extraction order;

[0017] The body line objects and the table objects in the Word announcement document are scanned one by one to obtain the document elements in the Word announcement document, and the formatted announcement plain text is obtained based on the document elements and the ordered dictionary, wherein the document elements include: the text string and coordinate value list corresponding to the body line object, and the string in the cell corresponding to the table object.

[0018] Optionally, the body line objects and the table objects in the Word announcement document are scanned one by one to obtain document elements in the Word announcement document, and formatted announcement plain text is obtained based on the document elements and the ordered dictionary, including:

[0019] Scan the Word bulletin document, sequentially obtain a line of text string from the text line object, and obtain a coordinate value list with the text string as a key from the ordered dictionary to form a tuple consisting of the text string and the coordinate value list;

[0020] Matching the two-tuple in a preset rule base, and formatting the two-tuple using the matched target rule to obtain an initial announcement plain text, wherein the preset rule base contains two types of rules, namely, coordinate-related rules and coordinate-independent rules;

[0021] Scan the Word announcement document and obtain each table sub-object from the table object in sequence;

[0022] Traversing the table sub-object row by row and cell by cell, extracting and correcting the character string in each cell of the table sub-object to obtain the target cell;

[0023] Split the character string in each target cell with a tab character, together forming a row of content of the table sub-object, and add it to the initial announcement plain text;

[0024] When a row of content corresponding to all the table sub-objects in the table object is added to the initial announcement plain text, the final formatted announcement plain text is obtained.

[0025] Optionally, the two-tuple is matched in a preset rule library, and the two-tuple is formatted using the matched target rule to obtain an initial announcement plain text, including:

[0026] Matching the coordinate value list with the coordinate related rules in the preset rule library, and formatting the coordinate value list using the matched target coordinate related rules to obtain a target coordinate value list;

[0027] Matching the text string with the coordinate-independent rule in the preset rule library, and formatting the text string using the matched target coordinate-independent rule to obtain a target text string;

[0028] The initial public plain text is obtained based on all the target coordinate value lists and the target text string.

[0029] Optionally, also include:

[0030] When the PDF announcement document is the non-standardized PDF document, extracting a picture from the PDF announcement document;

[0031] Using optical character recognition technology to recognize characters in the image line by line, and obtaining the initialization plain text corresponding to the PDF announcement document according to the line by line recognition results;

[0032] The announcement domain information is segmented on the initialization plain text using a preset rule algorithm to obtain the announcement plain text.

[0033] Optionally, the initialization plain text is segmented into announcement domain information using a preset rule algorithm to obtain the announcement plain text, including:

[0034] Extracting a line of main text data, a line of text data immediately above the main text data, and a line of text data immediately below the main text data from the initialization plain text in order to obtain a text triple;

[0035] Matching the text triples with non-statistical rules in a preset rule base, and formatting the body text data using the matched target non-statistical rules to obtain target body text data;

[0036] This process is repeated until all the text data in the initialization plain text are formatted, and the first round of scanning of the initialization plain text is completed;

[0037] Perform a second round of scanning on the initialization plain text, calculate the statistical value and threshold value corresponding to the initialization plain text after the first round of scanning according to the statistical rules in the preset rule base, and write them into the statistical rules;

[0038] Matching the target body text data with the statistical rules, and formatting the target body text data using the matched target statistical rules;

[0039] This process is repeated until all target text data obtained from the first round of scanning in the initialization plain text are formatted to obtain the final announcement plain text.

[0040] A system for processing announcement documents, comprising:

[0041] A document acquisition unit, used to acquire PDF announcement documents;

[0042] A judging unit, configured to judge whether the PDF announcement document is a standardized PDF document or a non-standardized PDF document based on the format and content of the PDF announcement document;

[0043] A conversion unit, used for converting the PDF announcement document from PDF format to a temporary Word announcement document in docx format when the PDF announcement document is the standardized PDF document, wherein the Word announcement document includes a text line object and a table object;

[0044] The first segmentation unit is used to segment the announcement domain information of the PDF announcement document and the Word announcement document by using a preset coordinate algorithm, and obtain the formatted announcement plain text corresponding to the PDF announcement document after the segmentation is completed, wherein the announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text.

[0045] Optionally, also include:

[0046] An extraction unit, configured to extract a picture from the PDF announcement document when the PDF announcement document is the non-standardized PDF document;

[0047] A recognition unit, used to recognize characters in the image line by line using optical character recognition technology, and obtain the initialization plain text corresponding to the PDF announcement document according to the line by line recognition results;

[0048] The second segmentation unit is used to segment the announcement domain information of the initialization plain text by using a preset rule algorithm to obtain the announcement plain text.

[0049] It can be seen from the above technical scheme that the present invention discloses a method and system for processing announcement documents. Based on the format and content of the acquired PDF announcement document, the PDF announcement document is judged to be a standardized PDF document or a non-standardized PDF document. When the PDF announcement document is a standardized PDF document, the PDF announcement document is converted from PDF format to a temporary Word announcement document in docx format. The announcement domain information of the PDF announcement document and the Word announcement document is segmented by a preset coordinate algorithm, and the formatted announcement plain text corresponding to the PDF announcement document is obtained after the segmentation. The announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text. The temporary Word announcement document converted in the present invention is an intermediate document, not the final formatted announcement document. By using the text line object and table object in the Word announcement document, the text line object and table object in the PDF announcement document can be effectively distinguished when the preset coordinate algorithm is used, so that the text line object and the table object after the PDF announcement document conversion are distinguished and isolated from each other, and the data of the non-complex table in the announcement document is formatted and restored to the plain text form, so that the converted announcement plain text meets the needs of natural language processing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without paying creative work.

[0051] Figure 1 A flowchart of a method for processing an announcement document disclosed in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of information of each announcement field in a PDF announcement document disclosed in an embodiment of the present invention;

[0053] Figure 3 A simplified diagram of the core concept of a preset coordinate algorithm disclosed in an embodiment of the present invention;

[0054] Figure 4 A flow chart of another method for processing an announcement document disclosed in an embodiment of the present invention;

[0055] Figure 5 A plain text schematic diagram disclosed in the prior art;

[0056] Figure 6 A brief schematic diagram of the core idea of ​​a preset rule algorithm disclosed in an embodiment of the present invention;

[0057] FIG. 7( a ) is a schematic diagram of an example of a PDF securities announcement before formatting and extraction of the body content disclosed in an embodiment of the present invention;

[0058] FIG. 7( b ) is a schematic diagram showing an example of formatted and extracted text content of a PDF securities announcement disclosed in an embodiment of the present invention;

[0059] FIG8( a ) is a schematic diagram of a table content of a PDF securities announcement before formatting and extraction according to an embodiment of the present invention;

[0060] FIG8( b ) is a schematic diagram of a formatted and extracted table content of a PDF securities announcement disclosed in an embodiment of the present invention;

[0061] Fig. 9 A schematic diagram of the structure of a system for processing announcement documents disclosed in an embodiment of the present invention;

[0062] Fig.10 A schematic diagram of the structure of another system for processing announcement documents disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] The embodiment of the present invention discloses a method and system for processing an announcement document. Based on the format and content of an acquired PDF announcement document, the PDF announcement document is judged to be a standardized PDF document or a non-standardized PDF document. When the PDF announcement document is a standardized PDF document, the PDF announcement document is converted from PDF format to a temporary Word announcement document in docx format. The announcement domain information of the PDF announcement document and the Word announcement document is segmented by a preset coordinate algorithm, and the formatted announcement plain text corresponding to the PDF announcement document is obtained after the segmentation. The announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text. The temporary Word announcement document obtained by conversion in the present invention is an intermediate document, not the final formatted announcement document. By using the text line object and table object in the Word announcement document, the text line object and table object in the PDF announcement document can be effectively distinguished when the preset coordinate algorithm is used, so that the text line object and the table object after the PDF announcement document conversion are distinguished and isolated from each other, and the data of the non-complex table in the announcement document is formatted and restored to the plain text form, so that the converted announcement plain text meets the requirements of natural language processing technology.

[0065] See also Figure 1 , a flow chart of a method for processing an announcement document disclosed in an embodiment of the present invention, the method comprising:

[0066] Step S101, obtaining a PDF announcement document;

[0067] PDF announcement documents are securities market announcement documents in PDF format.

[0068] Step S102: Based on the format and content of the PDF announcement document, determine whether the PDF announcement document is a standardized PDF document or a non-standardized PDF document;

[0069] In this embodiment, the judgment process is mainly divided into two steps:

[0070] (1) Whether the PDF announcement document can be opened correctly. The main process is: open the PDF announcement document in read-only mode, place the pointer at the beginning of the PDF announcement document, and continuously read the file content in the PDF announcement document in blocks, with the buffer size as the limit, until all the binary data of the PDF announcement document is read in and a file object is successfully generated. If an error such as data corruption or memory overflow occurs, the program will be terminated immediately.

[0071] (2) Whether the PDF announcement document can be parsed correctly. The main process is: according to the current international standard of PDF, the file object created in step (1) is analyzed and separated into four parts: file header, file content, cross-reference table and file footer. When parsing, the file footer is first parsed to obtain the cross-reference table and root object number, and then the document is parsed layer by layer through these two to construct the document page. In particular, the file content contains several page objects, which store various resources such as actual text content, image media information, font style, etc. Among them, once the image media information exceeds a certain threshold, the PDF announcement document is considered to be a scanned copy, that is, a non-standardized PDF announcement document, and the parsing program should be terminated immediately.

[0072] In short, if steps (1) and (2) can be successfully implemented, the input PDF announcement document is considered to be a standardized PDF document.

[0073] In this embodiment, the standardized PDF document means that the document should comply with the PDF international standard (ISO 32000-1 / 2) issued by Adobe and ISO organization, and various elements and structures in the document can be scanned, copied and parsed.

[0074] Non-standardized PDF documents refer specifically to PDF documents with a number of images, which may be scanned copies of a listed company's announcement. Such documents require optical recognition processing, but such documents are generally rare in the securities market.

[0075] Step S103: when the PDF announcement document is the standardized PDF document, convert the PDF announcement document from PDF format to a temporary Word announcement document in docx format;

[0076] It should be noted that the Word announcement document in this embodiment is only an intermediate document, which is generated in the initial stage of the algorithm and destroyed after the algorithm is executed.

[0077] It should be noted that the conversion of the Word announcement document itself does not meet the aforementioned "formatting" requirements. The purpose of setting this step is mainly to try to use the built-in table object of the Word announcement document to better restore the table content and format in the PDF announcement. In addition, the Word announcement document can be used as a supplementary information to assist in effectively distinguishing the text line objects and table objects in the original PDF announcement document in the preset coordinate algorithm. This is because the text line objects and table objects in the PDF announcement document after conversion are basically isolated from each other.

[0078] The Word announcement document includes paragraphs and tables.

[0079] Step S104: segment the announcement domain information of the PDF announcement document and the Word announcement document using a preset coordinate algorithm, and obtain the formatted announcement plain text corresponding to the PDF announcement document after segmentation.

[0080] The plain text of the announcement in this embodiment is also the plain text that meets the requirements of natural language processing technology.

[0081] The announcement domain information includes at least: announcement header information, announcement title, announcement description and announcement text.

[0082] When the PDF announcement document contains a table, the announcement domain information may also include: announcement table, see Figure 2 The following is a schematic diagram of the information in each announcement field in the PDF announcement document.

[0083] In this embodiment, it is assumed that the PDF announcement document is converted into a formatted TXT-formatted announcement plain text. The original PDF announcement document may contain body content and table content, and the converted TXT announcement plain text should restore these contents in a formatted manner. In the converted announcement TXT plain text, the body content and the table content should be separated (for example, the table text should be attached after the body text).

[0084] The “formatting” described in this embodiment is defined as follows:

[0085] ① Remove the blank lines in the announcement;

[0086] ② Remove the header, footer and page number of the announcement (only keep some important header information such as "announcement number", "securities code" and "securities abbreviation");

[0087] ③Normalize the announcement title: remove the line break in the title;

[0088] ④Normalize the announcement paragraphs: remove the line breaks in the middle of the paragraphs;

[0089] ⑤ Maintain the basic style of the table in the announcement, with each row of the table ending with a line break, and cells separated by tabs, while correcting the line breaks within the cells;

[0090] ⑥Specific punctuation marks, letter numbers, document numbers, numbers and special terms in the calibration announcement.

[0091] The plain text of the announcement obtained in this embodiment is the announcement text that meets the above-mentioned formatting definition.

[0092] In summary, the present invention discloses a method for processing an announcement document. Based on the format and content of an acquired PDF announcement document, the PDF announcement document is judged to be a standardized PDF document or a non-standardized PDF document. When the PDF announcement document is a standardized PDF document, the PDF announcement document is converted from PDF format to a temporary Word announcement document in docx format. The announcement domain information of the PDF announcement document and the Word announcement document are segmented using a preset coordinate algorithm, and the formatted announcement plain text corresponding to the PDF announcement document is obtained after the segmentation is completed. The announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text. The temporary Word announcement document obtained by conversion in the present invention is an intermediate document, not the final formatted announcement document. By using the text line object and table object in the Word announcement document, the text line object and table object in the PDF announcement document can be effectively distinguished when the preset coordinate algorithm is used, so that the text line object and the table object after the PDF announcement document conversion are distinguished and isolated from each other, and the data of the non-complex table in the announcement document is formatted and restored to the plain text form, so that the converted announcement plain text meets the needs of natural language processing technology.

[0093] To further optimize the above embodiment, step S103 may specifically include:

[0094] (1) When the PDF announcement document is the standardized PDF document, the open source module pdf2docx is used to extract the non-table content in the PDF announcement document to obtain a text line object, and to extract the table content in the PDF announcement document to obtain a table object;

[0095] (2) Obtain the Word document based on the text row object and the table object.

[0096] In step S104 of the above embodiment, the core idea of ​​using the preset coordinate algorithm to obtain the plain text of the announcement is: scan and analyze the elements and structures in the PDF announcement document and the Word announcement document, and obtain the horizontal and vertical coordinate values ​​of the text block (with the upper left corner of the page as the origin). The preset coordinate algorithm is detailed in Figure 3The core idea of ​​the preset coordinate algorithm is shown in the figure. The announcement is formatted and restored based on the text content and coordinate values ​​of several text blocks, supplemented by a small number of rules.

[0097] Therefore, step S104 may specifically include:

[0098] (1) Scan the PDF announcement document and extract the text block elements (LTTextBox) of the PDF announcement document page by page and line by line. The text block elements include: text characters, upper left corner coordinate values ​​and lower right corner coordinate values ​​of the text block rectangle.

[0099] (2) The text block elements extracted page by page and line by line are stored in a pre-established ordered dictionary in the order of extraction.

[0100] In this embodiment, the key of the ordered dictionary is the content of the text block in this row (stored in the form of a string "..."), and the key value is the coordinates corresponding to the text block in this row (stored in the form of a list [x0, y0, x1, y1]).

[0101] (3) Scan the text line objects and the table objects in the Word announcement document one by one to obtain the document elements in the Word announcement document, and obtain the formatted announcement plain text based on the document elements and the ordered dictionary.

[0102] The document elements include: a text string and a coordinate value list corresponding to the text line object, and a string in a cell corresponding to the table object.

[0103] The process of obtaining the formatted plain text of the announcement based on the document elements and the ordered dictionary in the above embodiment specifically includes:

[0104] ① Scan the Word announcement document, obtain a line of text string from the body line object in order, and obtain a coordinate value list with the text string as the key from the ordered dictionary to form a tuple (T1, L1) consisting of the text string T1 and the coordinate value list L1.

[0105] The "sequence" in "sequence" in this embodiment means that the obtained text data content should be kept in the order of front and back in the PDF announcement document *A, and this order is in units of lines in the page.

[0106] ② Match the two-tuple in a preset rule library, and format the two-tuple using the matched target rule to obtain the initial announcement plain text.

[0107] The preset rule base *M2 includes two types of rules, namely coordinate-dependent rules *M2-C and coordinate-independent rules *M2-NC. The coordinate-independent rules *M2-NC are simplified versions of the aforementioned non-statistical rules *M1-NS (the number of rules is much smaller than |*M1-NS|).

[0108] In this embodiment, when the target rule is not matched in the preset rule library, it is determined that the processing of the PDF announcement document has failed, and the processing flow ends.

[0109] ③ Scan the Word announcement document and obtain each table sub-object from the table object in sequence;

[0110] The process of obtaining each table sub-object includes:

[0111] Use the aforementioned open source module pdf2docx to convert the PDF announcement document into a Word announcement document *D in Docx format (as an intermediate result document). In the content storage of the Word announcement document, tables are naturally separated from general text (paragraphs). After receiving and successfully reading the incoming Word announcement document *D, the table object E and non-table object NE (or text object) can also be naturally obtained. Among them, the table object may contain several table sub-objects E i (Because a PDF announcement document may include multiple independent tables.) Table sub-object E i The content storage structure is: [row1, row2, ..., row m ], where row i The content storage structure is: [col1, col2, ..., col n ], row i Indicates the i-th row, col j represents the jth column.

[0112] ④ Traverse the table sub-object row by row and cell by cell, extract and correct the character string in each cell of the table sub-object, and obtain the target cell;

[0113] ⑤ Split the character strings in each target cell with a tab character to form a row of content of the table sub-object, and add it to the initial announcement plain text;

[0114] Repeat steps ③ to ⑤ until the last table sub-object in the Word announcement document is processed.

[0115] ⑥ When a row of content corresponding to all the table sub-objects in the table object is added to the initial announcement plain text, the final formatted announcement plain text is obtained.

[0116] To further optimize the above embodiment, ② the two-tuple is matched in a preset rule library, and the two-tuple is formatted using the matched target rule to obtain the initial announcement plain text, including:

[0117] 1) matching the coordinate value list with the coordinate-related rules in the preset rule library, and formatting the coordinate value list using the matched target coordinate-related rules to obtain a target coordinate value list;

[0118] In this embodiment, the preset rule library *M2 mainly includes two types of rules: coordinate-related rules *M2-C and coordinate-independent rules *M2-NC, where the coordinate-independent rules are a simplified version of the aforementioned non-statistical rules *M1-NS (the number of rules is far less than |*M1-NS|).

[0119] Among them, the coordinate value list is formatted, mainly normalizing the title and paragraph.

[0120] 2) matching the text string with the coordinate-independent rule in the preset rule library, and formatting the text string using the matched target coordinate-independent rule to obtain a target text string;

[0121] Among them, the text string is formatted, mainly to correct symbols and styles.

[0122] 3) Obtaining the initial public plain text based on all the target coordinate value lists and the target text string.

[0123] It should be noted that when the target coordinate-related rules and / or the target coordinate-independent rules are not matched, it is determined that the processing of the PDF announcement document has failed, and the processing flow is terminated at this time.

[0124] Normally, PDF announcement documents are standardized PDF documents, but it is inevitable that there are non-standard PDF documents in PDF announcement documents. Therefore, please refer to Figure 4 , a flowchart of another method for processing an announcement document disclosed in the present invention, Figure 1 Based on the embodiment shown, after step S102, the following may also be included:

[0125] Step S105: when the PDF announcement document is the non-standardized PDF document, extracting pictures from the PDF announcement document;

[0126] Step S106: using optical character recognition technology to recognize characters in the image line by line, and obtaining the initialization plain text corresponding to the PDF announcement document according to the recognition results line by line;

[0127] The full English name of optical character recognition technology is: Optical Character Recognition, abbreviated as: OCR.

[0128] In this embodiment, OCR technology only requires the recognition of characters in the scanned notice image line by line. The main process is: (1) Image preprocessing. Correct the imaging problems of the scanned notice image. The original image is often noisy, such as text distortion and image stains. Therefore, before entering the text recognition, it is generally necessary to perform grayscale, binarization, geometric transformation, distortion correction, blur removal, image enhancement, light correction, image smoothing, normalization and other preprocessing operations; (2) Text detection. Detect the location and range of the text and its layout, including layout analysis and text line detection. The purpose is to determine where the text is and how large the range of the text is; (3) Text recognition. Based on text detection, the text content is identified, mainly identifying what each text is. For a text image at a specific location, its features are calculated and input into the text classifier. The classifier classifies the input, that is, outputs which text this input feature should be identified as. OCR technology is relatively mature at present. The improvement of this patent is to use a neural network model to supervise and train a classification model in the massive corpus of securities announcement documents collected in advance. Its recognition is more accurate and more suitable for text recognition tasks of securities announcement document images.

[0129] This step uses the OCR module to complete the recognition operation. The OCR module can recognize the text in the image line by line and extract non-complex tables in the image.

[0130] It should be noted that the initialization plain text in this embodiment is also the final plain text obtained in the prior art, see Figure 5 What is shown is not the final plain text obtained by this application.

[0131] Step S107: segment the announcement domain information of the initialization plain text using a preset rule algorithm to obtain the announcement plain text.

[0132] It should be noted that when OCR fails to recognize characters in the image, it indicates that the processing of the PDF announcement document has failed. At this time, the processing flow is terminated.

[0133] In the present invention, the core idea of adopting a preset rule algorithm for the initialized plain text is as follows: The preset rule algorithm has a number of preset rules, and these rules represent whether a certain line of announcement text should be line-wrapped here. For example: as Figure 6 shown, line breaks should be made before and after sub-section headings in the form of "(1) XXX..."; line breaks should be made before and after specific financial terms; the number of characters in the last line of a paragraph usually does not exceed the average number of characters per line in the whole text, etc. Given the plain text of a PDF announcement document, the program will traverse the text line by line from beginning to end. If a rule match is found, "hard line breaks" will be made to it.

[0134] To further optimize the above embodiment, step S107 may specifically include:

[0135] (1) Sequentially extract a line of body text data T1, the text data T2 of the line immediately above the body text data, and the text data T3 of the line immediately below the body text data from the initialized plain text to obtain a text triple (T2, T1, T3);

[0136] (2) Match the text triple with the non-statistical rules in the preset rule library, and format the body text data by using the obtained target non-statistical rules to obtain target body text data;

[0137] Formatting the body text data T1 includes: deleting the line break character at the end of the text of line T1 or adding a line break character at its end; calibrating specific punctuation marks, letter numbers, document numbers, serial numbers, and special terms in the announcement; correcting incorrect table body styles and line breaks within cells, etc.

[0138] (3) Repeat the above steps until all the body text data in the initialized plain text are formatted, completing the first round of scanning of the initialized plain text;

[0139] (4) Conduct a second round of scanning of the initialized plain text, calculate the statistical values and thresholds corresponding to the initialized plain text after the first round of scanning according to the statistical rules in the preset rule library, and write them into the statistical rules;

[0140] (5) Match the target body text data with the statistical rules, and format the target body text data by using the obtained target statistical rules;

[0141] (6) Repeat the above steps until all the target body text data obtained from the first round of scanning in the initialized plain text are formatted, obtaining the final announcement plain text.

[0142] In this embodiment, the preset rule base *M1 mainly includes two types of rules: statistical rules *M1-S and non-statistical rules *M1-NS, where non-statistical rules *M1-NS include: context rules, lexical rules, syntactic rules, and grammatical rules. Statistical rules are aimed at the global information of the announcement, while non-statistical rules are aimed at the local information of the announcement. These rules are mainly derived from the experience summary of industry insiders on the type, structure, and writing format of the announcement documents.

[0143] It should be noted that the processing flow shown in the present invention takes a single PDF announcement document as an example. When multiple PDF announcement documents need to be processed, they can be processed in a serial or parallel manner (for example, multi-threaded or multi-process). When performing serial or parallel processing, the processing steps of each single PDF announcement document remain unchanged.

[0144] To facilitate further understanding of the solution to be protected by the present invention, the characteristics and usage scenarios of the preset rule algorithm (indicated by "*V1") and the preset coordinate algorithm (indicated by "*V2") are described as follows:

[0145] The preset rule algorithm (*V1) is applicable to any announcement text that can be extracted, without additional information. The restoration accuracy of this algorithm is above 85%, and its accuracy improvement depends on the completeness and coverage of the rules. The code is not easy to maintain.

[0146] The preset coordinate algorithm (*V2) is suitable for announcement files in standard PDF format, with a restoration accuracy of more than 95%, and only relies on a small number of rules, and the algorithm is simple. However, it requires additional information such as text, horizontal and vertical coordinates of the absolute position of the line, and requires the announcement file to be standard and without complex page layout or style.

[0147] Therefore, if the given original announcement file (i.e., PDF announcement file) is in a standardized PDF format, the preset coordinate algorithm *V2 algorithm is used first to ensure formatting accuracy; if the original announcement file is in a picture or other format, OCR and other technologies can be used to first extract the non-standardized announcement text, and then the preset rule algorithm *V1 algorithm is used to restore it as much as possible and reduce the investment in subsequent manual review and adjustment.

[0148] The method for processing announcement documents disclosed in the present invention is in accordance with the characteristics of securities announcements and is highly accurate and efficient. It uses a computer program to extract texts with high cleaning and formatting degrees from a large number of securities market PDF announcement documents. These texts meet the following requirements:

[0149] ① Unnecessary content was deleted, and the text content and table content were distinguished, retained and isolated;

[0150] ② Text that should be in independent lines or paragraphs (e.g. titles, sections, paragraphs) should be in independent lines or paragraphs;

[0151] ③The data in the non-complex tables in the announcement were formatted and restored to plain text;

[0152] ④ Specific punctuation marks, letter numbers, document numbers, serial numbers and special terms in the announcement have been processed;

[0153] ⑤ The potential differences in the announcement document styles of the Shanghai Stock Exchange, Shenzhen Stock Exchange and the Stock Transfer System have been adapted.

[0154] For examples of application results of the technical solution of the present invention, please refer to the example schematic diagrams of the text content of a PDF securities announcement before and after formatted extraction shown in Figures 7(a) and 7(b), respectively, and the schematic diagrams of the table content of a PDF securities announcement before and after formatted extraction shown in Figures 8(a) and 8(b).

[0155] Corresponding to the above method embodiment, the present invention also discloses a system for processing announcement documents.

[0156] See also Fig. 9 , a schematic diagram of the structure of a system for processing announcement documents disclosed in an embodiment of the present invention, the system comprising:

[0157] The document acquisition unit 201 is used to acquire the PDF announcement document;

[0158] A determination unit 202, configured to determine whether the PDF announcement document is a standardized PDF document or a non-standardized PDF document based on the format and content of the PDF announcement document;

[0159] In this embodiment, the standardized PDF document means that the document should comply with the PDF international standard (ISO 32000-1 / 2) issued by Adobe and ISO organization, and various elements and structures in the document can be scanned, copied and parsed.

[0160] Non-standardized PDF documents refer specifically to PDF documents with a number of images, which may be scanned copies of a listed company's announcement. Such documents require optical recognition processing, but such documents are generally rare in the securities market.

[0161] The conversion unit 203 is used for converting the PDF announcement document from PDF format to a temporary Word announcement document in docx format when the PDF announcement document is the standardized PDF document, wherein the Word announcement document includes a text line object and a table object;

[0162] It should be noted that the Word announcement document in this embodiment is only an intermediate document, which is generated in the initial stage of the algorithm and destroyed after the algorithm is executed.

[0163] It should be noted that the conversion of the Word announcement document itself does not meet the aforementioned "formatting" requirements. The purpose of setting this step is mainly to try to use the built-in table object of the Word announcement document to better restore the table content and format in the PDF announcement. In addition, the Word announcement document can be used as a supplementary information to assist in effectively distinguishing the text line objects and table objects in the original PDF announcement document in the preset coordinate algorithm. This is because the text line objects and table objects in the PDF announcement document after conversion are basically isolated from each other.

[0164] The first segmentation unit 204 is used to segment the announcement domain information of the PDF announcement document and the Word announcement document using a preset coordinate algorithm, and obtain the formatted announcement plain text corresponding to the PDF announcement document after the segmentation is completed, wherein the announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text.

[0165] The plain text of the announcement in this embodiment is also the plain text that meets the requirements of natural language processing technology.

[0166] The announcement domain information includes at least: announcement header information, announcement title, announcement description and announcement text.

[0167] When the PDF announcement document contains a table, the announcement domain information may also include: announcement table, see Figure 2 The following is a schematic diagram of the information in each announcement field in the PDF announcement document.

[0168] In summary, the present invention discloses a processing system for announcement documents. Based on the format and content of the acquired PDF announcement document, the PDF announcement document is judged to be a standardized PDF document or a non-standardized PDF document. When the PDF announcement document is a standardized PDF document, the PDF announcement document is converted from PDF format to a temporary Word announcement document in docx format. The announcement domain information of the PDF announcement document and the Word announcement document is segmented using a preset coordinate algorithm, and the formatted announcement plain text corresponding to the PDF announcement document is obtained after the segmentation is completed. The announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text. The temporary Word announcement document obtained by conversion in the present invention is an intermediate document, not the final formatted announcement document. By using the text line object and table object in the Word announcement document, the text line object and table object in the PDF announcement document can be effectively distinguished when the preset coordinate algorithm is used, so that the text line object and the table object after the PDF announcement document conversion are distinguished and isolated from each other, and the data of the non-complex table in the announcement document is formatted and restored to the plain text form, so that the converted announcement plain text meets the needs of natural language processing technology.

[0169] See also Fig.10 , a schematic diagram of the structure of another system for processing announcement documents disclosed in an embodiment of the present invention, Fig. 9 Based on the embodiment shown, the following may also be included:

[0170] An extraction unit 205, configured to extract a picture from the PDF announcement document when the PDF announcement document is the non-standardized PDF document;

[0171] The recognition unit 206 is used to recognize the characters in the image line by line using optical character recognition technology, and obtain the initialization plain text corresponding to the PDF announcement document according to the recognition results line by line;

[0172] The second segmentation unit 207 is used to segment the announcement domain information of the initialization plain text by using a preset rule algorithm to obtain the announcement plain text.

[0173] It should be noted that for the specific working principles of each component in the system embodiment, please refer to the corresponding part of the method embodiment, which will not be repeated here.

[0174] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0175] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0176] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing an announcement document, characterized in that: include: Get the PDF announcement document; Based on the format and content of the PDF announcement document, determining whether the PDF announcement document is a standardized PDF document or a non-standardized PDF document; When the PDF announcement document is the standardized PDF document, converting the PDF announcement document from PDF format to a temporary Word announcement document in docx format, wherein the Word announcement document includes a text line object and a table object; For the PDF announcement document and the Word announcement document, a preset coordinate algorithm is used to segment the announcement domain information, and after the segmentation is completed, a formatted announcement plain text corresponding to the PDF announcement document is obtained, wherein the announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement text; Wherein, for the PDF announcement document and the Word announcement document, a preset coordinate algorithm is used to segment the announcement domain information, and after the segmentation is completed, a formatted announcement plain text corresponding to the PDF announcement document is obtained, including: Scan the PDF announcement document, and extract text block elements of the PDF announcement document page by page and line by line, wherein the text block elements include: text characters, upper left corner coordinate values ​​and lower right corner coordinate values ​​of a text block rectangle; The text block elements extracted page by page and line by line are stored in a pre-established ordered dictionary according to the extraction order; Scan the text line objects and the table objects in the Word announcement document one by one to obtain document elements in the Word announcement document, and obtain the formatted announcement plain text based on the document elements and the ordered dictionary, wherein the document elements include: text strings and coordinate value lists corresponding to the text line objects, and strings in cells corresponding to the table objects; The method comprises scanning the text line objects and the table objects in the Word announcement document one by one to obtain the document elements in the Word announcement document, and obtaining the formatted announcement plain text based on the document elements and the ordered dictionary, including: Scan the Word bulletin document, sequentially obtain a line of text string from the text line object, and obtain a coordinate value list with the text string as a key from the ordered dictionary to form a tuple consisting of the text string and the coordinate value list; Matching the two-tuple in a preset rule base, and formatting the two-tuple using the matched target rule to obtain an initial announcement plain text, wherein the preset rule base contains two types of rules, namely, coordinate-related rules and coordinate-independent rules; Scan the Word announcement document and obtain each table sub-object from the table object in sequence; Traversing the table sub-object row by row and cell by cell, extracting and correcting the character string in each cell of the table sub-object to obtain the target cell; Split the character string in each target cell with a tab character, together forming a row of content of the table sub-object, and add it to the initial announcement plain text; When a row of content corresponding to all the table sub-objects in the table object is added to the initial announcement plain text, the final formatted announcement plain text is obtained.

2. The processing method according to claim 1, characterized in that: When the PDF announcement document is the standardized PDF document, converting the PDF announcement document from PDF format to a temporary Word document in docx format includes: When the PDF announcement document is the standardized PDF document, the open source module pdf2docx is used to extract the non-table content in the PDF announcement document to obtain the text line object, and the table content in the PDF announcement document is extracted to obtain the table object; The Word document is obtained based on the text row object and the table object.

3. The processing method according to claim 1, characterized in that: The two-tuple is matched in a preset rule library, and the two-tuple is formatted using the matched target rule to obtain an initial announcement plain text, including: Matching the coordinate value list with the coordinate-related rules in the preset rule library, and formatting the coordinate value list using the matched target coordinate-related rules to obtain a target coordinate value list; Matching the text string with the coordinate-independent rule in the preset rule library, and formatting the text string using the matched target coordinate-independent rule to obtain a target text string; The initial public plain text is obtained based on all the target coordinate value lists and the target text string.

4. The processing method according to claim 1, characterized in that: Also includes: When the PDF announcement document is the non-standardized PDF document, extracting a picture from the PDF announcement document; Using optical character recognition technology to recognize characters in the image line by line, and obtaining the initialization plain text corresponding to the PDF announcement document according to the line by line recognition results; The announcement domain information is segmented on the initialization plain text using a preset rule algorithm to obtain the announcement plain text.

5. The processing method according to claim 4, characterized in that: The initialization plain text is segmented into announcement domain information using a preset rule algorithm to obtain the announcement plain text, including: Extracting a line of main text data, a line of text data immediately above the main text data, and a line of text data immediately below the main text data from the initialization plain text in order to obtain a text triple; Matching the text triples with non-statistical rules in a preset rule base, and formatting the body text data using the matched target non-statistical rules to obtain target body text data; This process is repeated until all the text data in the initialization plain text are formatted, and the first round of scanning of the initialization plain text is completed; Perform a second round of scanning on the initialization plain text, calculate the statistical value and threshold value corresponding to the initialization plain text after the first round of scanning according to the statistical rules in the preset rule base, and write them into the statistical rules; Matching the target body text data with the statistical rules, and formatting the target body text data using the matched target statistical rules; This process is repeated until all target text data obtained from the first round of scanning in the initialization plain text are formatted to obtain the final announcement plain text.

6. A system for processing announcement documents, characterized in that: include: A document acquisition unit, used to acquire PDF announcement documents; A judging unit, configured to judge whether the PDF announcement document is a standardized PDF document or a non-standardized PDF document based on the format and content of the PDF announcement document; A conversion unit, used for converting the PDF announcement document from PDF format to a temporary Word announcement document in docx format when the PDF announcement document is the standardized PDF document, wherein the Word announcement document includes a text line object and a table object; A first segmentation unit is used to segment the announcement domain information of the PDF announcement document and the Word announcement document by using a preset coordinate algorithm, and obtain the formatted announcement plain text corresponding to the PDF announcement document after segmentation, wherein the announcement domain information at least includes: announcement header information, announcement title, announcement description and announcement body; Wherein, the first segmentation unit is specifically used for: Scan the PDF announcement document, and extract text block elements of the PDF announcement document page by page and line by line, wherein the text block elements include: text characters, upper left corner coordinate values ​​and lower right corner coordinate values ​​of a text block rectangle; The text block elements extracted page by page and line by line are stored in a pre-established ordered dictionary according to the extraction order; Scan the text line objects and the table objects in the Word announcement document one by one to obtain document elements in the Word announcement document, and obtain the formatted announcement plain text based on the document elements and the ordered dictionary, wherein the document elements include: text strings and coordinate value lists corresponding to the text line objects, and strings in cells corresponding to the table objects; The method comprises scanning the text line objects and the table objects in the Word announcement document one by one to obtain the document elements in the Word announcement document, and obtaining the formatted announcement plain text based on the document elements and the ordered dictionary, including: Scan the Word bulletin document, sequentially obtain a line of text string from the text line object, and obtain a coordinate value list with the text string as a key from the ordered dictionary to form a tuple consisting of the text string and the coordinate value list; Matching the two-tuple in a preset rule base, and formatting the two-tuple using the matched target rule to obtain an initial announcement plain text, wherein the preset rule base contains two types of rules, namely, coordinate-related rules and coordinate-independent rules; Scan the Word announcement document and obtain each table sub-object from the table object in sequence; Traversing the table sub-object row by row and cell by cell, extracting and correcting the character string in each cell of the table sub-object to obtain the target cell; Split the character string in each target cell with a tab character, together forming a row of content of the table sub-object, and add it to the initial announcement plain text; When a row of content corresponding to all the table sub-objects in the table object is added to the initial announcement plain text, the final formatted announcement plain text is obtained.

7. The processing system according to claim 6, characterized in that Also includes: An extraction unit, configured to extract a picture from the PDF announcement document when the PDF announcement document is the non-standardized PDF document; A recognition unit, used to recognize characters in the image line by line using optical character recognition technology, and obtain the initialization plain text corresponding to the PDF announcement document according to the line by line recognition results; The second segmentation unit is used to segment the announcement domain information of the initialization plain text by using a preset rule algorithm to obtain the announcement plain text.

Images