Paragraph detection method and device

Abstract

The invention provides a paragraph detection method and device, and relates to the technical field of computer vision, and the paragraph detection method comprises the steps: carrying out the text detection of a to-be-recognized document picture, obtaining at least one initial text detection box, and carrying out the paragraph detection of the document picture, and obtaining at least one initial paragraph detection box; according to the first position information of each initial text detection box and the second position information of each initial paragraph detection box, determining at least one target text detection box associated with each initial paragraph detection box from each initial text detection box; and for each initial paragraph detection box, determining a target paragraph detection box according to each target text detection box associated with the initial paragraph detection box, and labeling a target paragraph in the document picture according to the target paragraph detection box. Paragraphs and document structures can be effectively detected, so that the paragraphs and the document structures meet the requirements of downstream tasks.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a paragraph detection method and device. Background Art

[0002] With the rapid development of digital technology, the electronicization of paper documents has become a trend. However, due to the complexity and diversity of paper document formats, it is necessary to manually extract the document content and enter it into the electronic system in a structured manner. Manual entry will inevitably lead to missing items and wrong items, and when the input volume is large, a large number of human resources need to be allocated to perform highly repetitive work, which is time-consuming, labor-intensive, and inefficient. Therefore, an effective solution is urgently needed to solve the above problems. Summary of the invention

[0003] In view of the problems existing in the prior art, an embodiment of the present invention provides a paragraph detection method and device.

[0004] The present invention provides a paragraph detection method, comprising:

[0005] Performing text detection on the document image to be recognized to obtain at least one initial text detection frame, and performing paragraph detection on the document image to obtain at least one initial paragraph detection frame;

[0006] Determining at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame;

[0007] For each of the initial paragraph detection frames, a target paragraph detection frame is determined according to each target text detection frame associated with the initial paragraph detection frame, and the target paragraph in the document image is annotated according to the target paragraph detection frame.

[0008] According to a paragraph detection method provided by the present invention, the paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame, including:

[0009] According to the preset paragraph features corresponding to each paragraph type, paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame.

[0010] According to a paragraph detection method provided by the present invention, paragraph detection is performed on the document image according to preset paragraph features corresponding to each paragraph type to obtain at least one initial paragraph detection frame, including:

[0011] For each paragraph type, a preset paragraph feature corresponding to the paragraph type is used to identify an initial paragraph in the document image that belongs to the paragraph type, and annotate the initial paragraph with an initial paragraph detection frame;

[0012] The paragraph types include a regular paragraph type and a list paragraph type;

[0013] The regular paragraphs include a title type and a body paragraph type, and the body paragraph type includes a Chinese paragraph type and / or an English paragraph type.

[0014] According to a paragraph detection method provided by the present invention, the preset paragraph features corresponding to the list paragraph type include: the paragraph contains a plurality of continuous text lines, and the left side of the paragraph contains a plurality of bullet points and / or a plurality of continuous numbers;

[0015] The preset paragraph features corresponding to the title type include: the number of text lines in the paragraph is less than the preset number of title lines; the display position of the paragraph is the middle of the document, or the display position is the previous line of the body paragraph, the left side of the paragraph is aligned to the top, and the left side of the paragraph contains a serial number;

[0016] The preset paragraph features corresponding to the Chinese paragraph type include: the first line of the paragraph is indented and the paragraph is aligned to the left and right;

[0017] The preset paragraph features corresponding to the English paragraph type include: the first line of the paragraph is aligned or indented on the left side.

[0018] According to a paragraph detection method provided by the present invention, determining a target paragraph detection frame according to each target text detection frame associated with the initial paragraph detection frame includes:

[0019] Combining the target text detection frames associated with the initial paragraph detection frame to obtain a combined frame;

[0020] The minimum circumscribed rectangle of the combination box is determined as the target paragraph detection box.

[0021] According to a paragraph detection method provided by the present invention, the text detection is performed on the document image to be identified to obtain at least one initial text detection frame, and the paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame, including:

[0022] Input the document image to be recognized into the pre-trained text detection model for text detection to obtain at least one initial text detection frame;

[0023] The document image is input into a pre-trained paragraph detection model for paragraph detection to obtain at least one initial paragraph detection frame.

[0024] According to a paragraph detection method provided by the present invention, the text detection model includes a first feature extraction layer, a prediction layer, a processing layer and a conversion layer;

[0025] Accordingly, the document image to be identified is input into a pre-trained text detection model for text detection to obtain at least one initial text detection frame, including:

[0026] Inputting the document image to be identified into the first feature extraction layer for feature extraction to obtain a target feature map;

[0027] Inputting the target feature map into the prediction layer for text prediction to obtain a probability map and a threshold map;

[0028] Inputting the probability map and the threshold map into a processing layer for differentiable binarization fusion processing to obtain an approximate binarization map;

[0029] The approximate binarization image is input into the conversion layer for text box conversion to obtain at least one initial text detection box.

[0030] According to a paragraph detection method provided by the present invention, the text detection model includes an input layer, a second feature extraction layer, a feature fusion layer and an output layer;

[0031] Accordingly, the step of inputting the document image into a pre-trained paragraph detection model for paragraph detection to obtain at least one initial paragraph detection frame includes:

[0032] Inputting the document image into the input layer for normalization processing to obtain the normalized document image;

[0033] Inputting the normalized document image into the second feature extraction layer to extract features and obtain image features;

[0034] Inputting the image features into a feature fusion layer to perform multi-scale feature fusion to obtain fused features;

[0035] The fused features are input into the output layer for paragraph detection to obtain at least one initial paragraph detection frame.

[0036] According to a paragraph detection method provided by the present invention, determining at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame, comprising:

[0037] For each of the initial paragraph detection frames, compare the second position information of the initial paragraph detection frame with the first position information of each of the initial text detection frames, and determine target first position information in each of the first position information that overlaps with the second position information;

[0038] The initial text detection frame corresponding to the target first position information is determined as a target text detection frame associated with the initial paragraph detection frame.

[0039] The present invention also provides a paragraph detection device, comprising:

[0040] A detection module is configured to perform text detection on the document image to be recognized to obtain at least one initial text detection frame, and perform paragraph detection on the document image to obtain at least one initial paragraph detection frame;

[0041] a determination module configured to determine at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame;

[0042] The annotation module is configured to determine a target paragraph detection frame for each of the initial paragraph detection frames by associating the target text detection frames with the initial paragraph detection frame, and annotate the target paragraph in the document image according to the target paragraph detection frame.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the paragraph detection methods described above is implemented.

[0044] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the paragraph detection method described above is implemented.

[0045] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the paragraph detection method described above is implemented.

[0046] The paragraph detection method and device provided by the present invention perform text detection on the document image to be identified to obtain at least one initial text detection frame, and perform paragraph detection on the document image to obtain at least one initial paragraph detection frame; according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame, at least one target text detection frame associated with each initial paragraph detection frame is determined from each initial text detection frame; for each initial paragraph detection frame, a target paragraph detection frame is determined according to each target text detection frame associated with the initial paragraph detection frame, and the target paragraph in the document image is marked according to the target paragraph detection frame. Through the two-way combination of text frame and paragraph frame, paragraphs in the document image can be detected quickly and accurately, thereby realizing the detection of document structure; it has high generalization and can cope with the document structure detection requirements of various scenarios; it has high speed and can quickly realize document structure positioning. Customize document structure attributes in various complex and chaotic scenarios to meet the needs of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 It is one of the flow charts of the paragraph detection method provided by the present invention;

[0049] Figure 2 is a schematic diagram of a document image provided by the present invention;

[0050] Figure 3 This is the second flow chart of the paragraph detection method provided by the present invention;

[0051] Figure 4 It is a schematic diagram of the effect of the paragraph detection method provided by the present invention;

[0052] Figure 5 is a structural schematic diagram of a paragraph detection device provided by the present invention;

[0053] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In order to facilitate a clearer understanding of the embodiments of the present invention, some relevant background knowledge is first introduced as follows.

[0056] With the rapid development of digital technology, the electronicization of paper documents has become a trend. However, due to the complexity and diversity of paper document formats, it is necessary to manually extract the document content and enter it into the electronic system in a structured manner. Manual entry will inevitably lead to missing items and wrong items, and when the amount of entry is large, a large number of human resources need to be allocated to perform highly repetitive work, which is time-consuming, labor-intensive, and inefficient.

[0057] In order to solve the above problems, a multimodal document understanding pre-training model is introduced in the prior art to identify the document structure of document images, for example, using LayoutLM to identify paragraphs in document images.

[0058] However, these solutions have the following limitations: the dataset is difficult to customize, that is, the definitions of different paragraphs are different. For example, formal documents use carriage returns to mark paragraphs, but paragraphs cannot be identified based on carriage returns in tables or logos. For example, there are differences between Chinese and English paragraphs. The application scenarios are limited. For example, LayoutLM only targets the title, author, and paragraphs in the paper. For PPT-type document images, there are not so many text types such as title, author, and paragraph, resulting in inadaptability. Training and deployment require a lot of resources.

[0059] Traditional algorithms can cluster paragraph structures based on the location and length of the text, but the limitation of this solution is its poor generalization.

[0060] To this end, the present invention provides a paragraph detection method and device, which performs text detection on a document image to be identified to obtain at least one initial text detection frame, and performs paragraph detection on the document image to obtain at least one initial paragraph detection frame; according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame, at least one target text detection frame associated with each initial paragraph detection frame is determined from each initial text detection frame; for each initial paragraph detection frame, a target paragraph detection frame is determined according to each target text detection frame associated with the initial paragraph detection frame, and the target paragraph in the document image is marked according to the target paragraph detection frame. Through the two-way combination of text boxes and paragraph boxes, paragraphs in document images can be detected quickly and accurately, thereby realizing the detection of document structure; it has high generalization and can cope with the document structure detection requirements of various scenarios; it has high speed and can quickly realize document structure positioning. Customize document structure attributes in various complex and chaotic scenarios to meet the needs of downstream tasks.

[0061] Combine the following Figure 1-Figure 6 The paragraph detection method and device of the present invention are described.

[0062] Figure 1 is one of the flow charts of the paragraph detection method provided by the present invention, see Figure 1 As shown, it includes steps 101 to 103, wherein:

[0063] Step 101: performing text detection on a document image to be identified to obtain at least one initial text detection frame, and performing paragraph detection on the document image to obtain at least one initial paragraph detection frame.

[0064] First of all, it should be noted that the execution subject of the present invention can be any electronic device for detecting segments, for example, it can be any one of a smart phone, a smart watch, a desktop computer, a laptop computer, etc.

[0065] Specifically, a document image refers to a document form that uses an image as a carrier to record a document, that is, an image that records a document. A document image to be identified refers to a document image that requires paragraph detection, that is, a document image that requires document structure detection. Text detection refers to the process of locating text areas in a document image. A detection box refers to a boundary box that marks text in a document image. A text detection box refers to a detection box that marks text lines in a document image. An initial text detection box refers to a text detection box that is initially obtained by performing text detection. Paragraph detection refers to the process of locating paragraphs in a document image. A paragraph detection box refers to a detection box that marks paragraphs in a document image. An initial text detection box refers to a paragraph detection box that is initially obtained by performing paragraph detection.

[0066] In practical applications, before performing text detection and paragraph detection on a document image to be identified, the document image to be identified must be obtained first. There are many methods for obtaining the document image to be identified, and the present invention does not limit this.

[0067] Exemplarily, a user uploads a document image to be identified through a document upload page, and accordingly, the execution subject obtains the document image to be identified.

[0068] Exemplarily, the execution subject receives a document image acquisition instruction or a paragraph detection instruction, and accordingly, the execution subject acquires the document image to be identified from the storage area pointed to by the document image acquisition instruction or the paragraph detection instruction.

[0069] On the basis of obtaining the document image, further, text detection is performed on the document image, each text line in the document image is identified, and each text line in the document image is marked with an initial text detection frame. Paragraph detection is performed on the document image, each paragraph in the document image is identified, and each paragraph in the document image is marked with an initial paragraph detection frame.

[0070] Step 102: Determine at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame.

[0071] Specifically, the position information refers to the position of the detection box in the document image, which can be in DOTA data format, i.e. (x1, y1, x2, y2, x3, y3, x4, y4), where (xi, yi) represents the coordinates of the corner points of the detection box, or (x, y, w, h), where (x, y) is the coordinate of the upper left corner of the detection box, w and h are the width and height of the detection box respectively, and the origin can be any point in the document image, such as the point in the upper left corner. The target text detection box refers to the initial text detection box after screening, matching, etc.

[0072] In practical applications, for each initial paragraph detection frame, according to the second position information of the initial paragraph detection frame and the first position information of each initial text detection frame, the initial text detection frames included in the initial paragraph detection frame are determined from all the initial text detection frames of the document image. If the initial text detection frames whose distance from the initial paragraph detection frame is less than the set distance are determined, these initial text detection frames are determined as target text detection frames associated with the initial paragraph detection frame. According to the above method, all initial paragraph detection frames are traversed to determine at least one target text detection frame associated with each initial paragraph detection frame.

[0073] Step 103: for each of the initial paragraph detection frames, determine a target paragraph detection frame according to each target text detection frame associated with the initial paragraph detection frame, and annotate the target paragraph in the document image according to the target paragraph detection frame.

[0074] Specifically, the target paragraph detection box refers to the detection box finally obtained for marking the paragraph.

[0075] In practical applications, after determining the target text detection frames associated with each initial paragraph detection frame, for each initial paragraph detection frame, all target text detection frames associated with the initial paragraph detection frame are spliced, combined, etc. to obtain the target paragraph frame and the position of the target paragraph frame; according to the position of the target paragraph frame, the target paragraph frame is placed in the document image to mark the target paragraph in the document image. According to the above process, all initial paragraph detection frames are traversed to determine the target paragraph detection frame corresponding to each initial paragraph detection frame, and then each target paragraph frame is used to mark the target paragraph in the document image.

[0076] It should be noted that the target paragraph detection frame can be first determined based on the target text detection frames associated with an initial paragraph detection frame, and the target paragraph detection frame can be used to mark the target paragraph in the document image. Then, the target paragraph detection frame can be determined based on the target text detection frames associated with the next initial paragraph detection frame, and the target paragraph detection frame can be used to mark the target paragraph in the document image, and so on, until all the target text detection frames associated with all the initial paragraph detection frames are traversed. It is also possible to determine the target paragraph detection frame corresponding to each initial paragraph detection frame based on the target text detection frames associated with each initial paragraph detection frame, and then use each target paragraph detection frame to mark the target paragraph in the document image. It can also be other, and the present invention does not impose any limitation on this.

[0077] The paragraph detection method provided by the present invention performs text detection on the document image to be identified to obtain at least one initial text detection frame, and performs paragraph detection on the document image to obtain at least one initial paragraph detection frame; according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame, at least one target text detection frame associated with each initial paragraph detection frame is determined from each initial text detection frame; for each initial paragraph detection frame, a target paragraph detection frame is determined according to each target text detection frame associated with the initial paragraph detection frame, and the target paragraph in the document image is marked according to the target paragraph detection frame. Through the two-way combination of text frame and paragraph frame, paragraphs in the document image can be detected quickly and accurately, thereby realizing the detection of document structure; it has high generalization and can cope with the document structure detection requirements of various scenarios; it has high speed and can quickly realize document structure positioning. Customize document structure attributes in various complex and chaotic scenarios to meet the needs of downstream tasks.

[0078] In one or more optional embodiments of the present invention, paragraph detection can be performed directly on the document image without identifying the paragraph type of each paragraph, thereby improving the efficiency of paragraph detection.

[0079] In one or more optional embodiments of the present invention, the paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame, and the specific implementation process may be as follows:

[0080] According to the preset paragraph features corresponding to each paragraph type, paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame.

[0081] Specifically, paragraph type refers to a type preset according to the text content of a paragraph, including a regular paragraph type and a list paragraph type. Among them, the list paragraph type refers to the type corresponding to a paragraph displayed in the form of a list, such as paragraphs belonging to the list paragraph type including paragraphs corresponding to the reference area (reference list), paragraphs corresponding to the table of contents area, and lists corresponding to implementation steps in the text, etc. Regular paragraph type refers to the type corresponding to paragraphs other than list paragraphs, such as titles, text, etc., which belong to regular paragraph types. Preset paragraph features refer to features pre-set for paragraphs belonging to a certain paragraph type, or features possessed by paragraphs of a certain paragraph type.

[0082] In practical applications, paragraph detection can be performed on document images according to the preset paragraph features corresponding to each paragraph type, and paragraphs and paragraph types in the document image can be identified, and the initial paragraph frames can be used to mark them, so as to obtain initial paragraph detection frames corresponding to the document image. In this way, not only can each paragraph in the document image be detected, but also the paragraph type to which each paragraph belongs can be identified, which is conducive to improving the accuracy and precision of paragraph recognition, and the document structure in the document image can be further identified according to the paragraph type.

[0083] It should be noted that different paragraph types can be marked with different initial paragraph detection frames. For example, paragraphs of the regular paragraph type are marked with an initial paragraph detection frame composed of solid lines, and paragraphs of the list paragraph type are marked with an initial paragraph detection frame composed of dotted lines. A label can also be marked on the initial paragraph detection frame, and the label is the paragraph type of the paragraph, that is, a certain paragraph is marked with an initial paragraph detection frame, and the paragraph type of the paragraph is marked on the initial paragraph detection frame in the form of a label. In addition, the paragraph type of the paragraph can also be directly marked next to the paragraph in the form of a label.

[0084] In one or more optional embodiments of the present invention, the step of performing paragraph detection on the document image according to preset paragraph features corresponding to each paragraph type to obtain at least one initial paragraph detection frame includes:

[0085] For each preset paragraph feature corresponding to the paragraph type, an initial paragraph belonging to the paragraph type in the document image is identified, and the initial paragraph is marked with an initial paragraph detection frame.

[0086] In practical applications, for each paragraph type, paragraph detection can be performed on the document image according to the preset paragraph features corresponding to the paragraph type, so as to identify the initial paragraph of the paragraph type in the document image, mark the initial paragraph of the paragraph type with the initial paragraph detection frame, and mark the paragraph type of the initial paragraph. Traverse each paragraph type, identify all initial paragraphs in the document image and mark them with the initial paragraph detection frame. In this way, the paragraph type of the initial paragraph is also identified while identifying the initial paragraph, which can improve the efficiency and accuracy of paragraph detection.

[0087] For example, according to the preset paragraph features corresponding to the regular paragraph type, the regular paragraphs in the document image are identified, that is, the initial paragraphs belonging to the regular paragraph type, and then the regular paragraphs are marked with the initial paragraph detection frame, and the paragraph type of the regular paragraphs is marked. Then, according to the preset paragraph features corresponding to the list paragraph type, the list paragraphs in the document image are identified, that is, the initial paragraphs belonging to the list paragraph type, and then the list paragraphs are marked with the initial paragraph detection frame, and the paragraph type of the list paragraph is marked.

[0088] It should be noted that paragraph detection can be performed first according to the preset paragraph features corresponding to the regular paragraph type, and then paragraph detection can be performed according to the preset paragraph features corresponding to the list paragraph type; paragraph detection can be performed first according to the preset paragraph features corresponding to the list paragraph type, and then paragraph detection can be performed according to the preset paragraph features corresponding to the regular paragraph type; and paragraph detection can also be performed according to the preset paragraph features corresponding to the regular paragraph type while performing paragraph detection according to the preset paragraph features corresponding to the list paragraph type. The present invention does not impose any limitation on this.

[0089] In one or more optional embodiments of the present invention, paragraph detection is performed on the document image according to preset paragraph features corresponding to each paragraph type to obtain at least one initial paragraph detection frame, which may be as follows: paragraph detection is performed on the document image to obtain each initial paragraph in the document image; for each preset paragraph feature corresponding to the paragraph type, an initial paragraph belonging to the paragraph type is identified from each initial paragraph, and the initial paragraph is marked by an initial paragraph detection frame.

[0090] In practical applications, the paragraph detection technology can be used to perform paragraph detection on the document image to obtain the initial paragraphs in the document image. At this time, the initial paragraphs are not divided into paragraph types. Then, for each paragraph type, the paragraph type of each initial paragraph can be identified according to the preset paragraph features corresponding to the paragraph type, so as to identify the initial paragraph belonging to the paragraph type in the document image, mark the initial paragraph belonging to the paragraph type with the initial paragraph detection frame, and mark the paragraph type of the initial paragraph. Traverse each paragraph type, identify all the initial paragraphs belonging to each paragraph type in the document image and mark them with the initial paragraph detection frame. In this way, the efficiency and accuracy of paragraph detection can be improved.

[0091] In one or more optional embodiments of the present invention, the regular paragraphs include a title type and a body paragraph type.

[0092] Specifically, the title type refers to the paragraph type corresponding to the title, such as the first-level title, second-level title and other paragraphs, which all belong to the title type. The body paragraph type refers to the paragraph type corresponding to the main content of the document.

[0093] In one or more optional embodiments of the present invention, the text paragraph type includes but is not limited to at least one of a Chinese paragraph type, an English paragraph type and other language paragraph types.

[0094] Specifically, the Chinese paragraph type refers to the text paragraph type written in Chinese. The English paragraph type refers to the text paragraph type written in English. The other language paragraph type refers to the text paragraph type written in languages ​​other than Chinese and English.

[0095] In actual applications, since Chinese and English are relatively common, in order to reduce the number of preset paragraph types set for different paragraph types, preferably, the body paragraph type includes a Chinese paragraph type and / or an English paragraph type.

[0096] For some document images, the paragraphs belonging to the body paragraph type may all be of the Chinese paragraph type, such as a paper written in Chinese. For other document images, the paragraphs belonging to the body paragraph type may all be of the English paragraph type, such as images of English journals. For still other document images, the paragraphs belonging to the body paragraph type may be partially of the English paragraph type and partially of the Chinese paragraph type, such as a document image of a Chinese translation of a famous English work.

[0097] In one or more optional embodiments of the present invention, the preset paragraph features corresponding to the list paragraph type include: the paragraph contains multiple consecutive text lines, and the left side of the paragraph contains multiple bullet points and / or multiple consecutive numbers.

[0098] Specifically, the number includes at least one of letters, Arabic numerals, Chinese numerals and Roman numerals. The bullet symbol can be an asterisk "*", a comma "", a horizontal line "-" or an arrow ">" and the like.

[0099] In practical applications, see Figure 2 The schematic diagram of the document image provided by the present invention is shown as follows: From a visual point of view, a list paragraph generally appears as a plurality of continuous text lines, or appears as a continuous text line with similar structure, with symbols on the left, such as "*", ",", "-" or ">", etc., or the list paragraph contains serial letters, numbers and Chinese numbers on the left, such as Figure 2 Therefore, the preset paragraph feature corresponding to the list paragraph type can be set as a paragraph containing multiple continuous text lines, and the left side of the paragraph contains multiple bullet points and / or multiple continuous numbers. In this way, the accuracy of identifying list paragraphs can be improved.

[0100] In one or more optional embodiments of the present invention, the preset paragraph features corresponding to the title type include: the number of text lines in the paragraph is less than the preset number of title lines; the display position of the paragraph is the middle of the document, or the display position is the previous line of the body paragraph, the left side of the paragraph is aligned to the top of the page, and the left side of the paragraph contains a serial number.

[0101] Specifically, the preset number of title lines is the maximum number of text lines that a title can occupy, such as 1, 2, 3, etc. Top grid means that no blank space is left when editing text.

[0102] In practical applications, see Figure 2 , because visually, the title is usually a single line, appearing in the middle of the top of the document, such as the overall title of the document, or appearing in the previous line of each paragraph and at the top of the grid, usually with a serial number, such as a chapter title. Figure 2 Therefore, the preset paragraph feature corresponding to the title type can be set as follows: the number of text lines in the paragraph is less than the preset number of title lines; the display position of the paragraph is the middle of the document, or the display position is the previous line of the body paragraph, the left side of the paragraph is aligned, and the left side of the paragraph contains a serial number. In this way, the accuracy of title recognition can be improved.

[0103] In one or more optional embodiments of the present invention, the preset paragraph features corresponding to the Chinese paragraph type include: the first line of the paragraph is indented and the paragraph is aligned to the left and right.

[0104] Specifically, the first line indent means indenting the first line of a paragraph a certain distance from left to right, while all lines other than the first line remain unchanged, making it easier to read and distinguish the overall structure of the article.

[0105] In practical applications, see Figure 2, because from a visual point of view, Chinese paragraphs usually have the first line indented, and the other text lines are aligned to the left and right, such as Figure 2 Therefore, the preset paragraph features corresponding to the Chinese paragraph type can be set as the first line of the paragraph is indented and the paragraph is aligned to the left and right. In this way, the accuracy of identifying Chinese paragraphs can be improved.

[0106] In one or more optional embodiments of the present invention, the preset paragraph features corresponding to the English paragraph type include: the first line of the paragraph is aligned or indented on the left side.

[0107] In practical applications, see Figure 2 , because from a visual point of view, the first line of an English paragraph is usually left-aligned or indented, such as Figure 2 Therefore, the preset paragraph feature corresponding to the English paragraph type can be set to the left top grid or indent of the first line of the paragraph. In this way, the accuracy of identifying English paragraphs can be improved.

[0108] In addition, from a visual point of view, the last line of an English paragraph is generally not aligned to the top of the page, so the preset paragraph feature corresponding to the English paragraph type may also include: the last line of the paragraph is not aligned to the top of the page.

[0109] In one or more optional embodiments of the present invention, the target paragraph detection frame is determined according to each target text detection frame associated with the initial paragraph detection frame, and the specific implementation process may be as follows:

[0110] Combining the target text detection frames associated with the initial paragraph detection frame to obtain a combined frame;

[0111] The minimum circumscribed rectangle of the combination box is determined as the target paragraph detection box.

[0112] Specifically, the minimum bounding rectangle (MBR), also translated as the minimum bounding rectangle, the minimum containing rectangle or the minimum enclosing rectangle, refers to a rectangle with the smallest area that can completely cover a given shape. The combination box refers to a combination of target text detection boxes without changing their positions.

[0113] In practical applications, the target text detection frames associated with the initial paragraph detection frame are combined to obtain a combination frame, and then the minimum bounding rectangle of the combination frame is determined as the target paragraph detection frame. Since text detection technology is already very mature, the obtained text detection frame (initial text detection frame or target text detection frame) used to annotate text lines is very accurate. The target text detection frame associated with the initial paragraph detection frame is combined to obtain a combination frame that can annotate all the contents of the current paragraph, and then the minimum bounding rectangle of the combination frame is used as the target paragraph detection frame, so that the target paragraph detection frame can annotate the paragraph more accurately without covering the contents outside the paragraph or failing to fully contain the contents of the paragraph.

[0114] In one or more optional embodiments of the present invention, text detection may be performed on the document image to be recognized according to a set text detection strategy to obtain at least one initial text detection frame.

[0115] In one or more optional embodiments of the present invention, a text detection model may be used to perform text detection on a document image to be identified. That is, text detection is performed on the document image to be identified to obtain at least one initial text detection frame. The specific implementation process may be as follows:

[0116] The document image to be identified is input into a pre-trained text detection model for text detection to obtain at least one initial text detection frame.

[0117] Specifically, a text detection model refers to a trained model used to identify text lines in document images. Text detection algorithms include but are not limited to the YOLO (You Only Live Once) series of algorithms, the Single Shot MultiBox Detector (SSD) algorithm, the R-CNN (Region-Convolutional Neural Networks) series of algorithms, and the VIT (Vision Transformer) algorithm.

[0118] In practical applications, in order to improve the efficiency of text detection, a pre-trained text detection model can be used to perform text detection on the document image to be recognized, thereby obtaining each initial text detection frame corresponding to the document image. In this way, the efficiency of obtaining the initial text detection frame can be improved.

[0119] Optionally, the text detection model includes a first feature extraction layer, a prediction layer, a processing layer and a conversion layer; accordingly, the document image to be identified is input into a pre-trained text detection model for text detection to obtain at least one initial text detection frame, and the specific implementation process can be as follows:

[0120] Inputting the document image to be identified into the first feature extraction layer for feature extraction to obtain a target feature map;

[0121] Inputting the target feature map into the prediction layer for text prediction to obtain a probability map and a threshold map;

[0122] Inputting the probability map and the threshold map into a processing layer for differentiable binarization fusion processing to obtain an approximate binarization map;

[0123] The approximate binarization image is input into the conversion layer for text box conversion to obtain at least one initial text detection box.

[0124] Specifically, the first feature extraction layer refers to the unit used for feature extraction in the text detection model. The prediction layer refers to the unit that performs prediction in the text detection model. The processing layer refers to the unit that performs differentiable binarization fusion processing in the text detection model. The conversion layer refers to the unit that performs text box conversion in the text detection model. The probability map refers to a map in which the values ​​of all pixels are in the range of 0-1, representing the probability of each pixel being text. The threshold map refers to a map composed of thresholds corresponding to all pixels. Binarization refers to setting a threshold t, such as t=0.8, so that values ​​greater than 0.8 are 1, and values ​​less than 0.8 are 0, and binary means two values. An approximate binary map refers to a map in which the value of each pixel is 0 or 1.

[0125] In practical applications, the document image to be identified can be input into the first feature extraction layer of the text detection model, and the feature extraction backbone (Backbone) in the first feature extraction layer performs multi-scale feature extraction to obtain initial feature maps of different sizes; then the pyramid upsampling module in the first feature extraction layer upsamples each initial feature map to obtain intermediate feature maps of the same size; then, the feature cascade module in the first feature extraction layer performs feature cascade on each intermediate feature map to obtain a target feature map.

[0126] Furthermore, the target feature map is input into the prediction layer of the text detection model, and the prediction layer performs prediction based on the target feature map to obtain a probability map and a threshold map. Then, the probability map and the threshold map are input into the processing layer of the text detection model for differentiable binarization fusion processing to obtain an approximate binarization map. Finally, the approximate binarization map is input into the conversion layer of the text detection model, and the conversion layer performs text box conversion based on the approximate binarization map to obtain each initial text detection box corresponding to the document image. In this way, the accuracy of text detection and the accuracy of the initial text detection box can be improved.

[0127] The Differentiable Binarization NET (DBNET) model is used as an example of a text detection model. DBNET is an algorithm for text detection based on segmentation. In order to increase the feature extraction capability, the Backbone of DBNET uses the resnet101 network, the Net uses the FPN network structure, and the Head contains 3 branches, which eventually outputs the detection box of each text.

[0128] The processing process of the DBNET model: the document image is first preprocessed to obtain the first feature map, which is 1 / 2 in size; the first feature map is passed through the FPN network structure to obtain four feature maps, which are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 in size respectively; these four feature maps are upsampled to 1 / 4 in size, and then feature cascade or fusion (CON-CAT) is performed to obtain the target feature map. The target feature map is predicted by the prediction layer to obtain the probability map and the threshold map. The probability map and the threshold map are fused by the prediction layer through differentiable binarization (DB) to obtain the approximate binary map. Finally, the approximate binary map is input into the conversion layer for text box conversion to obtain the initial text detection boxes corresponding to the document image.

[0129] The DB process is a differentiable binarization process. Conventional binarization is to set a threshold. Here, an approximate binarization image is obtained based on the prediction image and the threshold image.

[0130] By performing differentiable binarization fusion processing on the result of comparing the probability map and the threshold map, an approximate binarization map is obtained, which can not only achieve the purpose of binarization, but also improve the performance of the DBNET model, so that the DBNET model can adaptively set the binarization threshold, which not only simplifies post-processing, but also enhances the performance of text detection. That is, the introduction of DB can increase the gradient amplitude, which is more conducive to optimization, so that the three output maps (probability map, threshold map and approximate binarization map) are better optimized, and the final segmentation result will be excellent.

[0131] In addition, before using the text detection model, it is necessary to train the text detection model. The training process of the text detection model can be as follows:

[0132] Obtain a first sample document image with a text box label;

[0133] Inputting the first sample document image into a first detection model to perform text detection to obtain a predicted text box;

[0134] Calculating a first loss value according to the text box label and the predicted text box;

[0135] According to the first loss value, adjusting the model parameters of the first detection model;

[0136] Continue to execute the step of obtaining the first sample document image carrying the text box label until the first training stop condition is reached to obtain the trained text detection model.

[0137] Specifically, the first sample document image refers to a document image used to train a text detection model. The text box label refers to a true value box used to mark text lines in the first sample document image. The first detection model refers to an untrained text detection model. The predicted text box is a predicted box used to mark text lines obtained by the first detection model performing text detection on the first sample document image. The training stop condition can be that the first loss value is less than a loss threshold, the number of iterations reaches a number threshold, the rate of change of the first loss value is lower than a rate of change threshold, or a combination of the foregoing. The first training stop condition refers to the training stop condition for training a text detection model.

[0138] In the case where the first detection model includes a first feature extraction layer, a prediction layer, a processing layer, and a conversion layer, the first sample document image is input into the first detection model for text detection to obtain a predicted text box, and the specific implementation process is as follows:

[0139] Inputting the first sample document image into the first feature extraction layer to perform feature extraction to obtain a predicted feature map;

[0140] Inputting the prediction feature map into the prediction layer to perform text prediction, and obtaining a prediction probability map and a prediction threshold map;

[0141] Inputting the prediction probability map and the prediction threshold map into the processing layer for differentiable binarization fusion processing to obtain a prediction approximate binarization map;

[0142] The predicted approximate binarization image is input into the conversion layer for text box conversion to obtain a predicted text box corresponding to the first sample document image.

[0143] In one or more optional embodiments of the present invention, paragraph detection may be performed on the document image to be identified according to a set paragraph detection strategy to obtain at least one initial paragraph detection frame.

[0144] In one or more optional embodiments of the present invention, a paragraph detection model may be used to perform paragraph detection on the document image to be identified. That is, paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame. The specific implementation process may be as follows:

[0145] The document image is input into a pre-trained paragraph detection model to perform paragraph detection, and at least one initial paragraph detection frame is obtained.

[0146] Specifically, the paragraph detection model refers to a model that is trained to identify paragraphs in document images. Paragraph detection algorithms include but are not limited to YOLO series algorithms, SSD algorithms, R-CNN series algorithms, and VIT algorithms.

[0147] In practical applications, in order to improve the efficiency of paragraph detection, the pre-trained document images to be recognized can be used to perform paragraph detection, thereby obtaining initial paragraph detection frames corresponding to the document images. In this way, the efficiency of obtaining the initial paragraph detection frames can be improved.

[0148] Optionally, the text detection model includes an input layer, a second feature extraction layer, a feature fusion layer and an output layer; accordingly, the document image is input into a pre-trained paragraph detection model for paragraph detection to obtain at least one initial paragraph detection frame, and the specific implementation process may be as follows:

[0149] Inputting the document image into the input layer for normalization processing to obtain the normalized document image;

[0150] Inputting the normalized document image into the second feature extraction layer to extract features and obtain image features;

[0151] Inputting the image features into a feature fusion layer to perform multi-scale feature fusion to obtain fused features;

[0152] The fused features are input into the output layer for paragraph detection to obtain at least one initial paragraph detection frame.

[0153] Specifically, the input layer refers to the unit in the paragraph detection model that receives the image and performs preprocessing. The second feature extraction layer refers to the unit in the paragraph detection model that performs feature extraction. The feature fusion layer refers to the unit in the paragraph detection model that performs feature fusion. The output layer refers to the unit in the paragraph detection model that performs paragraph box conversion.

[0154] In practical applications, the document image to be identified can be input into the input layer of the paragraph detection model, and the input layer performs normalization and other preprocessing to obtain a normalized initial feature map; the normalized initial feature map is then input into the second feature extraction layer of the paragraph detection model for feature extraction to obtain image features. The image features are then input into the feature fusion layer of the paragraph detection model, and the feature fusion layer performs multi-scale feature fusion based on the image features to obtain fused features. Next, the fused features are input into the output layer of the paragraph detection model, and the output layer performs paragraph frame conversion based on the fused features to obtain each initial paragraph detection frame corresponding to the document image. In this way, the accuracy of paragraph detection and the accuracy of the initial paragraph detection frame can be improved.

[0155] Take the YOLO v5 model as an example for paragraph detection model. The YOLO v5 model is a common target detection algorithm. The network structure of YOLO v5 is divided into four parts: input end (input layer), Backbone (second feature extraction layer), Neck (feature fusion layer) and Head (output layer).

[0156] Input: used for image normalization and other operations. The size of the input image is 608*608. This stage usually includes an image preprocessing stage, which is to scale the input image to a set size, such as 608*608, and perform normalization and other operations. During the training phase, YOLO v5 uses Mosaic data augmentation operations to improve the training speed of the model and the accuracy of the network; and proposes an adaptive anchor box calculation and adaptive image scaling method.

[0157] Backbone (benchmark network): uses the CSPDarknet53 structure for feature extraction. Backbone is usually a network of some excellent classifiers. This module is used to extract some common feature representations. YOLOv5 not only uses the CSPDarknet53 structure, but also uses the Focus structure as the benchmark network.

[0158] Neck (feature fusion layer): usually located between Backbone and Head, SPP (Spatial Pyramid Pooling) is used to further extract features, that is, multi-scale feature fusion. Using Neck can further improve the diversity and robustness of features.

[0159] Head (head network): Outputs the target detection results. Specifically, Head is used to complete the output of target detection results. It contains a classification branch (paragraph category) and a regression branch (regression bounding box coordinates).

[0160] In addition, before using the paragraph detection model, it is necessary to train the paragraph detection model. The training process of the paragraph detection model can be as follows:

[0161] Obtain a second sample document image with a paragraph box tag;

[0162] Inputting the second sample document image into a second detection model to perform paragraph detection to obtain a predicted paragraph box;

[0163] Calculating a second loss value according to the paragraph box label and the predicted paragraph box;

[0164] According to the second loss value, adjusting the model parameters of the second detection model;

[0165] Continue to execute the step of obtaining the second sample document image carrying the paragraph box label until the second training stop condition is reached, thereby obtaining the trained paragraph detection model.

[0166] Specifically, the second sample document image refers to a document image used to train the paragraph detection model. The paragraph box label refers to a true value box used to mark the paragraph in the second sample document image. The second detection model refers to an untrained paragraph detection model. The predicted paragraph box is the predicted box used to mark the paragraph obtained by the second detection model performing paragraph detection on the second sample document image. The second training stop condition refers to the training stop condition of the training paragraph detection model.

[0167] In the case where the first detection model includes an input layer, a second feature extraction layer, a feature fusion layer, and an output layer, the second sample document image is input into the second detection model for paragraph detection to obtain a predicted paragraph frame, and the specific implementation process is as follows:

[0168] Inputting the second sample document image into the input layer for normalization processing to obtain a normalized second sample document image;

[0169] Inputting the normalized second sample document image into the second feature extraction layer for feature extraction to obtain predicted image features;

[0170] Inputting the predicted image features into a feature fusion layer for multi-scale feature fusion to obtain predicted fusion features;

[0171] The predicted fusion features are input into the output layer for paragraph detection to obtain the predicted paragraph box corresponding to the document image.

[0172] In one or more optional embodiments of the present invention, determining at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame includes:

[0173] For each of the initial paragraph detection frames, compare the second position information of the initial paragraph detection frame with the first position information of each of the initial text detection frames, and determine target first position information in each of the first position information that overlaps with the second position information;

[0174] The initial text detection frame corresponding to the target first position information is determined as a target text detection frame associated with the initial paragraph detection frame.

[0175] In practical applications, the following steps are performed for each initial paragraph detection frame: the second position information of the initial paragraph detection frame is compared with each first position information, and the target first position information that overlaps or overlaps with the first position information is determined from each second position information. The initial text detection frame corresponding to each first position information is the target text detection frame associated with the initial paragraph detection frame. According to the above method, all initial paragraph detection frames are traversed to determine at least one target text detection frame associated with each initial paragraph detection frame. In this way, determining the target text detection frame based on position overlap can improve the accuracy of the target text detection frame, thereby improving the accuracy and efficiency of paragraph detection.

[0176] Combine the following Figure 3 The paragraph detection method provided by the present invention is further described. Figure 3 , Figure 3 This is the second flow chart of the paragraph detection method provided by the present invention:

[0177] First, define the document structure from a purely visual perspective and extract relatively independent structural attributes from complex documents. Mainly define the following two categories: regular paragraphs and list paragraphs. Regular paragraphs are divided into Chinese paragraphs, English paragraphs, and titles. These three categories can also be classified separately, which reflects the flexibility of classification. At the same time, the independence of the features of each classification must be guaranteed to ensure network convergence.

[0178] The title feature (the preset paragraph feature corresponding to the title type) is usually a single line, appearing in the middle of the top of the document, or in the previous line of each paragraph, aligned to the top of the page, and usually has a serial number.

[0179] Chinese paragraph features (preset paragraph features corresponding to the Chinese paragraph type): first line indentation, left and right margins.

[0180] English paragraph features (preset paragraph features corresponding to English paragraph types): the first line is aligned to the left or indented, and the last line is generally not aligned to the left.

[0181] List paragraph (preset paragraph feature corresponding to list paragraph type), several consecutive lines, with symbols on the left, such as *, -, >, etc., or serial letters, numbers, Chinese numbers, etc.

[0182] Then, paragraph detection is performed from a purely visual perspective, using a detection algorithm to detect the document structure attributes (preset paragraph features) defined above.

[0183] like Figure 3 As shown, first input a document image, the document image goes through two branches, respectively passing through the text detection model (DBNET model) and the paragraph detection model (YOLO v5), the text detection model obtains the initial text detection frame, and the paragraph detection model obtains the initial paragraph detection frame based on the preset paragraph features (the initial paragraph detection frame here is relatively rough, there may be small text boxes missed or offset due to scaling). Then, the initial paragraph detection frame and the initial text detection frame are input into the post-processing module, and the inclusion relationship between the initial text detection frame and the initial paragraph detection frame is calculated according to the position information of the initial text frame and the position information of the initial paragraph frame. Finally, the minimum external frame of the initial text detection frame (text frame set) contained in each initial paragraph detection frame is taken to obtain a more accurate target paragraph detection frame.

[0184] like Figure 4 The schematic diagram of the effect of the paragraph detection method provided by the present invention is shown, the paragraphs marked with dotted lines are list paragraphs, and the paragraphs marked with solid lines are regular paragraphs.

[0185] In this embodiment, the characteristics of the document structure in various scenarios are defined from a visual perspective; from the perspective of deep learning, the detection of the document structure is realized; based on actual industrial performance, the YOLO v5 model and the DBNET model are combined to realize the function of paragraph detection. It can be seen that the paragraph detection method provided by the present invention has high generalization and can cope with the document structure detection requirements of various scenarios; it has high speed and can quickly locate the document structure through common detection models; from a visual perspective, the characteristics of common document structures are defined, and a certain degree of customized document structure is accepted. The document structure can be effectively detected for document images of various scenarios. Customize the document structure attributes in various complex and chaotic scenarios to meet the needs of downstream tasks.

[0186] The paragraph detection device provided by the present invention is described below. The paragraph detection device described below and the paragraph detection method described above can be referred to each other.

[0187] Figure 5is a schematic diagram of the structure of the paragraph detection device provided by the present invention, such as Figure 5 As shown, the paragraph detection device 500 includes: a detection module 501, a determination module 502 and a marking module 503, wherein:

[0188] The detection module 501 is configured to perform text detection on the document image to be recognized to obtain at least one initial text detection frame, and perform paragraph detection on the document image to obtain at least one initial paragraph detection frame;

[0189] A determination module 502 is configured to determine at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame;

[0190] The marking module 503 is configured to determine a target paragraph detection frame for each of the initial paragraph detection frames by associating the target text detection frames with the initial paragraph detection frame, and mark the target paragraph in the document image according to the target paragraph detection frame.

[0191] The paragraph detection device provided by the present invention performs text detection on the document image to be identified to obtain at least one initial text detection frame, and performs paragraph detection on the document image to obtain at least one initial paragraph detection frame; according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame, at least one target text detection frame associated with each initial paragraph detection frame is determined from each initial text detection frame; for each initial paragraph detection frame, a target paragraph detection frame is determined according to each target text detection frame associated with the initial paragraph detection frame, and the target paragraph in the document image is marked according to the target paragraph detection frame. Through the two-way combination of text frame and paragraph frame, paragraphs in the document image can be detected quickly and accurately, thereby realizing the detection of document structure; it has high generalization and can cope with the document structure detection requirements of various scenarios; it has high speed and can quickly realize document structure positioning. Customize document structure attributes in various complex and chaotic scenarios to meet the needs of downstream tasks.

[0192] In one or more optional embodiments of the present invention, the detection module 501 is further configured to:

[0193] According to the preset paragraph features corresponding to each paragraph type, paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame.

[0194] In one or more optional embodiments of the present invention, the detection module 501 is further configured to:

[0195] For each preset paragraph feature corresponding to the paragraph type, an initial paragraph belonging to the paragraph type in the document image is identified, and the initial paragraph is marked with an initial paragraph detection frame.

[0196] In one or more optional embodiments of the present invention, the paragraph types include a regular paragraph type and a list paragraph type;

[0197] The regular paragraphs include a title type and a body paragraph type, and the body paragraph type includes a Chinese paragraph type and / or an English paragraph type.

[0198] In one or more optional embodiments of the present invention, the preset paragraph features corresponding to the list paragraph type include: the paragraph contains a plurality of continuous text lines, and the left side of the paragraph contains a plurality of bullet points and / or a plurality of continuous numbers;

[0199] The preset paragraph features corresponding to the title type include: the number of text lines in the paragraph is less than the preset number of title lines; the display position of the paragraph is the middle of the document, or the display position is the previous line of the body paragraph, the left side of the paragraph is aligned to the top, and the left side of the paragraph contains a serial number;

[0200] The preset paragraph features corresponding to the Chinese paragraph type include: the first line of the paragraph is indented and the paragraph is aligned to the left and right;

[0201] The preset paragraph features corresponding to the English paragraph type include: the first line of the paragraph is aligned or indented on the left side.

[0202] In one or more optional embodiments of the present invention, the determining module 502 is further configured to:

[0203] Combining the target text detection frames associated with the initial paragraph detection frame to obtain a combined frame;

[0204] The minimum circumscribed rectangle of the combination box is determined as the target paragraph detection box.

[0205] In one or more optional embodiments of the present invention, the detection module 501 is further configured to:

[0206] Input the document image to be recognized into the pre-trained text detection model for text detection to obtain at least one initial text detection frame;

[0207] The document image is input into a pre-trained paragraph detection model for paragraph detection to obtain at least one initial paragraph detection frame.

[0208] In one or more optional embodiments of the present invention, the text detection model includes a first feature extraction layer, a prediction layer, a processing layer, and a conversion layer;

[0209] Accordingly, the detection module 501 is further configured to:

[0210] Inputting the document image to be identified into the first feature extraction layer for feature extraction to obtain a target feature map;

[0211] Inputting the target feature map into the prediction layer for text prediction to obtain a probability map and a threshold map;

[0212] Inputting the probability map and the threshold map into a processing layer for differentiable binarization fusion processing to obtain an approximate binarization map;

[0213] The approximate binarization image is input into the conversion layer for text box conversion to obtain at least one initial text detection box.

[0214] In one or more optional embodiments of the present invention, the text detection model includes an input layer, a second feature extraction layer, a feature fusion layer and an output layer;

[0215] Accordingly, the detection module 501 is further configured to:

[0216] Inputting the document image into the input layer for normalization processing to obtain the normalized document image;

[0217] Inputting the normalized document image into the second feature extraction layer to extract features and obtain image features;

[0218] Inputting the image features into a feature fusion layer to perform multi-scale feature fusion to obtain fused features;

[0219] The fused features are input into the output layer for paragraph detection to obtain at least one initial paragraph detection frame.

[0220] In one or more optional embodiments of the present invention, the determining module 502 is configured to:

[0221] For each of the initial paragraph detection frames, compare the second position information of the initial paragraph detection frame with the first position information of each of the initial text detection frames, and determine target first position information in each of the first position information that overlaps with the second position information;

[0222] The initial text detection frame corresponding to the target first position information is determined as a target text detection frame associated with the initial paragraph detection frame.

[0223] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the paragraph detection method, which includes: performing text detection on the document image to be identified to obtain at least one initial text detection frame, and performing paragraph detection on the document image to obtain at least one initial paragraph detection frame; determining at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame; for each initial paragraph detection frame, determining the target paragraph detection frame according to each target text detection frame associated with the initial paragraph detection frame, and marking the target paragraph in the document image according to the target paragraph detection frame.

[0224] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0225] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the paragraph detection method provided by the above-mentioned methods, which method includes: performing text detection on a document image to be identified to obtain at least one initial text detection frame, and performing paragraph detection on the document image to obtain at least one initial paragraph detection frame; determining at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to first position information of each initial text detection frame and second position information of each initial paragraph detection frame; for each initial paragraph detection frame, determining a target paragraph detection frame according to each target text detection frame associated with the initial paragraph detection frame, and marking the target paragraph in the document image according to the target paragraph detection frame.

[0226] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the paragraph detection method provided by the above-mentioned methods, the method comprising: performing text detection on a document image to be identified to obtain at least one initial text detection frame, and performing paragraph detection on the document image to obtain at least one initial paragraph detection frame; determining at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to first position information of each initial text detection frame and second position information of each initial paragraph detection frame; for each initial paragraph detection frame, determining a target paragraph detection frame according to each target text detection frame associated with the initial paragraph detection frame, and marking the target paragraph in the document image according to the target paragraph detection frame.

[0227] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0228] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A paragraph detection method, characterized in that: include: Performing text detection on the document image to be recognized to obtain at least one initial text detection frame, and performing paragraph detection on the document image to obtain at least one initial paragraph detection frame; Determining at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame; For each of the initial paragraph detection frames, a target paragraph detection frame is determined according to each target text detection frame associated with the initial paragraph detection frame, and the target paragraph in the document image is annotated according to the target paragraph detection frame.

2. The paragraph detection method according to claim 1, characterized in that: The performing paragraph detection on the document image to obtain at least one initial paragraph detection frame includes: According to the preset paragraph features corresponding to each paragraph type, paragraph detection is performed on the document image to obtain at least one initial paragraph detection frame.

3. The paragraph detection method according to claim 2, characterized in that: The step of performing paragraph detection on the document image according to preset paragraph features corresponding to each paragraph type to obtain at least one initial paragraph detection frame includes: For each paragraph type, a preset paragraph feature corresponding to the paragraph type is used to identify an initial paragraph in the document image that belongs to the paragraph type, and annotate the initial paragraph with an initial paragraph detection frame; The paragraph types include a regular paragraph type and a list paragraph type; The regular paragraphs include a title type and a body paragraph type, and the body paragraph type includes a Chinese paragraph type and / or an English paragraph type.

4. The paragraph detection method according to claim 3, characterized in that: The preset paragraph features corresponding to the list paragraph type include: the paragraph contains multiple continuous text lines, and the left side of the paragraph contains multiple bullet points and / or multiple continuous numbers; The preset paragraph features corresponding to the title type include: the number of text lines in the paragraph is less than the preset number of title lines; the display position of the paragraph is the middle of the document, or the display position is the previous line of the body paragraph, the left side of the paragraph is aligned to the top, and the left side of the paragraph contains a serial number; The preset paragraph features corresponding to the Chinese paragraph type include: the first line of the paragraph is indented and the paragraph is aligned to the left and right; The preset paragraph features corresponding to the English paragraph type include: the first line of the paragraph is aligned or indented on the left side.

5. The paragraph detection method according to claim 1, characterized in that: The step of determining the target paragraph detection frame according to each target text detection frame associated with the initial paragraph detection frame includes: Combining the target text detection frames associated with the initial paragraph detection frame to obtain a combined frame; The minimum circumscribed rectangle of the combination box is determined as the target paragraph detection box.

6. The paragraph detection method according to any one of claims 1 to 5, characterized in that: The step of performing text detection on the document image to be identified to obtain at least one initial text detection frame, and performing paragraph detection on the document image to obtain at least one initial paragraph detection frame includes: Input the document image to be recognized into the pre-trained text detection model for text detection to obtain at least one initial text detection frame; The document image is input into a pre-trained paragraph detection model for paragraph detection to obtain at least one initial paragraph detection frame.

7. The paragraph detection method according to claim 6, characterized in that: The text detection model includes a first feature extraction layer, a prediction layer, a processing layer and a conversion layer; Accordingly, the document image to be identified is input into a pre-trained text detection model for text detection to obtain at least one initial text detection frame, including: Inputting the document image to be identified into the first feature extraction layer for feature extraction to obtain a target feature map; Inputting the target feature map into the prediction layer for text prediction to obtain a probability map and a threshold map; Inputting the probability map and the threshold map into a processing layer for differentiable binarization fusion processing to obtain an approximate binarization map; The approximate binarization image is input into the conversion layer for text box conversion to obtain at least one initial text detection box.

8. The paragraph detection method according to claim 6, characterized in that: The text detection model includes an input layer, a second feature extraction layer, a feature fusion layer and an output layer; Accordingly, the step of inputting the document image into a pre-trained paragraph detection model for paragraph detection to obtain at least one initial paragraph detection frame includes: Inputting the document image into the input layer for normalization processing to obtain the normalized document image; Inputting the normalized document image into the second feature extraction layer to extract features and obtain image features; Inputting the image features into a feature fusion layer to perform multi-scale feature fusion to obtain fused features; The fused features are input into the output layer for paragraph detection to obtain at least one initial paragraph detection frame.

9. The paragraph detection method according to any one of claims 1 to 5, characterized in that: The determining, from each of the initial text detection frames, at least one target text detection frame associated with each of the initial paragraph detection frames according to the first position information of each of the initial text detection frames and the second position information of each of the initial paragraph detection frames, comprises: For each of the initial paragraph detection frames, compare the second position information of the initial paragraph detection frame with the first position information of each of the initial text detection frames, and determine target first position information in each of the first position information that overlaps with the second position information; The initial text detection frame corresponding to the target first position information is determined as a target text detection frame associated with the initial paragraph detection frame.

10. A paragraph detection device, characterized in that: include: A detection module is configured to perform text detection on the document image to be recognized to obtain at least one initial text detection frame, and perform paragraph detection on the document image to obtain at least one initial paragraph detection frame; a determination module configured to determine at least one target text detection frame associated with each initial paragraph detection frame from each initial text detection frame according to the first position information of each initial text detection frame and the second position information of each initial paragraph detection frame; The annotation module is configured to determine a target paragraph detection frame for each of the initial paragraph detection frames by associating the target text detection frames with the initial paragraph detection frame, and annotate the target paragraph in the document image according to the target paragraph detection frame.

Images