A text detection and recognition method and system for complex layout images

Abstract

This invention relates to a method and system for text detection and recognition in complex layout images. The method includes: performing text detection on a high-resolution original image and generating multiple candidate text boxes containing text; post-processing the multiple candidate text boxes based on the layout structure characteristics of the text to obtain multiple effective text regions; cropping the original image according to the effective text regions to obtain text sub-images corresponding to the effective text regions; and performing text recognition on each of the multiple text sub-images to obtain text recognition results. The method and system of this invention improve the accuracy, stability, and efficiency of text recognition in complex layout image scenarios.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and text recognition technology; specifically, it relates to a method and system for text detection and recognition of complex layout images. Background Technology

[0002] With the development of computer vision and text recognition technologies such as OCR, existing text recognition systems typically employ a processing method that detects and directly recognizes text across the entire image.

[0003] However, in practical applications, especially in scenarios with complex layouts such as advertisements, packaging, and marketing posters, the original full-image often features complex layouts, dense text, varying font sizes, multi-line arrangement, and includes footnotes, superscripts, or multi-column structures. In such cases of high image resolution and text density, the effective feature scale of the text in the image is relatively reduced. Direct recognition methods in this situation are prone to problems such as unrecognizable text, repeated detection of the same text region, text blocks being split into multiple small regions, or the recognition area containing a large amount of non-target background. This leads to reduced recognition accuracy and hinders subsequent structured processing and utilization of the text content.

[0004] Furthermore, the recognition performance of existing text recognition models primarily depends on the effective character height of the input text, such as requiring the text pixels to be greater than 15 pixels, rather than on the original resolution of the input image. Therefore, when a high-resolution original image is input into a text recognition model, due to the model's own limitations on maximum resolution or maximum input side length, normalization processing is required. The high-resolution original image is proportionally reduced, resulting in the pixel count of text or symbols that would normally be recognizable by the human eye at their original size being lower than the effective character height required by the model, thus making them unrecognizable or incorrectly recognized.

[0005] In existing technologies, for high-resolution images containing complex text, a sliding window cropping method is typically used to evenly divide the original image into small sub-images before text recognition. However, in image scenarios with diverse layout features, this approach is prone to segmenting related text within the same paragraph, or even individual characters, into different sub-images. This can lead to problems such as lost or falsely detected text and incomplete information extraction after recognition, severely impacting the subsequent comprehensive and accurate understanding and processing of the text content in the image. Summary of the Invention

[0006] In view of this, the present invention provides a method and system for text detection and recognition of complex layout images, thereby solving or at least alleviating one or more of the above-mentioned problems and other problems existing in the prior art.

[0007] To achieve the aforementioned objectives, a first aspect of the present invention provides a method for text detection and recognition of complex layout images, the method comprising:

[0008] Step 1: Perform text detection on the high-resolution original image and generate multiple candidate text boxes containing text;

[0009] Step 2: Based on the layout structure characteristics of the text, post-process the multiple candidate text boxes to obtain multiple effective text regions;

[0010] Step 3: Crop the original image according to the effective text region to obtain a text sub-image corresponding to the effective text region;

[0011] Step 4: Perform text recognition on each of the multiple text sub-images to obtain text recognition results, wherein step 2 includes, but is not limited to:

[0012] Step 2-1: Perform non-maximum suppression processing on the candidate text boxes to eliminate overlapping duplicate text boxes;

[0013] Step 2-2: Divide the candidate text boxes into subtitle text boxes and text text boxes, and merge the subtitle text boxes with the text text boxes that are closest to them;

[0014] Steps 2-3: Determine the paragraph relationship between multiple candidate text boxes, and merge candidate text boxes belonging to the same paragraph region;

[0015] Steps 2-4: Determine the nesting relationship between the multiple candidate text boxes and delete the candidate text boxes nested inside other candidate text boxes;

[0016] Steps 2-5: Expand the borders of the candidate text boxes.

[0017] In the text detection and recognition method described above, optionally, in step 2-2:

[0018] Set area threshold ratio coefficient Height threshold Width threshold When the area of ​​the candidate text box satisfy And the height of the candidate text box and width satisfy When the candidate text box is identified as a superscript text box, it is determined to be a superscript text box. It also classifies other candidate text boxes that do not meet the criteria as text boxes. ,in, The total width of the original image. The total height of the original image.

[0019] In the text detection and recognition method described above, optionally, in step 2-2:

[0020] Determine the superscript text box With the text box Positional relationships, when satisfying

[0021]

[0022]

[0023]

[0024]

[0025] At that time, the text box will be... The text box is identified as the superscript text box. The relevant text boxes, among which, , , The superscript text boxes are respectively The x-coordinate of the left boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. , , , The text boxes are respectively The x-coordinate of the left boundary, the x-coordinate of the right boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. This is the tolerance coefficient for horizontal position. To horizontally extend the tolerance factor, This is the vertical tolerance coefficient above. This is the vertical tolerance coefficient below. The total width of the original image. The total height of the original image.

[0026] Select the text box that corresponds to the superscript text box in the relevant text box. The nearest text box ,satisfy

[0027]

[0028]

[0029] The superscript text box The text box closest to the nearest text box The text boxes are merged, and the new merged text box is the superscript text box. The text box closest to the nearest text box The bounding rectangle.

[0030] In the text detection and recognition method described above, optionally, in steps 2-3:

[0031] Set horizontal overlap threshold and line spacing threshold coefficient When the horizontal overlap ratio between the two candidate text boxes is... and line spacing satisfy and When the two candidate text boxes are determined to belong to the same paragraph region,

[0032]

[0033] The horizontal overlap width between the two candidate text boxes. The total width of the narrower text box in the two candidate text boxes. The total height of the original image.

[0034] In the text detection and recognition method described above, optionally, in steps 2-3:

[0035] When the candidate text box With multiple candidate text boxes When the candidate text boxes are determined to belong to the same paragraph area, the candidate text boxes are... and the nearest candidate text box The text boxes are merged, and the resulting new text box is the candidate text box. The nearest candidate text box The bounding rectangle of , where,

[0036]

[0037]

[0038] For the candidate text box With the candidate text box The line spacing between them The left boundary difference weighting coefficient. , The candidate text boxes are respectively , The left boundary x-coordinate.

[0039] In the text detection and recognition method described above, optionally, in steps 2-4:

[0040] Set position tolerance coefficient When the candidate text box With the candidate text box satisfy

[0041]

[0042]

[0043] When, determine the candidate text box Nested in the candidate text box Inside, and the candidate text box Delete, among which,

[0044]

[0045] , , , The candidate text boxes are respectively The x-coordinate of the left boundary, the x-coordinate of the right boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. , , , The candidate text boxes are respectively The x-coordinate of the left boundary, the x-coordinate of the right boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. This is the tolerance ratio coefficient. The total width of the original image. The total height of the original image.

[0046] In the text detection and recognition method described above, optionally, in steps 2-5:

[0047] Horizontal expansion amount of border extension Vertical upward expansion Vertical downward expansion amount satisfy

[0048]

[0049]

[0050]

[0051] in, This is the horizontal expansion factor. This is the upward expansion coefficient. For the downward expansion coefficient, satisfying , The total width of the original image. The total height of the original image.

[0052] The new text box after border expansion meets the requirements.

[0053]

[0054]

[0055]

[0056]

[0057] in, , , , The x-coordinates of the left and right borders, the y-coordinates of the top and bottom borders of the original text box. , , , This defines the x-coordinate of the left boundary, x-coordinate of the right boundary, y-coordinate of the top boundary, and y-coordinate of the bottom boundary of the new text box.

[0058] In the text detection and recognition method described above, optionally, in step 4, before performing text recognition on the text sub-image:

[0059] Calculate scaling ratio ,

[0060]

[0061] in, The height of the text sub-image. The width of the text sub-image. The target input height for the text recognition model. This is the maximum width limit of the text recognition model.

[0062] The text sub-image is scaled proportionally.

[0063]

[0064] in, For the text subgraph, This is a scaled-down text sub-image.

[0065] For the scaled text sub-image Perform text recognition.

[0066] To achieve the aforementioned objectives, a second aspect of the present invention provides a text detection and recognition system for complex layout images, capable of implementing the text detection and recognition method described above, wherein the text detection and recognition system includes:

[0067] The text detection module is used to perform text detection on the original image and generate multiple candidate text boxes containing text.

[0068] A post-processing module is used to perform post-processing on multiple candidate text boxes based on the layout structure characteristics of the text, and to obtain multiple valid text regions. The post-processing includes: performing non-maximum suppression on the candidate text boxes to eliminate overlapping duplicate text boxes; distinguishing the candidate text boxes into subscript text boxes and regular text boxes, and merging the subscript text boxes with the nearest regular text box; determining the paragraph relationship between the multiple candidate text boxes, and merging candidate text boxes belonging to the same paragraph region; determining the nesting relationship between the multiple candidate text boxes, and deleting candidate text boxes nested within other candidate text boxes; and expanding the borders of the candidate text boxes.

[0069] The image cropping module is used to crop the original image according to the effective text region and obtain a text sub-image corresponding to the effective text region;

[0070] The text recognition module is used to perform text recognition on multiple text sub-images respectively and obtain text recognition results.

[0071] This invention provides a text detection and recognition method for complex layout images. Through preliminary text detection and post-processing based on the layout structure, the high-resolution original image is cropped into multiple low-resolution text sub-images containing effective text regions. Text recognition is then performed on these sub-images. This method elevates the text features in the original image to a scale range easily identifiable, making text information unrecognizable in traditional methods identifiable while preserving text details, thus improving the clarity and accuracy of text recognition. Furthermore, the layout-based post-processing not only reduces redundant recognition areas and lowers background interference, improving the stability and efficiency of text recognition, but also preserves the structural information of the text, enhancing the completeness and accuracy of text recognition. This facilitates subsequent comprehensive and unbiased text information extraction and understanding.

[0072] In optional embodiments, post-processing methods such as non-maximum suppression, merging superscript and text boxes, merging paragraph-related text boxes, deleting nested text boxes, and expanding text box borders can further preserve the complete semantic information and logical structure of the text content. By eliminating redundancy, repeated cropping and recognition are avoided, and the integrity of the text is prevented from being destroyed during subsequent cropping, thereby improving the efficiency, accuracy, stability, and integrity of text recognition. In a further optional embodiment, when expanding the text box, the downward expansion range is greater than the upward expansion range, which better matches the asymmetry of Chinese characters and English letters in the vertical structure, protecting the descending part of the character from being cropped and destroying it, and improving the recognizability of the text.

[0073] The present invention further provides a text detection and recognition system for complex layout images, and thus the text detection and recognition system also has the above-mentioned advantages. Attached Figure Description

[0074] The disclosure of this invention will become more apparent from the accompanying drawings. It should be understood that these drawings are for illustrative purposes only and are not intended to limit the scope of protection of this invention. In the drawings:

[0075] Figure 1 This is a flowchart of an embodiment of the text detection and recognition method of the present invention.

[0076] Figure 2 This is a schematic diagram of an embodiment of the original image to which the present invention applies.

[0077] Figure 3 for Figure 2 A schematic diagram of the effective text region obtained from the original image using the post-processing method of this invention. Detailed Implementation

[0078] Referring to the accompanying drawings and specific embodiments, the structure, composition, features, and advantages of the text detection and recognition method and system of the present invention will be described below by way of example. However, all descriptions should not be construed as limiting the present invention in any way.

[0079] Furthermore, for any single technical feature described or implied in the embodiments mentioned herein, or any single technical feature shown or implied in the various figures, the present invention still allows for any combination or deletion of these technical features (or their equivalents) without any technical obstacle, and thus these further embodiments according to the present invention should also be considered within the scope of this description.

[0080] Figure 1This is a flowchart illustrating an embodiment of the text detection and recognition method of the present invention. The text detection and recognition method of the present invention is designed for high-resolution, complex layout images with diverse text types, high text density, and rich text-image layers, such as advertisements, product packaging, and poster images. Compared to form and document images with structured text content and key positioning information, and magazine and newspaper images with regular paragraph structures and clear reading order, such complex layout images, if simply cropped into uniform sections for text recognition, are prone to losing text information, damaging the integrity and semantic coherence of the text content, and producing incorrect recognition results.

[0081] Especially when these complex layouts contain text content, such as marketing text, they often include superscript text in addition to regular text. This superscript text is either placed in the upper right corner of the text for annotation or forms footnotes with the following text. During text recognition, it is particularly important to preserve the structural relevance of superscript text within paragraphs to ensure that multiple paragraphs corresponding to the same superscript can be matched completely and without bias during subsequent information extraction. This is beneficial for applications in various practical scenarios such as advertising copy review.

[0082] Optionally, the text detection and recognition method used in this invention can be implemented based on OCR (Opical Character Recognition) technology, or it can be based on other text recognition technologies. This invention does not limit this.

[0083] like Figure 1 As shown, for the high-resolution original image to be recognized, the present invention first performs region-level text detection in step 1 and generates multiple candidate text boxes containing text. Region-level text detection is a technique for locating and defining text regions in an image, using bounding boxes to provide regions of interest for subsequent text recognition. The present invention uses a rectangular bounding box as an example for detailed explanation later, but it should be noted that the text detection and recognition method and system of the present invention are also applicable to bounding boxes of other shapes after simple modifications.

[0084] Under high-resolution, complex layout conditions, region-level text detection inevitably encounters various problems such as excessive segmentation, insufficient segmentation, nesting, and / or overlap. Therefore, in step 2 of this invention, based on the layout structure characteristics of the text, post-processing is performed on the multiple candidate text boxes obtained in step 1. In optional embodiments, the post-processing method may include, but is not limited to, methods such as... Figure 1The process is as follows: Non-maximum suppression is used to eliminate overlapping text boxes; candidate text boxes are divided into subscript text boxes and text text boxes, and the subscript text boxes and the nearest text text boxes are merged; the paragraph relationship between multiple candidate text boxes is determined, and candidate text boxes belonging to the same paragraph area are merged; the nesting relationship between multiple candidate text boxes is determined, and candidate text boxes nested inside other candidate text boxes are deleted; the borders of candidate text boxes are expanded.

[0085] In step 2-1, non-maximum suppression (NMS) is a method used to remove duplicate bounding boxes after target detection. For the same target, only the bounding box with the highest confidence is retained. NMS is a common post-processing method in the prior art, and will not be elaborated upon here.

[0086] In step 2-2, to incorporate superscript symbols such as ¹²³, *, etc., into their corresponding text areas and maintain the correct labeling relationship between the text and superscripts, and between superscripts and footnotes in the image, to avoid them being cropped out separately and making it impossible to accurately distinguish between the text and footnotes used for annotation, it is first necessary to distinguish between the superscript text box and the text text box. In an optional embodiment, an area threshold ratio coefficient is set. Height threshold and width threshold When the area of ​​the candidate text box ,high and width satisfy , and When this happens, the candidate text box is determined to be a superscript text box. It also classifies other candidate text boxes that do not meet the criteria as text boxes. .in, The total width of the original image. This represents the total height of the original image.

[0087] Before merging the superscript text box with the regular text box, you need to confirm the superscript text box. With text boxes The interrelationship. Further, optionally, a horizontal position tolerance coefficient can be set. Horizontal expansion tolerance coefficient Vertical tolerance coefficient above Vertical tolerance coefficient below When satisfied

[0088]

[0089]

[0090]

[0091]

[0092] At that time, it can be known that the superscript text box The position is in the text box The text box is located at the top right corner. It is a superscript text box The relevant text boxes. Among them, , , These are the superscript text boxes. The x-coordinate of the left boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. , , , The text boxes are respectively The left boundary x-coordinate, right boundary x-coordinate, upper boundary y-coordinate, and lower boundary y-coordinate.

[0093] Single superscript text box There may be multiple related text boxes; in this case, select the one that corresponds to the superscript text box. The nearest text box Perform the merging. Construct the cost function.

[0094] The nearest text box satisfy

[0095]

[0096] The merged text box is a superscript text box. The nearest text box The bounding rectangle of the superscript. Thus, the superscript with annotation function is merged with its associated body text.

[0097] In addition, there are also superscript texts that constitute footnote content. The methods for merging such superscript texts are conventional techniques in this field, and will not be elaborated here.

[0098] Incorporating isolated superscript characters into the text box containing the most relevant text can prevent semantic breaks caused by superscripts being cropped and recognized separately later. This ensures accurate differentiation between the main text and footnotes used for annotation, as well as correct correspondence between the main text and footnotes that provide explanations of the main text. This improves the completeness and accuracy of the text recognition results in terms of understanding, making it more suitable for practical work scenarios such as information matching and document review.

[0099] In steps 2-3, when determining the paragraph relationship between multiple candidate text boxes, optionally, a horizontal overlap threshold can be set. and line spacing threshold coefficient When the candidate text box With candidate text boxes Horizontal overlap ratio between satisfy When two candidate text boxes are in the same column, they can be considered to be in the same column. With the candidate text box line spacing between satisfy If the two candidate text boxes are vertically adjacent, then they can be considered to belong to the same paragraph area.

[0100] Among them, the horizontal overlap ratio Defined as

[0101]

[0102] The horizontal overlap width between the two candidate text boxes. The total width of the narrower text box in the two candidate text boxes.

[0103] This method of judgment based on relative overlap can avoid the influence of factors such as text font size, absolute width of candidate text boxes, and text alignment format between paragraphs on the determination of column relationships, thus improving the robustness of the post-processing process.

[0104] Further, optionally, when the candidate text box With multiple candidate text boxes When they belong to the same paragraph area, select the candidate text box. and the nearest candidate text box Perform the merging. Set the left boundary difference weighting coefficient. Construct the cost function

[0105] in, For candidate text boxes The left boundary x-coordinate, For candidate text boxes The left boundary x-coordinate. The nearest candidate text box. satisfy

[0106] The merged text box is the candidate text box. The nearest candidate text box The bounding rectangle.

[0107] Thus, multiple separate candidate text boxes in the same column are merged into a new text box containing the complete paragraph content, ensuring the logical and structural integrity and continuity of the selected content, facilitating subsequent overall cropping and text recognition. Furthermore, this left-boundary alignment method also aligns with the left-to-right arrangement of text content within a paragraph.

[0108] In steps 2-4, the nesting relationship between multiple candidate text boxes is determined, and candidate text boxes nested within other candidate text boxes are deleted. In an optional embodiment, a position tolerance coefficient is set. When the candidate text box With candidate text boxes satisfy

[0109]

[0110]

[0111] When determining candidate text boxes The bounding boxes are all within the candidate text boxes. Inside the bounding box, thus placing the candidate text box Delete. Among them, , , , Candidate text boxes The x-coordinate of the left boundary, the x-coordinate of the right boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. , , , Candidate text boxes The left boundary x-coordinate, right boundary x-coordinate, upper boundary y-coordinate, and lower boundary y-coordinate.

[0112] Alternatively, a tolerance ratio factor can be set. Make the position tolerance coefficient satisfy

[0113]

[0114] By removing nested redundant candidate text boxes, repeated text recognition can be avoided, thus improving the efficiency of text recognition. At the same time, it also prevents situations where internal strokes of characters are misidentified as independent text, enhancing the robustness of post-processing steps.

[0115] In steps 2-5, the candidate text boxes are expanded horizontally and vertically. Optionally, a horizontal expansion coefficient is set. Upward expansion coefficient Downward expansion coefficient This increases the horizontal expansion of the candidate text box. Vertical upward expansion Vertical downward expansion amount satisfy

[0116]

[0117]

[0118]

[0119] Therefore, the new text box after border expansion satisfies

[0120]

[0121]

[0122]

[0123]

[0124] in, , , , The x-coordinates of the left and right borders, the y-coordinates of the top and bottom borders of the original text box. , , , This defines the x-coordinate of the left boundary, x-coordinate of the right boundary, y-coordinate of the top boundary, and y-coordinate of the bottom boundary of the new text box.

[0125] Proportionally expanding the bounding box of the candidate text box can prevent damage to the integrity of the text during subsequent cropping, thus improving the accuracy and stability of text recognition.

[0126] In a further optional embodiment, the downward expansion range of the candidate text box is set to be greater than the upward expansion range, i.e. Because commonly used characters such as Chinese and English have asymmetrical vertical structures, most of them have a descending part that extends downwards. The clipping of these descending parts can significantly affect the recognizability of the characters. Therefore, when expanding the border, the downward expansion is carried out over a larger range to ensure the accuracy of subsequent text recognition.

[0127] Based on the post-processing methods described above, multiple stable and valid text regions containing complete text information can be obtained from the candidate text boxes. It should be noted that... Figure 1The post-processing flow shown is merely an illustrative representation of one embodiment of the present invention. The present invention is not limited to the steps shown, nor is it intended to require or imply that the post-processing steps of each embodiment must be performed in the order shown. Those skilled in the art will understand that other steps or sequences of steps can be used to implement the post-processing flow of the present invention.

[0128] from Figure 1 As can be seen in step 3, the original image is cropped based on the effective text region obtained through post-processing in step 2, resulting in low-resolution text sub-images corresponding to the effective text regions. It's important to note that the "low" resolution of these text sub-images is a relative concept compared to the original image; that is, the pixel density of the text sub-images remains unchanged after cropping to meet the basic requirements of text recognition. After cropping, the resolution of these text sub-images is significantly reduced, and the relative proportion of text is significantly increased. This magnifies the feature scale of each character to a range suitable for text recognition, avoiding the problem of small-sized text being unrecognizable in the original image. Furthermore, the layout differences between text regions in the original image no longer affect the accuracy of text recognition. On the other hand, it reduces background noise interference in the text sub-images, improving the efficiency and accuracy of text recognition. Simultaneously, the memory usage of individual and collective text sub-images after cropping is relatively lower, reducing the computational requirements for subsequent text recognition.

[0129] In an optional embodiment, the coordinates of the effective text region can be represented in the form of pixel coordinates or scale coordinates, and a preview image containing region annotation information is generated so that the user can fine-tune and confirm the partitioning results.

[0130] In step 4, text recognition, such as OCR technology, is performed on each of the obtained text sub-images. Optionally, before text recognition, these text sub-images are scaled to keep the text height of the input text recognition model within a stable range, thereby improving the accuracy of text recognition. Text height refers to the vertical dimension of the entire text region, i.e., the input text sub-image.

[0131] Because the original text sub-images vary significantly in size—for example, the title text area is larger while the footnote text area is smaller—if the cropped text sub-images are directly input into a text recognition model, the smaller text areas may lead to recognition errors due to insufficient character pixel height. Character pixel height refers to the height of a single character in pixels within the image; that is, the number of pixel rows occupied by the character after digitization.

[0132] Based on the height of the text subgraph Width of text subimage Target input height of text recognition model Maximum width limit of text recognition model Calculate scaling ratio ,satisfy

[0133]

[0134] At this point, we can obtain the scaling ratio that adjusts the text height to the optimal recognition height of the model. At the same time, by constraining it by the maximum width limit, we can avoid excessive stretching and generating an oversized image.

[0135] Based on this scaling ratio For the original text subgraph By scaling the image proportionally, we obtain the scaled text sub-image. and the scaled text subgraph Perform text recognition. Among them,

[0136]

[0137] Scaled text sub-image The height of the text is equal to or slightly less than the target input height of the text recognition model.

[0138] Through the scale normalization process described above, the text sub-images are adaptively scaled to a suitable recognition scale, which not only improves the character pixel height and text recognition accuracy in small text regions, but also reduces the input scale differences between different text sub-images and improves the stability of text recognition.

[0139] Furthermore, the obtained text recognition results can be optionally used in various application scenarios such as information extraction, content matching, and document review, and this invention does not limit this.

[0140] Figure 2 and Figure 3 The illustration schematically shows an embodiment of the original image to which the present invention is applicable, and a schematic diagram of the effective text region obtained by the post-processing method of the present invention.

[0141] like Figure 2As shown, this is a product packaging image with a low proportion of text, diverse font sizes and layouts, various graphic elements besides text, multiple superscript annotations, and densely packed small-font text areas. Traditional text recognition techniques are prone to false positives, and the small-font text is often illegible. If existing uniform window segmentation methods are used for preprocessing, the resulting sub-image will exhibit incomplete text, missing paragraph structures, and significant background interference, making it impossible to extract complete and accurate paragraph semantic information from the recognized text, thus negating the practical application of text recognition for this image. Similarly, in other similar high-resolution images, relatively small-font text and superscript symbols may also be unrecognizable or misrecognized due to their low absolute size.

[0142] Figure 3 The image shows the bounding boxes and serial numbers of the effective text regions obtained after the region-level text detection and post-processing of this invention. These effective text regions divide the text content in the original image into sections according to the page layout, ensuring that information within the same paragraph is completely and unbiasedly preserved, thus giving the subsequent text recognition results semantic continuity. Furthermore, for example, superscript symbols such as *, ^, and # in bounding box 1 are completely preserved and recognized along with the main text, so that based on the text recognition results, they can be associated with text content in bounding box 6 that also has these superscript symbols such as *, ^, and #.

[0143] This invention also discloses a text detection and recognition system for complex layout images, used to implement the text detection and recognition method described above. The text detection and recognition system may include a text detection module, a post-processing module, an image cropping module, and a text recognition module. The text detection module performs region-level text detection on the original image and generates multiple candidate text boxes containing text. The post-processing module performs post-processing on the multiple candidate text boxes based on the layout structure characteristics of the text, obtaining multiple effective text regions containing complete text information. The image cropping module crops the original image according to the effective text regions, obtaining low-resolution text sub-images corresponding to the effective text regions. These text sub-images have text feature scales suitable for text recognition. The text recognition module performs text recognition on each text sub-image and obtains the text recognition results.

[0144] Optionally, the text detection and recognition system may also include a preview module for representing the effective text region in pixel coordinates or proportional coordinates, and generating a preview image containing region annotation information. Further, the text detection and recognition system of the present invention can also be deployed as a whole in, for example, an image information matching system or an advertising copy review system, and applied to various work scenarios.

[0145] The technical scope of this invention is not limited to the contents of the above specification. Those skilled in the art can make various modifications and variations to the above embodiments without departing from the technical concept of this invention, and all such modifications and variations should fall within the scope of this invention.

Claims

1. A method for text detection and recognition of complex layout images, characterized in that, The method includes: Step 1: Perform text detection on the high-resolution original image and generate multiple candidate text boxes containing text; Step 2: Based on the layout structure characteristics of the text, post-process the multiple candidate text boxes to obtain multiple effective text regions; Step 3: Crop the original image according to the effective text region to obtain a text sub-image corresponding to the effective text region; Step 4: Perform text recognition on each of the multiple text sub-images to obtain the text recognition results; Step 2 includes, but is not limited to: Step 2-1: Perform non-maximum suppression processing on the candidate text boxes to eliminate overlapping duplicate text boxes; Step 2-2: Divide the candidate text boxes into subtitle text boxes and text text boxes, and merge the subtitle text boxes with the text text boxes that are closest to them; Steps 2-3: Determine the paragraph relationship between multiple candidate text boxes, and merge candidate text boxes belonging to the same paragraph region; Steps 2-4: Determine the nesting relationship between the multiple candidate text boxes and delete the candidate text boxes nested inside other candidate text boxes; Steps 2-5: Expand the borders of the candidate text boxes.

2. The text detection and recognition method as described in claim 1, characterized in that, In step 2-2: Set area threshold ratio coefficient Height threshold Width threshold When the area of ​​the candidate text box satisfy And the height of the candidate text box and width satisfy When the candidate text box is identified as a superscript text box, it is determined to be a superscript text box. It also classifies other candidate text boxes that do not meet the criteria as text boxes. ,in, The total width of the original image. The total height of the original image.

3. The text detection and recognition method as described in claim 1, characterized in that, In step 2-2: Determine the superscript text box With the text box Positional relationships, when satisfying At that time, the text box will be... The text box is identified as the superscript text box. The relevant text boxes, among which, , , The superscript text boxes are respectively The x-coordinate of the left boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. , , , The text boxes are respectively The x-coordinate of the left boundary, the x-coordinate of the right boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. This is the tolerance coefficient for horizontal position. To horizontally extend the tolerance factor, This is the vertical tolerance coefficient above. This is the vertical tolerance coefficient below. The total width of the original image. The total height of the original image. Select the text box that corresponds to the superscript text box in the relevant text box. The nearest text box ,satisfy The superscript text box The text box closest to the nearest text box The text boxes are merged, and the new merged text box is the superscript text box. The text box closest to the nearest text box The bounding rectangle.

4. The text detection and recognition method as described in claim 1, characterized in that, In steps 2-3: Set horizontal overlap threshold and line spacing threshold coefficient When the horizontal overlap ratio between the two candidate text boxes is... and line spacing satisfy and When the two candidate text boxes are determined to belong to the same paragraph region, The horizontal overlap width between the two candidate text boxes. The total width of the narrower text box in the two candidate text boxes. The total height of the original image.

5. The text detection and recognition method as described in claim 1, characterized in that, In steps 2-3: When the candidate text box With multiple candidate text boxes When the candidate text boxes are determined to belong to the same paragraph area, the candidate text boxes are... and the nearest candidate text box The text boxes are merged, and the resulting new text box is the candidate text box. The nearest candidate text box The bounding rectangle of , where, For the candidate text box With the candidate text box The line spacing between them The left boundary difference weighting coefficient. , The candidate text boxes are respectively , The left boundary x-coordinate.

6. The text detection and recognition method as described in claim 1, characterized in that, In steps 2-4: Set position tolerance coefficient When the candidate text box With the candidate text box satisfy When, determine the candidate text box Nested in the candidate text box Inside, and the candidate text box Delete, among which, , , , The candidate text boxes are respectively The x-coordinate of the left boundary, the x-coordinate of the right boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. , , , The candidate text boxes are respectively The x-coordinate of the left boundary, the x-coordinate of the right boundary, the y-coordinate of the upper boundary, and the y-coordinate of the lower boundary. This is the tolerance ratio coefficient. The total width of the original image. The total height of the original image.

7. The text detection and recognition method as described in claim 1, characterized in that, In steps 2-5: Horizontal expansion amount of border extension Vertical upward expansion Vertical downward expansion amount satisfy in, This is the horizontal expansion factor. This is the upward expansion coefficient. For the downward expansion coefficient, satisfying , The total width of the original image. The total height of the original image. The new text box after border expansion meets the requirements. in, , , , The x-coordinates of the left and right borders, the y-coordinates of the top and bottom borders of the original text box. , , , This defines the x-coordinate of the left boundary, x-coordinate of the right boundary, y-coordinate of the top boundary, and y-coordinate of the bottom boundary of the new text box.

8. The text detection and recognition method as described in claim 1, characterized in that, In step 4, before performing text recognition on the text sub-image: Calculate scaling ratio , in, The height of the text sub-image. The width of the text sub-image. The target input height for the text recognition model. This is the maximum width limit of the text recognition model. The text sub-image is scaled proportionally. in, For the text subgraph, This is a scaled-down text sub-image. For the scaled text sub-image Perform text recognition.

9. A text detection and recognition system for complex layout images, capable of implementing the text detection and recognition method as described in any one of claims 1-8, characterized in that, The text detection and recognition system includes: The text detection module is used to perform text detection on the original image and generate multiple candidate text boxes containing text. A post-processing module is used to perform post-processing on the candidate text boxes based on the layout structure characteristics of the text, and to obtain effective text areas. The post-processing includes: performing non-maximum suppression on the candidate text boxes to eliminate overlapping duplicate text boxes; distinguishing the candidate text boxes into subscript text boxes and regular text boxes, and merging the subscript text boxes with the nearest regular text box; determining the paragraph relationship between multiple candidate text boxes, and merging candidate text boxes belonging to the same paragraph region; determining the nesting relationship between multiple candidate text boxes, and deleting candidate text boxes nested within other candidate text boxes; and expanding the borders of the candidate text boxes. The image cropping module is used to crop the original image according to the effective text region and obtain a text sub-image corresponding to the effective text region; The text recognition module is used to perform text recognition on multiple text sub-images respectively and obtain text recognition results.

Images