Image recognition and reconstruction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses an image recognition and reconstruction method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring a plurality of pieces of bounding box information corresponding to a plurality of layout areas in a target image; under the condition that the target image of the image is of a transverse multi-column layout type, performing page structure processing on the plurality of layout areas of the image based on the information of the plurality of bounding boxes of the image, and generating separation lines for page block division; sorting the plurality of layout areas of the image according to the image separation lines to obtain a plurality of reading sequence identifiers corresponding to the plurality of layout areas of the image; and rearranging the plurality of layout areas of the image according to the plurality of reading sequence identifiers of the image, and generating a content reconstruction result corresponding to the target image of the image. According to the embodiment of the invention, the accuracy and integrity of extracting the structured content of the complex multi-column document image can be improved.

Description

Image recognition and reconstruction methods, devices, electronic equipment and storage media Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an image recognition and reconstruction method, apparatus, electronic device and storage medium. Background Technology

[0002] Optical Character Recognition (OCR) technology recognizes printed or handwritten characters in document images and converts them into machine-readable text. It is widely used in document digitization, information retrieval, invoice processing, and assisted reading. However, when faced with large-format and complex documents such as instruction manuals and technical manuals, current technologies still struggle to achieve high-precision structured information extraction. Summary of the Invention

[0003] This application provides an image recognition and reconstruction method, apparatus, electronic device, and storage medium, which can improve the accuracy and completeness of extracting structured content from complex multi-column document images. The above technical solution is as follows: In a first aspect, this application provides an image recognition and reconstruction method, including: acquiring multiple bounding box information corresponding to multiple layout regions in a target image; when the target image is a horizontal multi-column layout type, performing page structure processing on the multiple layout regions based on the multiple bounding box information to generate dividing lines for page block division; sorting the multiple layout regions according to the dividing lines to obtain multiple reading order identifiers corresponding to the multiple layout regions; rearranging the multiple layout regions according to the multiple reading order identifiers to generate content reconstruction results corresponding to the target image.

[0004] In one possible implementation, the above-mentioned acquisition of multiple bounding box information corresponding to multiple layout regions in the target image includes: acquiring image scale parameters of the target image, and matching the image scale parameters with multiple preset parameter ranges to obtain matching results; determining target input scale parameters based on the matching results; scaling the target image according to the target input size to obtain a scaled target image; and inputting the scaled target image into a preset layout detection model for layout recognition to obtain multiple bounding box information corresponding to the multiple layout regions.

[0005] In one possible implementation, determining the target input scale parameter based on the matching result includes: determining the target parameter interval in which the image scale parameter is located based on the matching result, wherein the target parameter interval is any one of the plurality of parameter intervals; obtaining a preset scale parameter corresponding to the target parameter interval; and determining the preset scale parameter corresponding to the target parameter interval as the target input scale parameter.

[0006] In one possible implementation, the dividing line includes a first dividing line generated at intervals in the horizontal direction of the page to vertically divide the target image into blocks. The step of performing page structure processing on the multiple layout regions based on the multiple bounding box information to generate the dividing line for page block division includes: mapping the multiple bounding box information corresponding to the multiple layout regions in the horizontal direction of the page to obtain a first set of horizontal projection intervals; determining a first horizontal interval distance between adjacent horizontal projection intervals in the first set of horizontal projection intervals; for each set of adjacent horizontal projection intervals, if the first horizontal interval distance is less than a first preset distance, merging the corresponding adjacent horizontal projection intervals, and using the merged set of horizontal projection intervals as a second set of horizontal projection intervals; and generating the dividing line for page block division based on the second set of vertical projection intervals.

[0007] In one possible implementation, generating the dividing line for page block division based on the second vertical projection interval set includes: determining a second horizontal interval distance between adjacent horizontal projection intervals in the second horizontal projection interval set; and generating a first dividing line in the interval area between corresponding adjacent horizontal projection intervals when the second horizontal interval distance is greater than a second preset distance.

[0008] In one possible implementation, the aforementioned dividing line includes a second dividing line generated at intervals in the vertical direction of the page to horizontally divide the target image into blocks. The process of generating dividing lines for page block division by performing page structure processing on the multiple layout regions based on the multiple bounding box information includes: mapping the multiple bounding box information corresponding to the multiple layout regions in the vertical direction of the page to obtain a first set of vertical projection intervals; determining a first vertical interval distance between adjacent vertical projection intervals in the first set of vertical projection intervals; for each set of adjacent vertical projection intervals, if the first vertical interval distance is less than a third preset distance, merging the corresponding adjacent vertical projection intervals, and using the merged set of vertical projection intervals as a second set of vertical projection intervals; determining a second vertical interval distance between adjacent vertical projection intervals in the second set of vertical projection intervals; and generating a second dividing line within the interval region between the corresponding adjacent vertical projection intervals if the second vertical interval distance is greater than a fourth preset distance.

[0009] In one possible implementation, the dividing line is used to divide the target image into multiple page region blocks. The process of sorting the multiple layout regions according to the dividing line to obtain multiple reading order identifiers corresponding to the multiple layout regions includes: dividing the target image into multiple layout regions according to the dividing line to obtain multiple page region blocks and a set of layout regions to be sorted corresponding to each page region block; determining the intra-regional order identifier corresponding to each layout region based on the bounding box information of each layout region in the set of layout regions to be sorted; sorting the page region blocks inter-block based on their spatial positional relationship in the target image to obtain a block-level order identifier corresponding to each page region block; and merging and sorting the multiple layout regions according to the intra-regional order identifier and the block-level order identifier to generate multiple reading order identifiers corresponding to the multiple layout regions.

[0010] In one possible implementation, the process of rearranging the multiple layout regions based on the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image includes: globally sorting the multiple layout regions according to the numerical values ​​of the multiple reading order identifiers to obtain a sorted layout region sequence; obtaining the pixel content of the bounding box information corresponding to each layout region in the layout region sequence in the target image; rearranging the pixel content corresponding to each layout region based on the layout region sequence to generate a rearranged layout region set; and performing structural organization processing on the rearranged layout region set according to preset page structure rules to generate the content reconstruction result corresponding to the target image.

[0011] Secondly, embodiments of this application provide an image recognition and reconstruction apparatus, comprising: an acquisition module, configured to acquire bounding box information corresponding to multiple layout regions in a target image; a processing module, configured to perform page structure processing on the multiple layout regions based on the bounding box information when the target image is a horizontal multi-column layout type, generating dividing lines for page block division; a sorting module, configured to sort the multiple layout regions according to the dividing lines, obtaining multiple reading order identifiers corresponding to the multiple layout regions; and a generation module, configured to rearrange the multiple layout regions according to the multiple reading order identifiers, generating content reconstruction results corresponding to the target image.

[0012] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory; wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps provided in the first aspect of embodiments of this application.

[0013] Fourthly, embodiments of this application provide a computer storage medium storing multiple instructions adapted for loading and executing the method steps provided in the first aspect of embodiments of this application by a processor.

[0014] Fifthly, embodiments of this application provide a computer program product containing instructions that, when run on a computer or processor, cause the computer or processor to execute the method provided in the first aspect of embodiments of this application.

[0015] In this embodiment, multiple bounding box information corresponding to multiple layout regions in the target image is obtained. When the target image is a horizontal multi-column layout, page structure processing is performed on the multiple layout regions based on the bounding box information to generate dividing lines for page block division. The multiple layout regions are then sorted according to the dividing lines to obtain multiple reading order identifiers corresponding to the multiple layout regions. The multiple layout regions are then rearranged according to the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image. Therefore, without relying on the original text semantics, the correct reading order of complex multi-column pages can be automatically restored based solely on the spatial structure information of the layout regions. The original chaotic and structurally discrete pages are reconstructed into clear and consistent page images, avoiding resolution limitations and order errors caused by direct end-to-end recognition. This improves the accuracy and completeness of structured information extraction, thereby enhancing the accuracy of layout reconstruction for complex documents such as manuals, newspapers, and journals, and the reliability of subsequent content recognition, display, and editing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 is an exemplary system architecture diagram of an image recognition and reconstruction method provided in an embodiment of this application; Figure 2 is a flowchart of an image recognition and reconstruction method provided in an embodiment of this application; Figure 3 is a schematic diagram of a layout region mapping processing method provided in an embodiment of this application; Figure 4 is a schematic diagram of a separator line generation result provided in an embodiment of this application; Figure 5 is a schematic diagram of a reading order identifier generation result provided in an embodiment of this application; Figure 6 is a schematic diagram of the overall processing flow of an image recognition and reconstruction system provided in an embodiment of this application; Figure 7 is a structural schematic diagram of an image recognition and reconstruction device provided in an embodiment of this application; Figure 8 is a structural schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] To make the features and advantages of the embodiments of this application more apparent and understandable, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the embodiments of this application.

[0019] In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims. Furthermore, in the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; the word "and / or" in the text is merely a description of the association relationship between related objects, indicating that three relationships can exist, for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. Additionally, in the description of the embodiments of this application, "multiple" refers to two or more.

[0020] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as implying or suggesting relative importance or implicitly indicating the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0021] Optical character recognition (OCR) technology identifies printed or handwritten characters in document images and converts them into machine-readable text. It is widely used in document digitization, information retrieval, invoice processing, and assisted reading. However, when faced with large-format, complex documents such as instruction manuals and technical manuals, current technologies still struggle to achieve high-precision structured information extraction.

[0022] Therefore, embodiments of this application provide an image recognition and reconstruction method, apparatus, electronic device, and storage medium to solve the technical problem of difficulty in achieving high-precision structured information extraction when dealing with documents such as manuals and technical manuals that are too large in size and have complex structures.

[0023] Please refer to Figure 1, which is an exemplary system architecture diagram of an image recognition and reconstruction method provided in an embodiment of this application.

[0024] As shown in Figure 1, the system architecture may include a terminal 101, a network 102, and a server 103. The network 102 serves as the medium for providing a communication link between the terminal 101 and the server 103. The network 102 may include various types of wired or wireless communication links, such as wired communication links including fiber optic cables, twisted-pair cables, or coaxial cables, and wireless communication links including Bluetooth, Wireless-Fidelity (Wi-Fi), or microwave communication links.

[0025] Terminal 101 can interact with server 103 via network 102 to receive messages from or send messages to server 103. Alternatively, terminal 101 can interact with server 103 via network 102 to receive messages or data sent to server 103 by other users. Terminal 101 can be hardware or software. When terminal 101 is hardware, it can be various electronic devices, including but not limited to tablet computers, laptops, and desktop computers. When terminal 101 is software, it can be installed in the aforementioned electronic devices and can be implemented as multiple software programs or software modules (e.g., to provide distributed services) or as a single software program or software module; no specific limitation is made here.

[0026] In this embodiment, terminal 101 can upload a target image to server 103. Server 103 obtains multiple bounding box information corresponding to multiple layout regions in the target image. If the target image is a horizontal multi-column layout, page structure processing is performed on the multiple layout regions based on the multiple bounding box information to generate separator lines for page block division. The multiple layout regions are sorted according to the separator lines to obtain multiple reading order identifiers corresponding to the multiple layout regions. The multiple layout regions are rearranged according to the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image.

[0027] Server 103 can be a server that provides various services. It should be noted that server 103 can be hardware or software. When server 103 is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. When server 103 is software, it can be implemented as multiple software programs or software modules (e.g., used to provide distributed services), or as a single software program or software module; no specific limitations are made here.

[0028] Alternatively, the system architecture may not include server 103. In other words, server 103 can be an optional device in this application embodiment, meaning the method provided in this application embodiment can be applied to a system structure that only includes terminal 101, and this application embodiment does not limit this. In this embodiment, terminal 101 obtains multiple bounding box information corresponding to multiple layout regions in the target image; when the target image is a horizontal multi-column layout type, page structure processing is performed on the multiple layout regions based on the multiple bounding box information to generate dividing lines for page block division; the multiple layout regions are sorted according to the dividing lines to obtain multiple reading order identifiers corresponding to the multiple layout regions; the multiple layout regions are rearranged according to the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image.

[0029] It should be understood that the number of terminals, networks, and servers in Figure 1 is only illustrative, and can be any number of terminals, networks, and servers depending on the implementation needs.

[0030] Please refer to Figure 2, which is a flowchart illustrating an image recognition and reconstruction method provided in an embodiment of this application. The execution entity in this embodiment can be a terminal executing the image recognition and reconstruction method, a processor within the terminal executing the image recognition and reconstruction method, or an image recognition and reconstruction service within the terminal executing the image recognition and reconstruction method. For ease of description, the following uses the processor within the terminal as an example to illustrate the specific execution process of the image recognition and reconstruction method.

[0031] Please refer to Figure 2, which is a flowchart illustrating an image recognition and reconstruction method provided in an embodiment of this application. As shown in Figure 2, the image recognition and reconstruction method may include at least: S210: obtaining information on multiple bounding boxes corresponding to multiple layout regions in a target image.

[0032] The target image can be a document page image to be subjected to content recognition and structure reconstruction. It can be derived from scanned copies, photographs, or rendered screenshots of Portable Document Format (PDF) pages, and can specifically include various layout elements such as text, images, and tables.

[0033] Optionally, the multiple layout regions can be page area units in the target image that have clear layout semantics, including but not limited to body text areas, title areas, image areas, table areas, etc.

[0034] Optionally, the layout region corresponds one-to-one with the bounding box information. The bounding box information corresponding to the layout region can be a rectangular box information that describes the spatial position of the layout region in the target image. Specifically, it can include the coordinates of the upper left corner, the coordinates of the lower right corner, or the width and height parameters, which are used to characterize the range of the layout region.

[0035] It is understandable that the target image may include text content, which can be recognizable character information in the target image, including printed or handwritten text, and is the main object of subsequent optical character recognition and reading order reconstruction.

[0036] Specifically, a pre-defined layout detection model can be used to perform layout detection processing on the target image and output multiple bounding box information corresponding to multiple layout regions in the target image. The layout detection model can be a deep learning-based layout analysis model used to identify various layout elements in the target image, such as text regions, title regions, image regions, and table regions.

[0037] S220: When the target image is a horizontal multi-column layout, the page structure of the multiple layout areas is processed based on the multiple bounding box information to generate dividing lines for page block division.

[0038] The horizontal multi-column layout type can be a layout type where the target image has two or more columns arranged in the horizontal direction of the page, such as the multi-column text structure in newspapers, product manuals, or periodicals. Optionally, a preset classification model can be used to classify the target image to determine whether the target image is a horizontal multi-column layout type.

[0039] Optionally, page structure processing of the above-mentioned multiple layout areas can refer to analyzing the spatial structure of the target image to determine whether there are structural intervals for block division, thereby determining whether to generate page dividing lines.

[0040] Optionally, the dividing line can be a virtual dividing line generated in the horizontal or vertical direction of the page, used to divide the target image into multiple page region blocks. The dividing line may include a first dividing line generated at intervals in the horizontal direction of the page to divide the target image into vertical blocks; further, the dividing line may also include a second dividing line generated at intervals in the vertical direction of the page to divide the target image into horizontal blocks.

[0041] It is understandable that for target images with a horizontal single-column layout, there are no two or more columns arranged side-by-side in the horizontal direction of the page, and the page structure is relatively simple. In this case, the page structure processing steps may not generate a first dividing line for vertical block division, or the number of first dividing lines may be set to zero; alternatively, a second dividing line for horizontal block division may also not be generated. Furthermore, for the horizontal single-column layout type, the target image as a whole can be regarded as a page area block, and the reading order identifier can be directly determined based on the bounding box information corresponding to the layout area. The content within the layout area is then rearranged and stitched according to the reading order identifier to generate the content reconstruction result corresponding to the target image.

[0042] S230: Sort the above-mentioned multiple layout areas according to the above-mentioned dividing lines to obtain multiple reading order identifiers corresponding to the above-mentioned multiple layout areas.

[0043] Each layout region corresponds one-to-one with a reading order identifier. The reading order identifier represents the sequential number assigned to the corresponding layout region, which is used to indicate the reading order of the layout region in the target image.

[0044] Specifically, for each page region, the spatial location information of the bounding boxes of each layout region can be input into a layout reader to perform order regression processing on the layout regions, obtaining the reading order identifiers corresponding to each layout region. The reading order model can be a deep learning-based reading order regression model. The input of the reading order model can include the spatial location information of the bounding boxes corresponding to each layout region. This bounding box spatial location information can include the coordinates of the top-left corner, bottom-right corner, width, height, center point coordinates, and the relative positional relationships between the bounding boxes. The output of the reading order model is the order prediction result corresponding to each layout region. This order prediction result can be represented as a reading order identifier or a ranking score corresponding to the reading order identifier. Optionally, the layout regions can be ranked according to the ranking scores output by the reading order model, and the ranking results can be mapped to discrete reading order identifiers to obtain the final reading order of multiple layout regions in the target image.

[0045] S240: The above-mentioned multiple layout regions are rearranged according to the above-mentioned multiple reading order identifiers to generate the content reconstruction result corresponding to the above-mentioned target image.

[0046] The content reconstruction result can be structured data generated by rearranging the layout areas in the original target image based on the reading order identifiers of the above-mentioned multiple layout areas. The content reconstruction result visually conforms to the natural reading order, and reconstructs multi-column, misaligned or complex layout pages into a page form with consistent order and clear structure.

[0047] Optionally, the content reconstruction result can be reconstructed images arranged in reading order, text sequences, structured tabular data, markup language formatted page description data, or any combination thereof.

[0048] Specifically, firstly, the multiple layout regions are sorted according to the aforementioned reading order identifiers to obtain a sequence of layout regions arranged according to reading order; then, each layout region is placed sequentially on a preset target canvas according to the aforementioned layout region sequence, and each layout region is cropped according to the content area corresponding to its original bounding box and copied to the corresponding position on the target canvas; during the placement process, the layout regions can be scaled proportionally, aligned in position, or have their spacing adjusted according to preset arrangement rules, so that the layout regions are arranged continuously in the content reconstruction result according to the reading order from top to bottom and from left to right; finally, the rearranged layout regions are combined into a complete page image, which serves as the content reconstruction result corresponding to the aforementioned target image.

[0049] In this embodiment, multiple bounding box information corresponding to multiple layout regions in the target image is obtained. When the target image is a horizontal multi-column layout, page structure processing is performed on the multiple layout regions based on the bounding box information to generate dividing lines for page block division. The multiple layout regions are then sorted according to the dividing lines to obtain multiple reading order identifiers corresponding to the multiple layout regions. The multiple layout regions are then rearranged according to the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image. Therefore, without relying on the original text semantics, the correct reading order of complex multi-column pages can be automatically restored based solely on the spatial structure information of the layout regions. The original chaotic and structurally discrete pages are reconstructed into clear and consistent page images, avoiding resolution limitations and order errors caused by direct end-to-end recognition. This improves the accuracy and completeness of structured information extraction, thereby enhancing the accuracy of layout reconstruction for complex documents such as manuals, newspapers, and journals, and the reliability of subsequent content recognition, display, and editing.

[0050] In one embodiment, in the above S210, obtaining the bounding box information corresponding to multiple layout regions in the target image may include the following steps: S211: obtaining the image scale parameters of the target image, and matching the image scale parameters with multiple preset parameter ranges to obtain a matching result.

[0051] Among them, the image scale parameter can be the geometric scale information of the target image in pixel space, such as the height, width or long side size of the target image, which is used to characterize the resolution of the target image.

[0052] Optionally, multiple preset parameter intervals are used to segment the scale range of the target image. These intervals can be flexibly set according to actual application requirements. The number and boundaries of the parameter intervals are not limited to fixed values; for example, they can include a first parameter interval, a second parameter interval, and a third parameter interval. The first parameter interval corresponds to the judgment threshold for small-scale images, the second parameter interval corresponds to the judgment threshold for medium-scale images, and the third parameter interval corresponds to the judgment threshold for large-scale images. The first, second, and third parameter intervals increase sequentially in value, and each pair of intervals does not overlap. They continuously cover the scale range of the target image in a preset order, ensuring that the scale parameters of the target image can be uniquely mapped to the corresponding input scale parameters.

[0053] Optionally, the matching result is used to indicate the parameter range into which the scale parameter of the target image falls. For example, the matching result can be used to characterize any of the following: the image scale parameter is in the first parameter range; the image scale parameter is in the second parameter range; or the image scale parameter is in the third parameter range.

[0054] S212: Determine the target input scale parameters based on the above matching results.

[0055] The target input scale parameter is used to indicate the model input resolution used by the relevant layout detection model during inference.

[0056] Specifically, in S212, determining the target input scale parameter based on the matching result includes: determining the target parameter interval in which the image scale parameter is located based on the matching result, wherein the target parameter interval is any one of the plurality of parameter intervals; obtaining the preset scale parameter corresponding to the target parameter interval; and determining the preset scale parameter corresponding to the target parameter interval as the target input scale parameter.

[0057] The parameter intervals and preset scale parameters can be matched one-to-one. For example, the aforementioned parameter intervals include a first parameter interval, a second parameter interval, and a third parameter interval. If the matching result indicates that the image scale parameter falls within the first parameter interval, the first preset parameter is determined as the target input scale parameter; if the matching result indicates that the image scale parameter falls within the second parameter interval, the second preset parameter is determined as the target input scale parameter; and if the matching result indicates that the image scale parameter falls within the third parameter interval, the third preset parameter is determined as the target input scale parameter.

[0058] Therefore, by dividing the image scale parameters of the target image into different parameter ranges and mapping them to the corresponding preset parameters as target input scale parameters, the layout detection model can adaptively select the appropriate input scale according to the target images of different resolutions during the inference stage. This avoids the problems of insufficient features in small-sized images or excessive memory usage in large-sized images caused by using a fixed input size, and improves the accuracy and stability of layout detection in multi-resolution document scenarios.

[0059] S213: Scale the target image according to the target input size to obtain the scaled target image.

[0060] The scaling process includes, but is not limited to, the following: scaling the height and width of the target image proportionally or non-proportionally according to the target input scale parameters, so that the dimensions of the long or short side of the scaled target image meet the constraints of the aforementioned target input scale parameters, while maintaining the relative spatial relationships of the layout regions in the target image. This scaling process allows the target image to adapt to the input scale of the layout detection model without disrupting the original layout structure's proportions, thereby improving the stability and accuracy of subsequent layout detection.

[0061] Understandably, the resolution of the original target image may be much larger or smaller than the standard input scale of the layout detection model. Directly inputting the original target image into the model may lead to problems such as insufficient GPU memory, feature loss, or decreased inference accuracy. Therefore, the target image is scaled proportionally or non-proportionally according to the target input scale parameter, so that the scaled target image matches the input resolution requirements of the layout detection model in terms of spatial dimensions. Through this scaling process, the layout detection model can stably and accurately output the bounding box information corresponding to each layout region while ensuring that the proportion of the layout structure of the target image remains basically unchanged.

[0062] S214: Input the scaled target image into the preset layout detection model for layout recognition to obtain multiple bounding box information corresponding to the above multiple layout regions.

[0063] For example, the layout detection model described above can be a PaddlePaddleDocument Layout (PP-DocLayout) analysis model based on the PaddlePaddle framework, or a Layout You Only Look Once (Layout-YOLO) model based on the One-Time Object Detection Network, or other layout detection models in related technologies.

[0064] Furthermore, assuming that the image scale parameter of the target image is determined to be greater than the maximum parameter value of the third parameter range mentioned above, for example, the length and width of the target image both exceed 12000×12000 pixels, a preset coarse-to-fine two-stage processing method can be used for processing. That is, firstly, a lightweight layout detection model is used to quickly detect text blocks in the target image to obtain multiple coarse-grained block regions; then, the dynamic multi-segment input scale determination and scaling steps of S211–S213 mentioned above are performed on each block region, and each block region is input into the layout detection model mentioned above for fine layout recognition, thereby significantly reducing the computational complexity of ultra-large resolution target images while ensuring improved layout detection accuracy.

[0065] In this embodiment, through the aforementioned dynamic multi-segment input scale adjustment and coarse-to-fine two-stage layout detection mechanism, the input scale of the layout detection model can be adaptively determined according to the actual resolution of the target image, achieving high-precision layout recognition while maintaining the original layout structure ratio. In addition, when facing ultra-large resolution document images, coarse-grained segmentation is performed first, followed by fine-grained layout detection, effectively avoiding the problems of excessive memory usage and feature loss caused by direct overall inference. This significantly reduces computational complexity and improves inference stability, thereby improving the accuracy and robustness of layout detection in complex, multi-resolution document scenarios.

[0066] In one embodiment, when the content in the target image is rotated, tilted, or inverted, either entirely or partially, the target image can be rotated and corrected before executing S220. Specifically, the rotation angle of the target image or each layout region can be determined based on the bounding box information, text line direction information, or a preset direction detection model. When the rotation angle of the target image or a local layout region deviates from a preset direction range, the target image or the corresponding layout region is rotated and corrected to align the main text direction of the layout region with the horizontal or vertical direction of the page. Subsequently, the step of performing page structure processing on the multiple layout regions based on the multiple bounding box information to generate dividing lines for page block division is performed on the target image or layout region after rotation correction.

[0067] By rotating and correcting target images that are rotated, tilted, or inverted in whole or in part before performing page structure processing, the main text direction of the layout area is aligned with the horizontal or vertical direction of the page. This effectively reduces the problem of projection direction distortion caused by image shooting angle deviation or page rotation, thereby improving the accuracy of subsequent projection interval mapping, separator line generation, and reading order reconstruction, and enhancing the robustness of page structure analysis and image reconstruction in complex scenarios.

[0068] In one embodiment, in the above S220, the above page structure processing of the above multiple layout regions based on the above multiple bounding box information to generate the dividing line for page block division may include the following steps: S2211: Mapping the multiple bounding box information corresponding to the above multiple layout regions in the horizontal direction of the page to obtain a first horizontal direction projection interval set.

[0069] The horizontal direction of the page can be the horizontal direction of the page where the target image is located, corresponding to the X-axis direction in the image coordinate system, and is used to characterize the spatial distribution of each layout area in the left and right directions of the page.

[0070] Optionally, the first set of horizontal projection intervals may include at least one horizontal projection interval, each horizontal projection interval corresponding to the coverage area of ​​a layout area in the horizontal direction of the page, and the left and right boundaries of the horizontal projection intervals may be determined by the minimum and maximum horizontal coordinates of the corresponding bounding boxes.

[0071] For example, the minimum and maximum horizontal coordinates of the bounding box information of each layout area are extracted in the horizontal direction of the page, and this range is taken as a horizontal projection interval. The horizontal projection intervals corresponding to all layout areas constitute the first horizontal projection interval set.

[0072] S2212: Determine the first horizontal interval distance between adjacent horizontal projection intervals in the above first horizontal direction projection interval set.

[0073] Here, adjacent horizontal projection intervals can refer to two adjacent projection intervals arranged in order of their horizontal coordinate positions within the first set of horizontal projection intervals. The first horizontal interval distance can refer to the spacing between any set of adjacent horizontal projection intervals in the horizontal direction of the page, which can be calculated from the distance between the left boundary of the subsequent projection interval and the right boundary of the previous projection interval.

[0074] Specifically, the first horizontal projection interval set can be sorted according to the left boundary coordinates. For any two adjacent horizontal projection intervals after sorting, the distance between the left boundary of the latter horizontal projection interval and the right boundary of the former horizontal projection interval can be calculated to obtain the first horizontal interval distance.

[0075] S2213: For each group of adjacent horizontal projection intervals, if the first horizontal interval distance is less than the first preset distance, the corresponding adjacent horizontal projection intervals are merged, and the merged set of horizontal projection intervals is used as the second set of horizontal projection intervals.

[0076] The first preset distance can be used to determine whether adjacent horizontal projection intervals belong to the same column or the same structure. When the first horizontal interval distance is less than the first preset distance, it can be determined that there is no effective structural interval between the corresponding adjacent horizontal projection intervals.

[0077] The second set of horizontal projection intervals can refer to the new set of horizontal projection intervals obtained after performing the horizontal projection interval fusion operation. Specifically, it can include at least one new horizontal projection interval, where each new horizontal projection interval may be formed by fusing multiple first horizontal projection intervals.

[0078] It should be noted that fusion can be a process of merging adjacent projection intervals that are too close together horizontally to form an effective structural segmentation into a new, larger projection interval. Specifically, when the first horizontal interval between two or more adjacent horizontal projection intervals is detected to be less than a first preset distance, it indicates that these horizontal projection intervals belong to the same column or the same content block in the page structure, rather than independent page block boundaries. In this case, the corresponding multiple horizontal projection intervals are merged. The left boundary of the merged horizontal projection interval is taken as the minimum value of the left boundaries of the original horizontal projection intervals, and the right boundary is taken as the maximum value of the right boundaries of the original horizontal projection intervals, thus forming a new merged projection interval. Through this fusion process, pseudo-segmentation areas caused by character spacing jitter, small gaps in the page layout, or detection noise can be eliminated, avoiding the generation of too many invalid dividing lines and improving the stability and accuracy of subsequent page block division and reading order reconstruction.

[0079] S2214: Determine the second horizontal interval distance between adjacent horizontal projection intervals in the above set of second horizontal projection intervals.

[0080] The second horizontal interval distance refers to the horizontal spacing between any two adjacent projection intervals in the second horizontal projection interval set.

[0081] Specifically, the second horizontal projection interval set can be reordered according to the left boundary coordinates, and the horizontal spacing between any adjacent fused intervals can be calculated to obtain the second horizontal interval distance.

[0082] S2215: When the second horizontal interval distance is greater than the second preset distance, a first dividing line is generated in the interval area between the corresponding adjacent horizontal projection intervals.

[0083] Optionally, the second preset distance is used to determine whether there is an effective structural interval between adjacent fused projection intervals that can be used as a basis for page block division. When the second horizontal interval distance is greater than the second preset distance, it can be indicated that the interval position can be used as a page structure division position.

[0084] Specifically, when the second horizontal interval distance of a certain group of adjacent fusion intervals is greater than the second preset distance, a first dividing line is generated in the horizontal interval area of ​​the page between the two fusion intervals to divide the target image into different page region blocks.

[0085] In this embodiment, by projecting and mapping the layout area in the horizontal direction of the page, calculating the interval, and merging the data, and generating a first dividing line when the preset interval conditions are met, the true structural intervals between different columns or different content blocks in the page can be accurately identified. This effectively eliminates false dividing areas caused by character spacing jitter, detection noise, or small blank spaces, thereby avoiding the generation of redundant or incorrect page dividing lines, improving the accuracy and stability of page block division, and providing a more reliable structural foundation for subsequent reading order analysis and image reconstruction.

[0086] In a further embodiment, in the above S220, the above page structure processing of the above multiple layout regions based on the above multiple bounding box information to generate the dividing line for page block division may include the following steps: S2221: Mapping the multiple bounding box information corresponding to the above multiple layout regions in the vertical direction of the page to obtain a first vertical direction projection interval set.

[0087] The vertical direction of the page can be the vertical direction of the page where the target image is located, corresponding to the Y-axis direction in the image coordinate system, and is used to characterize the spatial distribution of each layout area in the page in the vertical direction.

[0088] Optionally, the first set of vertical projection intervals may include at least one vertical projection interval, each vertical projection interval corresponding to the coverage area of ​​a layout area in the vertical direction of the page, and the upper and lower boundaries of the vertical projection intervals may be determined by the maximum and minimum ordinates of the corresponding bounding boxes.

[0089] For example, the minimum and maximum ordinates of the bounding box information of each layout area are extracted in the vertical direction of the page, and this range is taken as a vertical projection interval. The vertical projection intervals corresponding to all layout areas constitute the first vertical projection interval set.

[0090] S2222: Determine the first vertical interval distance between adjacent vertical projection intervals in the above first vertical direction projection interval set.

[0091] Here, adjacent vertical projection intervals can refer to two adjacent projection intervals arranged in order of their vertical coordinate positions within the first set of vertical projection intervals. The first vertical interval distance can refer to the distance between any set of adjacent vertical projection intervals in the vertical direction of the page, which can be calculated from the distance between the upper boundary of the subsequent vertical projection interval and the lower boundary of the previous vertical projection interval.

[0092] Specifically, the first vertical projection interval set can be sorted according to the upper boundary coordinates. For any two adjacent vertical projection intervals after sorting, the distance between the upper boundary of the latter vertical projection interval and the lower boundary of the former vertical projection interval can be calculated to obtain the first vertical interval distance.

[0093] S2223: For each group of adjacent vertical projection intervals, if the first vertical interval distance is less than the third preset distance, the corresponding adjacent vertical projection intervals are merged, and the merged set of vertical projection intervals is used as the second set of vertical projection intervals.

[0094] The third preset distance can be used to determine whether adjacent vertical projection intervals belong to the same column or the same structure. When the first vertical interval distance is less than the third preset distance, it can be determined that there is no effective structural interval between the corresponding adjacent vertical projection intervals.

[0095] The second vertical projection interval set can refer to the new vertical projection interval set obtained after performing the vertical projection interval fusion operation. Specifically, it can include at least one new vertical projection interval, where each new vertical projection interval may be formed by fusing multiple first vertical projection intervals.

[0096] It should be noted that fusion can be a process of merging adjacent projection intervals that are too close together vertically to form an effective structural segmentation into a new, larger projection interval. Specifically, when the first vertical interval between two or more adjacent vertical projection intervals is detected to be less than a third preset distance, it indicates that these vertical projection intervals belong to the same column or the same content block in the page structure, rather than independent page block boundaries. In this case, the corresponding multiple vertical projection intervals are merged. The upper boundary of the merged vertical projection interval is taken as the maximum value of the upper boundaries of the original vertical projection intervals, and the lower boundary is taken as the minimum value of the lower boundaries of the original vertical projection intervals, thus forming a new merged projection interval. Through this fusion process, pseudo-segmentation areas caused by character spacing jitter, small gaps in the page layout, or detection noise can be eliminated, avoiding the generation of too many invalid dividing lines and improving the stability and accuracy of subsequent page block division and reading order reconstruction.

[0097] S2224: Generate dividing lines for page block division based on the above-mentioned second vertical direction projection interval set.

[0098] This application's embodiments, through the aforementioned mapping, spacing calculation, and fusion processing based on vertical projection intervals, can effectively eliminate pseudo-separation regions caused by small gaps, even in the presence of character jitter, uneven line spacing, or detection noise. Separation lines are generated only at locations where structural intervals truly exist, thereby avoiding excessive segmentation of page blocks. This improves the stability and accuracy of page block division.

[0099] Optionally, in S2224, generating the dividing line for page block division based on the above-mentioned second vertical projection interval set may include the following steps: determining the second vertical interval distance between adjacent vertical projection intervals in the above-mentioned second vertical projection interval set; and generating the second dividing line in the interval area between the corresponding adjacent vertical projection intervals when the above-mentioned second vertical interval distance is greater than the fourth preset distance.

[0100] The second vertical spacing distance refers to the vertical spacing between any two adjacent projection intervals in the second vertical projection interval set.

[0101] Specifically, the second vertical projection interval set can be reordered according to the upper boundary coordinates, and the vertical spacing between any adjacent fused intervals can be calculated to obtain the second vertical interval distance.

[0102] Optionally, the fourth preset distance is used to determine whether there is an effective structural interval between adjacent fused projection intervals that can be used as a basis for dividing page blocks. When the second vertical interval distance is greater than the fourth preset distance, it can be indicated that the interval position can be used as a page structure division position.

[0103] In this embodiment, by fusing and generating dividing lines on the projection intervals of the layout area in the vertical direction of the page, the true structural boundaries of different content blocks in the vertical direction of the page can be accurately identified. This avoids missegmentation caused by fluctuations in text line spacing, gaps inside tables, or layout detection noise. As a result, the robustness and accuracy of page area block division are effectively improved in complex layouts or multi-area mixed layout scenarios, providing a more reliable structural foundation for subsequent reading order analysis and image reconstruction.

[0104] Figure 3 is a schematic diagram of a layout area mapping processing method provided by an exemplary embodiment of this application. As shown in Figure 3, the target image is a product instruction manual image. The target image is a horizontal multi-column layout type, where the left column includes layout areas such as "Service Guide," "Product Parameters," "Product Appearance Diagram," and "Environmental Protection List," and the right column includes layout areas such as "Warranty Card" and "Repair Record." In this embodiment, the bounding box information corresponding to each layout area can first be obtained. For example, page elements such as "Service Guide," "Product Parameters," "Product Appearance Diagram," "Environmental Protection List," "Warranty Card," and "Repair Record" all correspond to independent rectangular bounding boxes. Subsequently, the bounding box information corresponding to the above multiple layout areas is mapped in the horizontal direction (X-axis direction) of the page, and the minimum and maximum horizontal coordinates of each bounding box in the horizontal direction of the page are extracted to form multiple horizontal projection intervals. The multiple intervals marked with "Horizontal Projection Interval" at the bottom of the figure are the mapping results. In addition, the bounding box information corresponding to the above multiple layout areas is mapped in the vertical direction (Y-axis direction) of the page, and the minimum and maximum vertical coordinates of each bounding box in the vertical direction of the page are extracted to form multiple vertical projection intervals.

[0105] Further, as shown in Figure 4, after completing the above-mentioned horizontal and vertical projection interval mapping processing, the horizontal projection interval set is subjected to interval analysis and fusion processing. When the interval distance between adjacent horizontal projection intervals is greater than a preset second preset distance, a first dividing line is generated between the corresponding adjacent projection intervals. The "first dividing line" shown in Figure 4 is a vertical dividing line located between the layout areas such as "Service Guide", "Product Parameters", "Product Appearance Diagram", and "Environmental Protection List" in the left column and the layout areas such as "Warranty Card" and "Repair Record" in the right column. This first dividing line divides the target image into two page area blocks, corresponding to the left column page area block and the right column page area block of the product manual, respectively, providing a structural basis for subsequent layout area sorting and reading order reconstruction within each page area block.

[0106] Furthermore, the aforementioned dividing line may also include a second dividing line (not shown in the figure). The second dividing line is used to generate at intervals in the vertical direction of the page to divide the target image into horizontal blocks. It is understood that in the product manual page shown in this embodiment, the main layout structure is a horizontal two-column arrangement. There is no structural blank band (e.g., a dividing white space spanning the page width) in the vertical direction that satisfies the aforementioned fourth preset distance. Therefore, the interval between the vertical projection intervals does not meet the preset condition for generating the second dividing line. Consequently, only the first dividing line is generated for the target image in Figure 4, and the second dividing line is not generated. In other words, in this example page, the effective block division of the left and right columns can be completed using the first dividing line. Subsequent reading order reconstruction can be performed independently within each page area block without additional horizontal block division.

[0107] By establishing projection intervals in the horizontal and vertical directions and generating separator lines based on preset interval conditions, the embodiments of this application can adaptively determine whether to generate a first separator line and / or a second separator line for multi-column pages, thereby achieving reliable block division of the page structure. In scenarios where there are only left and right columns but no top and bottom structural separation, redundant second separator lines can be avoided, reducing erroneous segmentation and sorting interference, thereby improving the accuracy and stability of page area block division and providing a more consistent structural basis for subsequent layout area sorting and image reconstruction.

[0108] In one embodiment, in the above S230, the sorting of the above multiple layout regions according to the above dividing line to obtain multiple reading order identifiers corresponding to the above multiple layout regions may include the following steps: S231: Divide the above multiple layout regions in the above target image according to the above dividing line to obtain multiple page region blocks and a set of layout regions to be sorted corresponding to each page region block in the above multiple page region blocks.

[0109] Among them, multiple page area blocks can refer to multiple page sub-regions obtained by dividing the target image according to the dividing lines. Each page area block corresponds to a continuous page area in the target image, such as the overall page block containing a column of text, an independent image area, or a table area.

[0110] Optionally, the set of layout areas to be sorted corresponding to each page area block can be a set of all layout areas located within the same page area block, such as multiple text paragraph areas, title areas, image areas, or table areas located in the same column. The above set is used to determine the reading order within the block later.

[0111] Specifically, the position parameters of the first dividing line and / or the second dividing line in the target image coordinate system can be obtained; and the boundary range of the page region block can be constructed based on the position parameters of the dividing line, dividing the target image into multiple page region blocks; then, for each page region block, it is determined whether the bounding box information of each layout region and the boundary range of the page region block meet the preset attribution rules; if the preset attribution rules are met, the corresponding layout region is added to the unsorted layout region set of the page region block.

[0112] Optionally, the preset attribution rules may include: the center point of the layout area boundary box falls within the boundary range of the page area block; or the overlapping area of ​​the layout area boundary box and the page area block boundary range is greater than the ratio of the layout area boundary box area to a preset ratio threshold.

[0113] S232: For the set of layout regions to be sorted corresponding to each of the above page area blocks, determine the intra-regional order identifier corresponding to each of the above layout regions based on the boundary box information of each layout region in the set of layout regions to be sorted.

[0114] The sequential identifier within a region can be a sequential number assigned to multiple layout regions within the same page region block based on the spatial relationship of their bounding boxes within the page region block. This number is used to characterize the reading order of each layout region within the page region block.

[0115] Specifically, firstly, the bounding box information of each layout region in the set of layout regions to be sorted is obtained, and the spatial position features of each layout region are extracted based on the bounding box information. The aforementioned spatial position features include at least the upper boundary coordinates, left boundary coordinates, center point coordinates, width or height of the bounding box. Subsequently, the reading order of the layout regions within the page area block can be determined based on the spatial position features, and an intra-regional sequence identifier is assigned to the layout regions.

[0116] Optionally, the order of reading can be determined by regression output using a reading order model, or by rule-based sorting. For example, rule-based sorting can generate the order identifier within the region according to the sorting rule of "top to bottom, left to right in the same row".

[0117] In an optional embodiment, when the page area block includes a table area or an image area, before determining the order identifier within the area, the layout areas within the same page area block can be subjected to image-text association fusion processing or table-text association fusion processing. Specifically, an image area can be merged with its corresponding image title area to generate a merged image area bounding box; a table area can be merged with its corresponding table title area and table description area to generate a merged table area bounding box. Further, before inputting the merged image area bounding box and the merged table area bounding box into the reading order model, the merged image area bounding box and the merged table area bounding box can be segmented. The segmentation process includes: according to the average bounding box height of the text layout areas within the same page area block, the merged image area bounding box and the merged table area bounding box can be segmented into multiple rows of sub-bounding boxes in the vertical direction of the page to construct an input bounding box consistent with the form of the text layout area. Subsequently, the multi-line sub-boundary boxes obtained from the above segmentation, along with the bounding boxes of the original text layout region, are input into the LayoutReader reading order model. Based on the spatial location information corresponding to each bounding box, order regression processing is performed, and the order identifier corresponding to each bounding box is output. Finally, the median or average of the order identifiers corresponding to each sub-boundary box within the same fusion region is taken as the regional order identifier for that fusion region.

[0118] S233: Based on the spatial positional relationship of each page region block in the target image, sort the page region blocks to obtain the block-level sequence identifier corresponding to each page region block.

[0119] The spatial relationship of each page region block in the target image can be the overall spatial arrangement of each page region block in the target image, such as the left-right position relationship between page region blocks in the horizontal direction of the page or the up-down position relationship in the vertical direction of the page.

[0120] Optionally, inter-block sorting can be used to sort multiple page region blocks to determine the overall reading order among different page region blocks. The resulting block-level order identifier can then be a sequence number assigned to each page region block, representing the overall reading order of each page region block in the target image.

[0121] Specifically, the block-level boundary range or block-level representative position parameter of each page region block in the target image is obtained. The block-level representative position parameter can be the center point coordinate or the upper left corner coordinate of the page region block. Then, the relative positional relationship between each page region block is determined based on the block-level representative position parameter. Then, the page region blocks are sorted according to the relative positional relationship, and a block-level order identifier is assigned to each page region block.

[0122] Optionally, the block sorting can be performed according to a preset reading direction. For example, the block order can be determined first from left to right in the horizontal direction of the page, and then the block order can be determined from top to bottom in the vertical direction of the page within the same block. Alternatively, when there is a horizontal block division, the blocks can be sorted first from top to bottom in the vertical direction of the page, and then sorted from left to right in the horizontal direction of the page.

[0123] S234: Based on the sequence identifier within the above-mentioned area and the sequence identifier at the block level, merge and sort the above-mentioned multiple layout areas to generate multiple reading sequence identifiers corresponding to the multiple layout areas.

[0124] Optionally, merging and sorting can refer to jointly processing the regional sequence identifier and the block-level sequence identifier to perform a global sorting of all layout regions, thereby obtaining a unified reading order for cross-page region blocks. The resulting reading order identifier corresponds one-to-one with the layout region, with each reading order identifier representing the final sequence number assigned to each layout region, used to characterize the reading order of that layout region within the entire target image.

[0125] Specifically, for each layout region, the block-level order identifier of the page region block to which the layout region belongs, and the intra-regional order identifier of the layout region within the page region block are obtained; a global sorting key value for the layout region is generated based on the block-level order identifier and the intra-regional order identifier; and all layout regions are globally sorted according to the global sorting key value to obtain multiple reading order identifiers corresponding to multiple layout regions.

[0126] Optionally, the global sorting key value can be obtained by combining the block-level order identifier and the intra-regional order identifier according to a preset weight or a preset concatenation rule, so as to ensure that the block-level order takes precedence over the intra-regional order, thereby achieving a unified reading order output for cross-page regional blocks.

[0127] In another embodiment, the bounding box information corresponding to multiple layout regions in the target image can be directly input into the reading order model for sequential regression processing, outputting reading order identifiers corresponding to multiple layout regions. This reading order model can be a deep learning-based layout reading order prediction model, used to predict the reading order of each layout region based on its spatial location information in the target image. Its input can include spatial features such as the top-left corner coordinates, bottom-right corner coordinates, center point coordinates, width, and height of the bounding box corresponding to each layout region. The output is the sequential prediction result or ranking score corresponding to each layout region, and the multiple reading order identifiers are generated based on the ranking score.

[0128] Figure 5 is a schematic diagram of a reading order identifier generation result provided by an exemplary embodiment of this application. As shown in Figure 5, the target image is divided into left and right page regions by a first dividing line. Multiple layout regions within the left page region are sequentially assigned reading order identifiers 1-13, and multiple layout regions within the right page region are sequentially assigned reading order identifiers 14-24. In this embodiment, the target image is first divided into left and right page regions according to the first dividing line, and the set of layout regions to be sorted within each page region is determined. Subsequently, for the layout regions within the left page region, according to the regional sorting rule of "top to bottom, left to right in the same row", or through regression output by a reading order model, regional order identifiers 1-13 are sequentially assigned to each layout region. Similarly, for the layout regions within the right page region, regional order identifiers 1-11 are sequentially assigned. Next, based on the spatial relationship between the two page area blocks in the horizontal direction of the page, the block-level order identifier of the left page area block is determined to be smaller than that of the right page area block. Accordingly, the intra-regional order identifiers within the left page area block are mapped to global reading order identifiers 1-13, and the intra-regional order identifiers within the right page area block are mapped to global reading order identifiers 14-24. Thus, the numbers shown in Figure 5 are the final reading order identifiers for the corresponding layout areas.

[0129] In this embodiment, by using page area block division, intra-block sorting, inter-block sorting, and merging sorting mechanisms, the target image can first be divided into multiple semantically consistent page area blocks using separator lines. Then, within each page area block, the intra-regional order identifier of the layout area is independently determined. Furthermore, based on the spatial positional relationship between page area blocks, the block-level order identifier is determined. Finally, the two are merged to generate a global reading order identifier, thereby avoiding cross-column missorting and cross-block skipping phenomena, and improving the accuracy and stability of layout area reading order reconstruction in complex layout scenarios.

[0130] In one embodiment, in the above S240, rearranging the above multiple layout regions according to the above multiple reading order identifiers to generate the content reconstruction result corresponding to the above target image may include the following steps: S241: According to the above multiple reading order identifiers, the above multiple layout regions are globally sorted according to the numerical value of the above reading order identifiers to obtain a sorted layout region sequence.

[0131] The sorted layout region sequence can be an ordered list, and the order of arrangement in this sequence is the order of each layout region in the target image in terms of actual reading meaning.

[0132] Specifically, each layout region can be paired with its corresponding reading order identifier to form a key-value pair, and then sorted in ascending order according to the reading order identifier to obtain a sequence of layout regions arranged in reading order.

[0133] S242: Obtain the pixel content of the bounding box information corresponding to each layout region in the above layout region sequence in the above target image.

[0134] Pixel content refers to the set of all pixel data within the rectangular area defined by the bounding box corresponding to a certain layout region in the target image. Specifically, for any layout region, pixel content consists of all pixels in the target image located within the coordinate range represented by the corresponding bounding box information, along with their pixel values.

[0135] Specifically, for the sorted layout region sequence, the bounding box information corresponding to each layout region is read, and the pixel content within the bounding box range is cropped from the original target image to serve as the original image content data of the corresponding layout region.

[0136] S243: Based on the above layout region sequence, rearrange the pixel content corresponding to each layout region to generate a rearranged layout region set.

[0137] Specifically, the pixel content corresponding to each of the above layout areas can be rearranged into a new page coordinate system to generate a set of rearranged layout areas. The new page coordinate system can refer to a pre-defined unified layout coordinate system in the content reconstruction result, used to represent the rearranged layout areas. This coordinate system can be a two-dimensional pixel coordinate system with the top-left corner of the content reconstruction result as its origin, used to determine the display position of each rearranged layout area in the content reconstruction result.

[0138] Optionally, each element in the rearranged set of layout regions may contain the pixel content of the layout region and its target position parameters in the content reconstruction result.

[0139] Specifically, in the new page coordinate system, a target display position is assigned to each layout area according to the sorted sequence of layout areas. For example, starting from the top of the page, the layout areas are arranged vertically according to their height, and the target position parameters of each layout area in the new page coordinate system are recorded, thereby generating a set of rearranged layout areas.

[0140] S244: Based on the preset page structure rules, perform structural organization processing on the above rearranged layout area set to generate the content reconstruction result corresponding to the above target image.

[0141] Among them, page structure rules can be used to constrain the arrangement and hierarchical relationship of each layout area in the reconstruction result. The aforementioned page structure rules may include at least one of the following: page layout rules, reading direction rules, area spacing rules, alignment rules, and hierarchical nesting rules.

[0142] For example, taking the content reconstruction result as image data, the page structure rules may include layout rules. The layout rules can be a set of rules used to constrain the arrangement of layout areas in the reconstructed image. For example, the layout rules may specifically be: arranged from top to bottom according to the reading order, automatically switching columns when there is insufficient vertical space, unifying the vertical or horizontal spacing between layout areas, and setting different default spacing or alignment methods for different types of areas such as titles, body text, images, and tables.

[0143] Optionally, the structural organization process may include performing sequential mapping, hierarchical merging, position alignment, spacing adjustment, rendering processing, formatting encapsulation, or structured output on the rearranged set of layout areas to form a content reconstruction result that conforms to the natural reading order.

[0144] For example, taking the content reconstruction result as image data, the structural organization processing can include position alignment and spacing adjustment. Fine-tuning the positions of each rearranged layout area can visually satisfy alignment rules (such as left alignment and center alignment) and preset area spacing requirements, avoiding area overlap or inconsistent spacing. Specifically, according to preset page layout rules, the rearranged layout area set is aligned and its spacing is adjusted, such as unifying the left margin, setting top and bottom spacing, and centering the title area. Then, the pixel content of each layout area is drawn into a new page coordinate system according to the adjusted target position parameters. Pixel-level rendering and compositing are performed on all layout areas to generate the reconstructed image corresponding to the target image, which serves as the content reconstruction result.

[0145] In another embodiment, the content reconstruction result is the text content reconstruction result in the target image, and the preset page structure rules may include reading direction rules. Structural organization processing of the rearranged layout area set may include: splicing the text content corresponding to each layout area according to the reading order based on the multiple reading order identifiers to generate a continuous text sequence; hierarchically organizing the text sequence according to the page structure rules, classifying the text content corresponding to different layout areas into title text, body text, image description text, or table description text, and constructing corresponding structured text fields; encapsulating the structured text fields into a target text data format, which may be plain text format, tabular text format, markup language format, or any combination thereof, thereby generating the content reconstruction result corresponding to the target image.

[0146] In this embodiment, by globally sorting each layout area according to the reading order identifier and obtaining its corresponding pixel content, and then rearranging the layout areas based on a new page coordinate system, this embodiment can automatically reconstruct the layout of multi-column, misaligned, or disordered document images without relying on the original page structure. This avoids the problem of lost layout information caused by sorting solely based on text, allowing non-text areas such as images, tables, and titles to maintain their original visual semantic relationship with the main text, achieving unified rearrangement across areas and columns. Furthermore, by using preset page structure rules to organize the layout areas, the content reconstruction result is more regular, continuous, and conforms to human reading habits in its overall structure, thus significantly improving the readability and structural consistency of complex document images.

[0147] The following describes this solution with reference to a specific system architecture diagram. Figure 6 is a schematic diagram of the overall processing flow of an image recognition and reconstruction system provided by an exemplary embodiment of this application. As shown in Figure 6, the system as a whole includes a layout detection module, a projection block segmentation module, and a layout sorting module. The layout detection module is used to perform layout detection on the input target image, and can output bounding box information for various layout regions, such as text region bounding boxes, table region bounding boxes, image region bounding boxes, and other region bounding boxes. The projection block segmentation module is used to perform horizontal and vertical projection operations based on the above bounding box information to generate page region blocks, realizing structured segmentation of the target image. The layout sorting module is used to perform image-text fusion, region segmentation, and reading order model inference on the page region blocks obtained by projection segmentation, output reading order identifiers corresponding to each layout region, and uniformly rearrange the order of the layout regions. Furthermore, the original layout region bounding boxes can be rearranged and mapped according to the reading order identifiers to output a set of target bounding boxes that conform to the actual reading order, thereby providing a reliable basis for subsequent image reconstruction. Through the above cascaded system architecture, a complete closed-loop processing flow from target image input to layout structure parsing, page block segmentation, and final reading order reconstruction is realized.

[0148] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0149] Based on the inventive concept of the above-mentioned image recognition and reconstruction method, as shown in FIG7, this application embodiment also provides an image recognition and reconstruction device 700 for implementing the image recognition and reconstruction method involved above. The image recognition and reconstruction device 700 includes: an acquisition module 710, used to acquire multiple bounding box information corresponding to multiple layout regions in a target image; a processing module 720, used to perform page structure processing on the multiple layout regions based on the multiple bounding box information when the target image is a horizontal multi-column layout type, and generate dividing lines for page block division; a sorting module 730, used to sort the multiple layout regions according to the dividing lines, and obtain multiple reading order identifiers corresponding to the multiple layout regions; and a generation module 740, used to rearrange the multiple layout regions according to the multiple reading order identifiers, and generate the content reconstruction result corresponding to the target image.

[0150] In one possible implementation, the acquisition module 710 is specifically used to: acquire the image scale parameters of the target image, and match the image scale parameters with a plurality of preset parameter ranges to obtain a matching result; determine the target input scale parameters based on the matching result; scale the target image according to the target input size to obtain a scaled target image; and input the scaled target image into a preset layout detection model for layout recognition to obtain multiple bounding box information corresponding to the plurality of layout regions.

[0151] In one possible implementation, the acquisition module 710 is specifically used to: determine the target parameter interval in which the image scale parameter is located based on the matching result, wherein the target parameter interval is any one of the plurality of parameter intervals; acquire the preset scale parameter corresponding to the target parameter interval; and determine the preset scale parameter corresponding to the target parameter interval as the target input scale parameter.

[0152] In one possible implementation, the aforementioned dividing line includes a first dividing line generated at intervals in the horizontal direction of the page to vertically divide the target image into blocks. The processing module 720 is specifically configured to: map the bounding box information corresponding to the multiple layout regions in the horizontal direction of the page to obtain a first set of horizontal projection intervals; determine a first horizontal interval distance between adjacent horizontal projection intervals in the first set of horizontal projection intervals; for each set of adjacent horizontal projection intervals, if the first horizontal interval distance is less than a first preset distance, merge the corresponding adjacent horizontal projection intervals and use the merged set of horizontal projection intervals as a second set of horizontal projection intervals; determine a second horizontal interval distance between adjacent horizontal projection intervals in the second set of horizontal projection intervals; and generate a first dividing line in the interval region between the corresponding adjacent horizontal projection intervals if the second horizontal interval distance is greater than a second preset distance.

[0153] In one possible implementation, the dividing line includes a second dividing line for generating at intervals in the vertical direction of the page to divide the target image into horizontal blocks. The processing module 720 is specifically used to: map the bounding box information corresponding to the multiple layout areas in the vertical direction of the page to obtain a first vertical projection interval set.

[0154] In one possible implementation, the processing module 720 is specifically used to: determine the first vertical interval distance between adjacent vertical projection intervals in the first vertical projection interval set; for each group of adjacent vertical projection intervals, if the first vertical interval distance is less than a third preset distance, merge the corresponding adjacent vertical projection intervals, and use the merged vertical projection interval set as the second vertical projection interval set; determine the second vertical interval distance between adjacent vertical projection intervals in the second vertical projection interval set; if the second vertical interval distance is greater than a fourth preset distance, generate a second dividing line in the interval area between the corresponding adjacent vertical projection intervals.

[0155] In one possible implementation, the dividing line is used to divide the target image into multiple page region blocks. The sorting module 730 is specifically used to: divide the multiple layout regions in the target image according to the dividing line to obtain multiple page region blocks and a set of layout regions to be sorted corresponding to each page region block; for the set of layout regions to be sorted corresponding to each page region block, determine the intra-regional order identifier corresponding to each layout region based on the bounding box information of each layout region in the set of layout regions to be sorted; sort the page region blocks inter-block based on the spatial positional relationship of each page region block in the target image to obtain the block-level order identifier corresponding to each page region block; and merge and sort the multiple layout regions according to the intra-regional order identifier and the block-level order identifier to generate multiple reading order identifiers corresponding to the multiple layout regions.

[0156] In one possible implementation, the generation module 740 is specifically used to: globally sort the multiple layout regions according to the numerical values ​​of the multiple reading order identifiers to obtain a sorted layout region sequence; obtain the pixel content of the bounding box information corresponding to each layout region in the layout region sequence in the target image; rearrange the pixel content corresponding to each layout region based on the layout region sequence to generate a rearranged layout region set; and perform structural organization processing on the rearranged layout region set according to preset page structure rules to generate the content reconstruction result corresponding to the target image.

[0157] The division of modules in the above-described image recognition and reconstruction device is for illustrative purposes only. In other embodiments, the image recognition and reconstruction device can be divided into different modules as needed to complete all or part of the functions of the above-described image recognition and reconstruction device. The implementation of each module in the image recognition and reconstruction device provided in this application embodiment can be in the form of a computer program. This computer program can run on a terminal or server. The program modules constituted by this computer program can be stored in the memory of the terminal or server. When the computer program is executed by a processor, it implements all or part of the steps of the image recognition and reconstruction method described in the embodiments of this application.

[0158] This application also provides an electronic device, which can be a server, and its internal structure is shown in Figure 8. The electronic device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. The processor executes a computer program to implement an image recognition and reconstruction method.

[0159] Those skilled in the art will understand that the structure shown in FIG8 is merely a block diagram of a portion of the structure related to the embodiments of this application, and does not constitute a limitation on the electronic device to which the embodiments of this application are applied. The specific electronic device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0160] In one possible implementation, a computer storage medium is provided that stores instructions, which, when executed on a computer or processor, cause the computer or processor to perform one or more steps in the above embodiments. If the constituent modules of the above-described electronic device are implemented as software functional units and sold or used as independent products, they can be stored in the aforementioned computer-readable storage medium.

[0161] In one possible implementation, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0162] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer storage medium or transmitted through the computer storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., Digital Versatile Discs (DVDs)), or semiconductor media (e.g., Solid State Disks (SSDs)).

[0163] It should be noted that the information (including but not limited to pixel information), data (including but not limited to data used for analysis, data stored, data displayed), and signals involved in the embodiments of this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the image information and pixel information involved in this application were obtained under full authorization.

[0164] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks. Unless otherwise specified, the technical features of this embodiment and its implementation can be combined arbitrarily.

[0165] The above-described embodiments are merely preferred embodiments of the present application and are not intended to limit the scope of the present application. Various modifications and improvements made by those skilled in the art to the technical solutions of the present application without departing from the design spirit of the present application should fall within the protection scope defined by the claims.

[0166] The foregoing has described specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims may be performed in a different order than that shown in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Claims

1. An image recognition and reconstruction method, characterized in that, include: Obtain information on multiple bounding boxes corresponding to multiple layout regions in the target image; When the target image is a horizontal multi-column layout, page structure processing is performed on the multiple layout regions based on the multiple bounding box information to generate separator lines for page block division; the multiple layout regions are sorted according to the separator lines to obtain multiple reading order identifiers corresponding to the multiple layout regions; the multiple layout regions are rearranged according to the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image.

2. The method as described in claim 1, characterized in that, The step of obtaining multiple bounding box information corresponding to multiple layout regions in the target image includes: obtaining image scale parameters of the target image and matching the image scale parameters with multiple preset parameter ranges to obtain matching results; determining target input scale parameters based on the matching results; scaling the target image according to the target input size to obtain a scaled target image; and inputting the scaled target image into a preset layout detection model for layout recognition to obtain multiple bounding box information corresponding to the multiple layout regions.

3. The method as described in claim 2, characterized in that, The step of determining the target input scale parameter based on the matching result includes: determining the target parameter interval in which the image scale parameter is located based on the matching result, wherein the target parameter interval is any one of the plurality of parameter intervals; obtaining the preset scale parameter corresponding to the target parameter interval; and determining the preset scale parameter corresponding to the target parameter interval as the target input scale parameter.

4. The method as described in claim 1, characterized in that, The dividing line includes a first dividing line generated at intervals in the horizontal direction of the page to vertically divide the target image into blocks. The step of processing the page structure of the multiple layout regions based on the multiple bounding box information to generate dividing lines for page block division includes: mapping the multiple bounding box information corresponding to the multiple layout regions in the horizontal direction of the page to obtain a first set of horizontal projection intervals; determining a first horizontal interval distance between adjacent horizontal projection intervals in the first set of horizontal projection intervals; for each set of adjacent horizontal projection intervals, if the first horizontal interval distance is less than a first preset distance, merging the corresponding adjacent horizontal projection intervals, and using the merged set of horizontal projection intervals as a second set of horizontal projection intervals; determining a second horizontal interval distance between adjacent horizontal projection intervals in the second set of horizontal projection intervals; and generating a first dividing line in the interval region between the corresponding adjacent horizontal projection intervals if the second horizontal interval distance is greater than a second preset distance.

5. The method as described in claim 4, characterized in that, The dividing lines include second dividing lines generated at intervals in the vertical direction of the page to horizontally divide the target image into blocks. The step of processing the page structure of the multiple layout regions based on the multiple bounding box information to generate dividing lines for page block division includes: mapping the multiple bounding box information corresponding to the multiple layout regions in the vertical direction of the page to obtain a first set of vertical projection intervals; determining a first vertical interval distance between adjacent vertical projection intervals in the first set of vertical projection intervals; for each group of adjacent vertical projection intervals, if the first vertical interval distance is less than a third preset distance, merging the corresponding adjacent vertical projection intervals, and using the merged set of vertical projection intervals as a second set of vertical projection intervals; and generating dividing lines for page block division based on the second set of vertical projection intervals.

6. The method as described in claim 5, characterized in that, The step of generating a dividing line for page block division based on the second vertical projection interval set includes: determining a second vertical interval distance between adjacent vertical projection intervals in the second vertical projection interval set; and generating a second dividing line in the interval area between corresponding adjacent vertical projection intervals when the second vertical interval distance is greater than a fourth preset distance.

7. The method as described in claim 1, characterized in that, The dividing line is used to divide the target image into multiple page region blocks. The process of sorting the multiple layout regions according to the dividing line to obtain multiple reading order identifiers corresponding to the multiple layout regions includes: dividing the multiple layout regions in the target image according to the dividing line to obtain multiple page region blocks and a set of layout regions to be sorted corresponding to each page region block; for each set of layout regions to be sorted corresponding to each page region block, determining the intra-regional order identifier corresponding to each layout region based on the bounding box information of each layout region in the set of layout regions to be sorted; sorting the page region blocks inter-block based on their spatial positional relationship in the target image to obtain the block-level order identifier corresponding to each page region block; and merging and sorting the multiple layout regions according to the intra-regional order identifier and the block-level order identifier to generate multiple reading order identifiers corresponding to the multiple layout regions.

8. The method as described in claim 1, characterized in that, The step of rearranging the multiple layout regions according to the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image includes: globally sorting the multiple layout regions according to the numerical value of the multiple reading order identifiers to obtain a sorted layout region sequence; obtaining the pixel content of the bounding box information corresponding to each layout region in the layout region sequence in the target image; rearranging the pixel content corresponding to each layout region based on the layout region sequence to generate a rearranged layout region set; and performing structural organization processing on the rearranged layout region set according to preset page structure rules to generate the content reconstruction result corresponding to the target image.

9. An image recognition and reconstruction device, characterized in that, include: The acquisition module is used to acquire information about multiple bounding boxes corresponding to multiple layout regions in the target image; The processing module is used to perform page structure processing on the multiple layout regions based on the multiple bounding box information when the target image is a horizontal multi-column layout type, and generate dividing lines for page block division. The sorting module is used to sort the multiple layout areas according to the dividing line to obtain multiple reading order identifiers corresponding to the multiple layout areas; The generation module is used to rearrange the multiple layout regions according to the multiple reading order identifiers to generate the content reconstruction result corresponding to the target image.

10. An electronic device, characterized in that, include: Processor and memory; The processor is connected to the memory; The memory is used to store executable program code; The processor runs a program corresponding to the executable program code stored in the memory to perform the method as described in any one of claims 1-8.

11. A computer storage medium, characterized in that, The computer storage medium stores a plurality of instructions adapted for loading by a processor and executing the method steps as claimed in any one of claims 1-8.

12. A computer program product containing instructions, characterized in that, When the computer program product is run on a computer or processor, it causes the computer or processor to perform the steps of the method as described in any one of claims 1-8.

Images