Document image reading order arrangement system and method based on convolutional neural network

Abstract

The disclosure provides a document image reading order arrangement method and system based on a convolutional neural network, relating to the technical field of image processing, comprising obtaining document image data to be sorted and preprocessing; inputting the preprocessed document image data into a global sequence positioning model of a deep convolutional neural network to obtain a global sequence positioning arrangement result; then performing heuristic local sorting on the text area inside the global sequence positioning arrangement result, calculating the image moment of the string instance in the text area to obtain the center point coordinates of the string instance, sorting the center point coordinates of the string instance according to a set heuristic rule, and outputting the final sequence arrangement result in combination with the text area reading order.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, specifically to a document image reading order arrangement system and method based on convolutional neural networks. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] In recent years, document image analysis and recognition have gradually become a research hotspot both domestically and internationally, and recognition systems for various types of document images are also increasing. However, most recognition systems often only output unordered recognition results, or rely solely on relatively simple heuristic rules to sort the results. When faced with document images with complex and varied layouts, these systems often disrupt the original reading order of the document images, reducing the actual reading experience. Therefore, many scholars have proposed sorting algorithms for the reading order of text regions.

[0004] Based on the technical means employed, sorting algorithms can be categorized into heuristic sorting algorithms and deep learning-based sorting algorithms. Early sorting methods were mostly heuristic. Besides directly sorting regions from top to bottom and left to right, some researchers attempted to determine the reading order of document images from different perspectives. For example, Jaekyu Ha et al. proposed the XY-cut sorting algorithm, Recursive XY Cut using Bounding Boxes of Connected Components. This algorithm utilizes connected component analysis and XY tree structures to hierarchically determine the reading order in document images with relatively simple layouts. Addressing the issue of XY-cut sorting algorithms being overly sensitive to parameter selection, J.-L. Meunier, in Optimized XY-cut for determining a page reading order, proposed treating the parameter selection of the XY-cut algorithm as an optimization problem, thus allowing for more flexible handling of different types of document images. Yucun Pan, addressing the L-shaped layout document images that XY methods cannot handle, proposed a top-down sorting algorithm in Document Layout Analysis and ReadingOrder Determination for a Reading Robot. This algorithm, combined with heuristic rules, can predict the reading order relatively accurately.

[0005] While heuristic sorting algorithms have made some progress, they often lack robustness when dealing with document images with varying layouts, making it difficult to provide accurate sorting results. Therefore, some researchers have attempted to treat the reading order problem as a sequence modeling problem, utilizing neural network models to achieve more robust order prediction. For example, in LayoutReader, Zilong Wang et al. improved the sequence generation module of the LayoutLM algorithm, enabling the network to predict the correct reading order based on the input sequence. Liangcheng Li et al. used graph neural networks and attention mechanisms in "An End-to-End OCR TextRe-organization Sequence Learning for Rich-Text Detail Image Comprehension" to predict the reading order of e-commerce document images.

[0006] However, the inventors discovered that existing technical solutions either rely on traditional algorithms with strong heuristics or on sequence modeling and structure mining based on Transformers and graph neural networks, but there is no good solution for using convolutional neural networks. Summary of the Invention

[0007] To address the aforementioned issues, this disclosure proposes a document image reading order arrangement system and method based on convolutional neural networks. It utilizes global sequence localization and heuristic local sorting of deep convolutional neural networks to provide the reading order of text regions from a global perspective, and to sort each text region locally.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions:

[0009] Methods for arranging the reading order of document images based on convolutional neural networks include:

[0010] Acquire and preprocess the image data of the documents to be sorted;

[0011] The preprocessed document image data is input into a deep convolutional neural network global sequential localization model to obtain the global sequential localization arrangement result; then, a heuristic local sorting is performed on the text region of the global sequential localization arrangement result, the image moments of string instances within the text region are calculated, the center point coordinates of the string instances are obtained, the center point coordinates of the string instances are sorted according to the set heuristic rules, and the final sequential arrangement result is output in combination with the reading order of the text region.

[0012] According to some embodiments, the present disclosure adopts the following technical solutions:

[0013] A document image reading order arrangement system based on convolutional neural networks includes:

[0014] The data preprocessing module is configured to: acquire image data of documents to be sorted and perform preprocessing;

[0015] The reading order arrangement module is configured to: input the preprocessed document image data into a deep convolutional neural network global order localization model to obtain a global order localization arrangement result; then perform heuristic local sorting within the text region of the global order localization arrangement result, calculate the image moments of string instances within the text region, obtain the center point coordinates of the string instances, sort the center point coordinates of the string instances according to a set heuristic rule, and output the final order arrangement result in combination with the reading order of the text region.

[0016] According to some embodiments, the present disclosure adopts the following technical solutions:

[0017] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the document image reading order arrangement method based on a convolutional neural network.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the document image reading order arrangement method based on convolutional neural networks.

[0020] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0021] This disclosure sets up two models: a global sequential localization model using a deep convolutional neural network and a heuristic local sorting model. The global model determines the reading order of text regions, while the local model sorts the text instances within each region.

[0022] This disclosure utilizes a more robust deep convolutional neural network for global sequential localization, enabling it to handle document images with different layouts. Furthermore, it employs a more rule-based heuristic sorting algorithm to ensure the flexibility of string instance sorting within text regions. Attached Figure Description

[0023] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0024] Figure 1 This is a schematic diagram of the document image reading order arrangement model architecture based on convolutional neural networks in an embodiment of this disclosure;

[0025] Figure 2 This is a flowchart of the document image reading order arrangement method based on convolutional neural networks in this embodiment of the present disclosure;

[0026] Figure 3 This is a diagram of the U-shaped neural network model structure in the embodiments of this disclosure;

[0027] Figure 4 This is a flowchart illustrating the training process of the multi-U-shaped neural network model in this embodiment of the present disclosure;

[0028] Figure 5 (a) and Figure 5 (c) in the figure represents the input image rendered from the annotation file and the actual annotation image;

[0029] Figure 5 (b) in the figure is the prediction result diagram of the global sequential localization output of the deep convolutional neural network in the embodiment of this disclosure.

[0030] Figure 6 This is a schematic diagram of heuristic local sorting in an embodiment of this disclosure.

[0031] in, Figure 6 (a) shows the result of sorting the horizontal string instances from right to left. Figure 6 (b) shows the result of sorting the vertical string instances from top to bottom. Figure 6 (c) Figure 6 (d) in the diagram shows the heuristic local sorting results for a mix of horizontal and vertical rows. Detailed implementation method:

[0032] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0033] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0034] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0035] Terminology Explanation:

[0036] String instance region: The region where the string image is located.

[0037] Detector: An algorithmic model that can be used to locate string instances in a document image.

[0038] Morphological operations: Operations that change the shape of an image

[0039] Connected component analysis: Identify and label the connected regions in the image.

[0040] Text region: A region containing multiple string instances obtained through morphological operations and connected component analysis, based on the string instance region.

[0041] Normalization: Makes the values ​​of data strictly within the range [0,1].

[0042] Convolutional Neural Networks (CNNs) are a type of feedforward neural network that includes convolutional computations and has a deep structure.

[0043] Fully convolutional networks: These are a type of convolutional neural network that does not contain fully connected layers.

[0044] U-shaped neural network: A type of fully convolutional neural network that has proven to be an efficient network architecture. This network consists of a nearly symmetrical encoder and decoder, exhibiting an overall U-shaped structure. Its input can be a three-channel color image, and its output can be a single-channel grayscale image.

[0045] Network parameters: numerical values ​​used for calculations in the network.

[0046] Hyperparameters: Parameters in a network that cannot be obtained through training and usually need to be set manually.

[0047] True annotation: Data with little or no errors, usually generated by manual annotation.

[0048] Loss function: A function used to calculate the error between the actual output and the true annotation.

[0049] Optimizer: A tool that adjusts network parameters based on errors.

[0050] Example 1

[0051] One embodiment of this disclosure provides a method for arranging the reading order of document images based on a convolutional neural network, including:

[0052] Step 1: Obtain the image data of the documents to be sorted and perform preprocessing;

[0053] Step 2: Input the preprocessed document image data into the deep convolutional neural network global sequential localization model to obtain the global sequential localization arrangement result; then perform heuristic local sorting within the text region of the global sequential localization arrangement result, calculate the image moments of string instances within the text region, obtain the center point coordinates of the string instances, sort the center point coordinates of the string instances according to the set heuristic rules, and output the final sequential arrangement result in combination with the reading order of the text region.

[0054] As one example, the preprocessing process is as follows: read the document image data JSON file, render the input image and the ground truth annotation based on the string instance location information contained in the JSON file, normalize and appropriately scale the rendered image to obtain an image pair that matches the convolutional neural network model.

[0055] The annotation file is read, and the input image and ground truth annotations are rendered based on the string instance location information contained in the file. In the ground truth annotations, the pixel value of the i-th region is i+1, the background pixel value is 0, and the pixel value of all string instances in the input image is 1. Subsequently, the rendered image is normalized and appropriately scaled to obtain an image pair that matches the network model. The pixel values ​​of the string instances in the ground truth annotations depend on the actual reading order of the strings.

[0056] The process of inputting preprocessed document image data into a deep convolutional neural network global sequential localization model to obtain the global sequential localization results is as follows: the preprocessed rendered image is fed into the trained deep convolutional neural network global sequential localization model to obtain the output image. Through morphological operations and connected component analysis techniques, the pixel mean of the text region is obtained, and the reading order of the text region is obtained based on the mean value.

[0057] As one embodiment, the step of obtaining the average pixel value of the text region through morphological operations and connected component analysis, and determining the reading order of the text region based on the average value, specifically includes: performing morphological dilation and erosion operations on the string region to obtain the text region outline, obtaining the average pixel value of the text region based on the outline, and determining the reading order of the text region based on the average value.

[0058] As one implementation, the image moments of string instances within the text region are calculated to obtain the coordinates of the center points of the string instances; the center point coordinates of the string instances are sorted according to a set heuristic rule, and the final sequential arrangement result is output in combination with the reading order of the text region, specifically including:

[0059] Traverse the text region, determine if there is a next text region, and obtain the first result.

[0060] If the first judgment result indicates that there is a next text region, then the string instances in that text region are heuristically sorted according to the center point coordinates.

[0061] If the first judgment result indicates that there is no next text region, then the text regions and string instances are integrated according to their order to determine the reading order of all string instances.

[0062] As one example, the construction process of the deep convolutional neural network global sequential localization model is as follows: the preprocessed rendered image is fed into the trained deep convolutional neural network global sequential localization model to obtain the output image; the average pixel value of the text region is obtained through morphological operations and connected component analysis, and the reading order of the text region is obtained according to the average value.

[0063] The heuristic local sorting model uses existing traditional algorithms. The construction process is as follows: calculate the image moments of string instances within the text region to obtain the coordinates of the center points of the string instances; sort the center point coordinates of the string instances according to the set heuristic rules; and output the final sorting result by combining the reading order of the text region.

[0064] The global sequential localization model of a deep convolutional neural network is a U-shaped neural network. This U-shaped neural network model utilizes skip connections to fuse shallow spatial information and deep semantic information. Through skip connections, shallow and deep feature maps are fused in a concatenated manner. Shallow information refers to image features extracted by the shallow neural network, such as color features, edge features, and corner features. As the network deepens, the obtained features acquire more accurate semantics, meaning that the features can better represent different individuals.

[0065] Deep convolutional neural networks (CNNs) consist of a U-shaped network, which comprises a contraction path and an expansion path, presenting an overall symmetrical U-shape. The contraction path consists of four typical "convolution-activation function-pooling" structures. The expansion path uses a "convolution-activation function-upsampling" structure to gradually reconstruct the low-resolution feature map to its original size, and concatenates rich shallow features onto a feature map of the same size through skip connections to supplement spatial information, achieve more accurate segmentation, and ultimately obtain an output that is exactly the same size as the input image.

[0066] U-shaped neural networks need to be trained before use. Figure 4 The training flowchart for the U-shaped neural network model is as follows, with the main steps being:

[0067] S401 loads labeled data: It reads the dataset from the storage, i.e., reads the JSON file, and renders it according to the labeled file to obtain the input image and the ground truth labels.

[0068] S402 Dataset Augmentation: This involves augmenting the read dataset, including cropping, skew transformation, and scaling. The augmentation operation requires simultaneous processing of both the input image and the ground truth annotations.

[0069] S403 Initializes the network model and sets hyperparameters: Defines the network model structure, including the selection of the optimizer, the definition of the loss function, etc., and sets the hyperparameters required by the network, such as the number of training epochs and the learning rate.

[0070] S404 Network Training: To begin training the network, images are fed into the network for forward propagation to obtain the actual output. The loss between the actual output and the labeled image is calculated using a loss function. An optimizer is used to continuously adjust the network parameters in order to minimize the error between the actual output and the labeled image in the next training round.

[0071] The loss function chosen is the L1 Smooth function, and the optimizer chosen is the Adam optimizer.

[0072] S405 Save Network Model: After the network training is completed, its network model and parameters are saved together to the pth model file.

[0073] Figure 5 (a) and Figure 5 In the example (c), the input image is rendered from the annotation file and compared with the actual annotation. Figure 5 Image (b) shows the prediction result output of the global sequential localization of a deep convolutional neural network. For this type of image, the reading order is from right to left and from top to bottom. The text instance in the upper right corner has the highest reading priority, and its corresponding pixel mean is the smallest; the text instance in the lower left corner has the lowest reading priority, and its corresponding pixel mean is the largest.

[0074] Figure 6 The diagram illustrates a heuristic local sorting algorithm. Solid boxes represent text regions, dashed boxes represent string instances within those regions, and numbers represent the heuristic local sorting results. Figure 6 (a) shows the result of sorting the horizontal string instances from right to left. Figure 6 (b) shows the result of sorting the vertical string instances from top to bottom. Figure 6 (c) Figure 6 (d) in the diagram shows the heuristic local sorting results of the mixed horizontal and vertical arrangement.

[0075] Example 2

[0076] One embodiment of this disclosure provides a document image reading order arrangement system based on a convolutional neural network, including:

[0077] The data preprocessing module is configured to: acquire image data of documents to be sorted and perform preprocessing;

[0078] The reading order arrangement module is configured to: input the preprocessed document image data into a deep convolutional neural network global order localization model to obtain a global order localization arrangement result; then perform heuristic local sorting within the text region of the global order localization arrangement result, calculate the image moments of string instances within the text region, obtain the center point coordinates of the string instances, sort the center point coordinates of the string instances according to a set heuristic rule, and output the final order arrangement result in combination with the reading order of the text region.

[0079] It also includes a training module, which is configured to train a U-shaped neural network model using the labeled images, generate a trained U-shaped neural network model, and save the network model parameters; the network model parameters include the network model structure, weight parameters, and hyperparameters, the network model structure includes the selection of the optimizer and the definition of the loss function, and the hyperparameters include the number of training epochs and the learning rate.

[0080] The present invention achieves an average relative distance (ARD) of 4.3 on the local validation set, which is a significant improvement over 8.1 achieved using only heuristic sorting algorithms.

[0081] like Figure 2 As shown, a method for arranging the reading order of document images based on a convolutional neural network includes:

[0082] S201: Data preprocessing;

[0083] S202: Obtain the output of the global sequential localization model of the deep convolutional neural network;

[0084] S203: Use the arrangement results obtained by global sequential localization of deep convolutional neural networks to perform heuristic local sorting within the text region;

[0085] The data preprocessing process involves reading a JSON file and rendering the input image and ground truth annotations based on the string instance location information contained in the file. In the ground truth annotations, the pixel value of the i-th region is i+1, the background pixel value is 0, and the pixel value of all string instances in the input image is 1. Subsequently, the rendered image is normalized and appropriately scaled to obtain an image pair that matches the network model.

[0086] The construction process of the deep convolutional neural network global sequential localization model is as follows: the preprocessed rendered image is fed into the trained deep convolutional neural network global sequential localization model to obtain the output image; through morphological operations and connected component analysis, the pixel mean of the text region is obtained, and the reading order of the text region is obtained according to the mean value.

[0087] The process of constructing the heuristic local sorting model is as follows: calculate the image moments of string instances within the text region to obtain the coordinates of the center point of the string instances; optionally, sort the coordinates of the center points of the string instances according to the set heuristic rules, such as from top to bottom or from left to right, and output the final sorting result in combination with the reading order of the text region.

[0088] The advantages of the above technical solution are that it can handle document images with different layouts by using a more robust deep convolutional neural network for global sequential localization, and it can ensure the flexibility of string instance sorting within local text regions by using a more rule-based heuristic sorting algorithm.

[0089] Example 3

[0090] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions. When these computer instructions are executed by a processor, they implement the steps of the method for analyzing and handling single-phase grounding faults in a low-current grounding system.

[0091] Example 4

[0092] One embodiment of this disclosure provides an electronic device, including a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the steps of the method for analyzing and handling single-phase grounding faults in a low-current grounding system.

[0093] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. A method for arranging the reading order of document images based on convolutional neural networks, characterized in that, include: Acquire and preprocess the image data of the documents to be sorted; The preprocessed document image data is input into a deep convolutional neural network global sequential localization model to obtain the global sequential localization arrangement result; then, a heuristic local sorting is performed on the text region of the global sequential localization arrangement result, the image moments of string instances within the text region are calculated, the center point coordinates of the string instances are obtained, the center point coordinates of the string instances are sorted according to the set heuristic rules, and the final sequential arrangement result is output in combination with the reading order of the text region.

2. The document image reading order arrangement method based on convolutional neural networks as described in claim 1, characterized in that, The preprocessing process is as follows: read the document image data JSON file, render the input image and the real annotation based on the string instance location information contained in the JSON file, normalize and appropriately scale the rendered image to obtain an image pair that matches the convolutional neural network model.

3. The document image reading order arrangement method based on convolutional neural networks as described in claim 1, characterized in that, The process of inputting the preprocessed document image data into the deep convolutional neural network global sequential localization model to obtain the global sequential localization result is as follows: the preprocessed rendered image is fed into the trained deep convolutional neural network global sequential localization model to obtain the output result image. The average pixel value of the text region is obtained through morphological operations and connected component analysis techniques. The reading order of the text region is obtained based on the average value.

4. The document image reading order arrangement method based on convolutional neural networks as described in claim 2, characterized in that, The first in the real annotation The pixel value of each region The background pixel value is 0, and the pixel value of all string instances in the input image is 1.

5. The document image reading order arrangement method based on convolutional neural networks as described in claim 2, characterized in that, The pixel values ​​of string instances in the actual annotation depend on the actual reading order of the strings.

6. The document image reading order arrangement method based on convolutional neural networks as described in claim 1, characterized in that, The global sequential localization model of the deep convolutional neural network is a U-shaped neural network, which uses skip connections to fuse shallow spatial information and deep semantic information.

7. The document image reading order arrangement method based on convolutional neural networks as described in claim 3, characterized in that, The method of obtaining the average pixel value of a text region through morphological operations and connected component analysis, and determining the reading order of the text region based on the average value, specifically includes: performing morphological dilation and erosion operations on the string region to obtain the text region outline, obtaining the average pixel value of the text region based on the outline, and determining the reading order of the text region based on the average value.

8. A document image reading order arrangement system based on convolutional neural networks, characterized in that, include: The data preprocessing module is configured to: acquire image data of documents to be sorted and perform preprocessing; The reading order arrangement module is configured to: input the preprocessed document image data into a deep convolutional neural network global order localization model to obtain a global order localization arrangement result; then perform heuristic local sorting within the text region of the global order localization arrangement result, calculate the image moments of string instances within the text region, obtain the center point coordinates of the string instances, sort the center point coordinates of the string instances according to a set heuristic rule, and output the final order arrangement result in combination with the reading order of the text region.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the document image reading order arrangement method based on a convolutional neural network as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the document image reading order arrangement method based on a convolutional neural network as described in any one of claims 1-7.

Images