A method for extracting and correcting ticket images in natural scenes

Abstract

This invention relates to the field of image classification and segmentation, and more particularly to a method for extracting and correcting natural scene ticket images. The specific steps are as follows: Manually label the ticket type and text direction; perform image enhancement operations on the labeled data; construct a multi-label classification network for training; after obtaining the trained model, classify the ticket and correct the text direction; label the main region of the natural scene ticket image; after adding image enhancement operations to the labeled data, generate the mask labels needed for training; and use a salient region network (U...) 2 The model is trained using a pre-processing network to extract the main body region of a natural scene ticket image, and then post-processing methods are used to perform secondary correction on the ticket viewpoint and text angle. This invention effectively eliminates background interference in natural scenes and solves problems such as viewpoint changes and text tilting for different ticket types, facilitating subsequent scene text recognition and key information extraction tasks.

Description

Technical Field

[0001] This invention relates to the field of image classification and segmentation, and more specifically, to a method for extracting and correcting natural scene ticket images. Background Technology

[0002] Various types of receipts are ubiquitous in commercial settings, serving as primary vouchers for transactions, rent payments, and expense reimbursements. Receipt digitization primarily utilizes OCR and key information extraction technologies, first identifying the content of receipts captured in natural scenes and then extracting crucial information. With the deepening application of deep learning technology in computer vision and natural language processing, and the continuous development of OCR and key information extraction technologies, many new research achievements have emerged in academia, enabling the gradual implementation of receipt digitization projects in industry.

[0003] The informatization of invoices primarily focuses on two areas: scene text recognition and key information extraction. Compared to traditional optical character recognition (OCR), scene text recognition faces more complex challenges. Traditional OCR mostly processes scanned, fixed-image, and uniformly formatted invoices, achieving high accuracy through simple image processing. However, scene text recognition suffers from variations in text shape and orientation, background text interference, and changing perspectives during manual photography, significantly impacting accuracy due to the influence of natural scenes. Key information extraction is limited by OCR accuracy and the type of invoice. As the number of invoice types increases, it becomes necessary to extract corresponding key information based on each type.

[0004] Existing methods for correcting document rotation in natural scenes mainly fall into two categories: 1. Rotating the forward-facing image by multiple angles, then training a text angle classification model using image classification methods; 2. Extracting key points using neural network heatmaps, then uniformly correcting them to the same scale using image processing methods. The first type of method requires extensive angle classification and rotation work, without considering the influence of image perspective and the deformation of the document image itself. In the second type, the regression prediction accuracy based on document key points needs improvement. Furthermore, the practice of unifying all types of documents to the same scale makes subsequent image processing methods unsuitable for application scenarios with documents of various shapes, resulting in poor generalization. In addition, neither of these methods can handle background interference in natural scenes. When image text exists in the natural background, the accuracy of text detection and recognition decreases, and the extraneous text significantly interferes with the extraction of subsequent key information. Therefore, a method that can handle natural background interference, perspective changes, and text angle variations is of significant importance for subsequent OCR and key information extraction tasks. Summary of the Invention

[0005] To address the problems of existing technologies and meet the needs of key information extraction, this invention provides a method for extracting and correcting invoices in natural scenes. The aim is to accomplish different tasks using two deep learning methods: First, extracting the main body region of the invoice from an image taken in a natural scene, and using different post-processing methods according to different invoice types to remove background regions from the natural scene invoice image; second, correcting the viewing angle and text angle of the invoice. These two methods effectively solve the problem of background interference. The segmentation method based on salient region detection has high accuracy in pixel classification and can handle multiple invoice types in different scenes. Post-processing correction of the shooting perspective can improve the deformation of the invoice itself to a certain extent. To achieve the above tasks, the main steps of the method for extracting and correcting invoice images in natural scenes are as follows:

[0006] Step S01: Collect various ticket images taken in natural scenes, and manually classify them according to the type of ticket and the direction of the text on the ticket to obtain the type label and direction label of each ticket image;

[0007] Step S02: Preprocess the ticket image by scaling it to a fixed size and performing image enhancement operations such as random cutting, rotation, and cropping to increase the amount of training data and enhance the robustness of the data.

[0008] Step S03: Construct a deep neural network for multi-label classification to classify the ticket image by type and text direction. This neural network consists of a backbone network DenseNet-169 and two fully connected branches. DenseNet-169 is responsible for extracting features from the image, and each fully connected layer shares the extracted feature information and is responsible for different classification tasks.

[0009] Step S04: During the training phase, the cross-entropy loss function is used to measure the error between the prediction result output by the fully connected layer and the label. The loss is obtained by multiplying the error calculated by the two fully connected layers by different weight parameters and adding them together. Using different weight parameters can balance the training difficulty of different tasks. During backpropagation, the SGD optimizer is used to obtain the multi-label classification model with the lowest loss.

[0010] Step S05: Use the trained multi-label classification model to classify the ticket image to be classified, obtain the ticket type and text direction of the image, and then perform rotation correction on the ticket image according to the obtained text direction.

[0011] Step S06: Collect various types of tickets photographed in natural scenes, and manually mark the main area of ​​the tickets using polygons on the collected images;

[0012] Step S07: Perform data cleaning on the labeled images to reduce data labeling errors. Randomly add image enhancement operations to the labeled images to obtain more training data. Convert the labeled coordinates into mask images as training labels.

[0013] Step S08: The deep learning salient region detection network U2-Net is used to process the ticket images in natural scenes and extract the main body regions of various types of tickets. This neural network consists of nested U-Net networks. Each layer of the U-Net outputs the feature map predicted by that layer, and these output feature maps are fused to obtain the probability map of the main body region of the ticket.

[0014] Step S09: The loss function of U2-Net during training consists of the sum of the cross-entropy loss of the feature map output by each layer and the mask of the labeled region. The Adam optimizer is used during training, and the model with the lowest loss is saved at the end.

[0015] Step S10: Input the ticket image from the natural scene into the trained model to obtain the probability map of the main body region of the ticket. Map the output probability map to the region in the original image to obtain the main body region of the ticket in the natural scene image.

[0016] Step S11: Based on the bill type obtained in step S05 and the main body area of ​​the bill obtained in step S10, the post-processing method is used to fine-tune the viewpoint and small angle of the text on the bill to obtain a main body area of ​​the bill with no background interference and the text direction is positive.

[0017] Furthermore, the deep neural network for multi-label classification described in step S03 consists of a backbone network DenseNet-169 and two fully connected branch layers, with the following main structure:

[0018] DenseNet-169 mainly consists of four dense blocks with close connections. There is a transition layer between two adjacent dense blocks. The purpose of the transition layer is to change the size of the feature map through convolution and pooling so as to connect two adjacent dense blocks.

[0019] Two fully connected layers are connected to the feature map output by DenseNet-169 to share the feature map information for different classification tasks.

[0020] Furthermore, the data augmentation and training label generation described in step S07 are specifically implemented as follows:

[0021] Step S07-1: Check whether the key points of the main body area of ​​the invoice are correct on the labeled data to form the initial dataset;

[0022] Step S07-2: Randomly select one or more image enhancement methods such as image scaling, histogram transformation, brightness change, Gaussian blur, orientation rotation, and salt and pepper noise to obtain more robust data and use it as part of the training dataset.

[0023] Step S07-3: The key points of the main body area of ​​the ticket, which are manually labeled in the training dataset, are divided and extracted from the labeled main body area of ​​the ticket and the natural background area, and a mask of the original image is generated as the training label.

[0024] Furthermore, the deep learning model U2-Net in step S08 consists of nested U-Net networks, with the following main structure:

[0025] The first-layer U-Net module uses a similar structure to FPN, consisting of Encode and Decode. The Encode module first performs downsampling, and the result of this downsampling is used as the corresponding input to the Decode module. The last downsampling step serves as the concatenation between the Encode and Decode modules. The Decode module primarily takes the downsampling result of each Encode step as input, and after feature fusion, it outputs the feature map of the current Decode step while simultaneously participating in the upsampling calculation of the current part, thus calculating the loss.

[0026] The second-layer embedded U-Net also uses an Encode and Decode structure. In order to obtain multi-scale features, the number of convolutional modules and upsampling modules in the Encode and Decode structures is changed layer by layer as the network layers increase.

[0027] Furthermore, the process described in step S10, where the image of the ticket in the natural scene is input into the trained model to obtain a probability map of the main body region of the ticket, is as follows:

[0028] Step S10-1: After the input image is scaled proportionally, preprocessing operations such as normalization are performed on the scaled probability map.

[0029] Step S10-2: The preprocessed data is processed through a network with an Encode and Decode structure to obtain the feature map output of each layer, and these feature maps are fused to obtain the final output feature map;

[0030] Step S10-3: After obtaining the maximum and minimum values ​​of the output feature map, the probability map of the main body region of the bill is obtained by normalizing the output feature map by dividing the difference between the output feature map and the minimum value by the difference between the maximum and minimum values.

[0031] Furthermore, the post-processing method described in step S11 for fine-tuning the perspective and text angle of the ticket refers to performing image post-processing operations based on the probability map output by U2-Net and the predicted ticket type. The specific steps are as follows:

[0032] Step S11-1: Restore the predicted probability map to the size of the original image, and perform boundary filling, grayscale conversion, and binarization; detect contours on the binary image, find the contour with the largest area, and calculate its minimum bounding rectangle.

[0033] Step S11-2: Based on the input ticket type, take the long or short side of the smallest bounding rectangle as the extracted parallel side; at the same time, after merging and filtering on the largest contour, take a group of line segments with longer lengths, and calculate the intersection point of the extracted parallel side and the contour line segment.

[0034] Step S11-3: Combine different intersection points and calculate the area of ​​the quadrilaterals separately. Exclude quadrilaterals whose shapes differ too much from rectangles. Select the quadrilateral with the largest intersection ratio of the smallest circumscribed rectangle and the quadrilateral. At the same time, a verification is required. If the area of ​​the quadrilateral is too small, the smallest circumscribed rectangle is selected as the final quadrilateral.

[0035] Step S11-4: Fill the original ticket image with the same boundary as described in step S11-1. Based on the quadrilateral obtained in step S11-3, perform a perspective transformation on the ticket image with the boundary filled, and finally obtain the main body area of ​​the ticket with the text and the viewpoint direction being positive.

[0036] This invention incorporates a series of data cleaning and image enhancement operations during the image preprocessing and training stages, expanding the existing dataset while improving its generalization ability. The constructed dataset exhibits better representativeness and higher accuracy in training text direction classification and ticket region extraction.

[0037] This invention uses the DenseNet network as the feature extraction network for classifying tickets and text directions. This network can effectively extract multi-scale and deep network information. While using this network for feature extraction, this invention also incorporates fully connected layers with multi-branch outputs and a branch weight mechanism to optimize the multi-branch outputs. This allows for a balance between accuracy and efficiency in both network selection and multi-branch optimization during the classification stage, resulting in better classification performance.

[0038] This invention incorporates post-processing into the extraction of the main body region of the invoice. This process considers the characteristics of invoices of different types, from different perspectives, and even with varying degrees of damage. Image-based methods are used to uniformly process the extracted main body region, finding the smallest quadrilateral that encompasses it, thus minimizing background interference. Furthermore, after perspective transformation, this smallest quadrilateral allows for adjustments to the image's viewing angle and the text angle, thereby enabling consistent processing of invoice images of different types, angles, and perspectives, resulting in excellent uniformity.

[0039] This invention provides a method for extracting and correcting ticket images in natural scenes, which can effectively handle ticket images in complex natural scenes. This invention can solve problems such as background interference, perspective shifts, and text tilt in various ticket images under different natural scenes, exhibiting strong robustness in processing capabilities and strong generalization across different ticket types.

[0040] This invention effectively eliminates interference from background, text direction, and perspective while obtaining document type information. In scene text recognition tasks, this invention can eliminate interference from natural scenes, especially in complex backgrounds, achieving improved text detection and recognition accuracy. In the post-processing task of OCR for key information extraction, better OCR accuracy and additional document type information both contribute to the extraction of structured information from the document. Therefore, this invention can be widely applied to upstream tasks of OCR and key information extraction in natural scenes, demonstrating strong usability. Attached Figure Description

[0041] Figure 1 A flowchart of the natural document extraction and correction method provided in the embodiments of the present invention.

[0042] Figure 2 This is an architecture diagram of a multi-label classification network provided in an embodiment of the present invention. (Multi-branch network architecture diagram)

[0043] Figure 3 This is a structural diagram of the dense blocks inside the multi-label classification network provided in an embodiment of the present invention.

[0044] Figure 4 This is an architecture diagram of the U2-Net document extraction network provided in an embodiment of the present invention.

[0045] Figure 5 This is a structural diagram of the residual U module inside the U2-Net document extraction network provided in an embodiment of the present invention. Detailed Implementation

[0046] To describe the objectives, basic ideas, and technical solutions of this invention, this section provides a detailed description of the invention in conjunction with the accompanying drawings and embodiments. It should be understood that these embodiments are merely intended to enable those skilled in the art to better understand and reproduce the invention, and are not intended to limit the invention in any way. This example represents one or a class of implementations, not all examples. Based on the examples of this invention, other examples obtained by those skilled in the art without creative effort fall within the scope of protection of this invention.

[0047] This invention implements a method for extracting and correcting invoice images in natural scenes. The basic idea is to first use a deep learning network, DenseNet, to classify the invoice type and text direction in the natural scene invoice image, and then perform initial correction of the text direction based on the classification results. Secondly, a salient region detection network, U2-Net, is used to extract the invoice region in the natural scene. Then, based on the classified invoice type, a post-processing method is used to find the smallest quadrilateral region on the invoice, performing a second correction of the viewpoint and text angle of the invoice. This invention can effectively handle the deformation and different shooting angles of different types of invoices in different scenes, improving the accuracy of invoice scene text recognition and key information extraction, which is of great significance for invoice informatization.

[0048] like Figure 1 As shown, this invention provides a method for extracting and correcting natural scene ticket images. The overall process includes the following steps:

[0049] Step 01: Collect various ticket images taken in natural scenes and manually classify them according to the type of ticket and the direction of the text on the ticket. The direction of the text on the ticket needs to be classified according to the overall orientation of the text within the ticket: the text orientation within ±45° of the positive angle is the first category, and then every 90° interval is divided into a new category. Finally, the orientation of the text on the ticket can be divided into four categories. The type of ticket can be classified according to actual needs. Finally, obtain the type label and orientation label for each ticket image.

[0050] Step 02: Preprocess the ticket image by scaling it to a fixed size and performing image enhancement operations such as shearing, rotation, and cropping. The shearing angle is randomly selected from [-0.1, 0.1], the rotation angle is randomly selected from [-10°, 10°], and the cropping area ratio is randomly selected from [0.7, 1). This is done to increase the amount of training data and enhance the robustness of the training data.

[0051] Step 03: Construct a deep neural network for multi-label classification to classify the type and text direction of the ticket image. This neural network consists of a backbone network DenseNet-169 and two fully connected layers. DenseNet-169 is responsible for extracting features from the image, and each fully connected layer shares the extracted feature information and is responsible for the ticket type classification task and the text direction classification task, respectively.

[0052] like Figure 2 As shown, the backbone network DenseNet-169 mainly consists of four dense blocks with tight connections. The internal structure of the dense blocks is as follows: Figure 3 As shown, the dense block contains multiple modules named "DenseNet-BC", with the structure: BN+ReLU+1*1Conv+BN+ReLU+3*3Conv. Figure 2 "DenseBlock 1" contains 6 DenseNet-BC modules, "Dense Block 2" contains 12 DenseNet-BC modules, and "DenseBlock 3" and "Dense Block 4" each contain 32 DenseNet-BC modules. There is a transition layer between two adjacent dense blocks, with the structure: BN+ReLU+1*1Conv+2*2 AvgPooling. Its function is to change the size of the feature map through convolution and pooling, thereby connecting two adjacent dense blocks.

[0053] After the last dense block, a Dropout layer with a random inactivation rate of 0.5 is connected, which resists overfitting and greatly reduces the network parameters.

[0054] The Dropout layer is followed by a global average pooling layer with a 7x7 pooling kernel, which transforms the two-dimensional feature map into a one-dimensional vector. The global pooling layer performs structural regularization on the entire network to prevent overfitting and directly assigns actual meaning to each channel.

[0055] Two fully connected layers are connected after the global pooling layer for different classification tasks. Both fully connected layers use sigmoid as the activation function.

[0056] Taking an input image of size 224*224*3 as an example, after passing through 7*7 Conv and 3*3 MaxPooling, it goes through four dense blocks and three transition layers, then through a Dropout layer and a global pooling layer, and is then fed into two fully connected layers, finally outputting two classification results. The dimensionality changes as follows: 224*224*3->112*112*64->57*57*64->57*57*256->29*29*128->29*29*512->15*15*256->15*15*1280->8*8*640->8*8*1664->8*8*1664->1*1*1664->1*1*(number of ticket types), 1*1*4.

[0057] In this step, "BN" refers to Batch Normalization, which can effectively avoid gradient vanishing and speed up model convergence; "ReLU" refers to Rectified Linear Unit, a commonly used activation function in deep neural networks; "Conv" refers to Convolution Layer, and "1*1Conv" means a convolution layer with a 1*1 kernel; "AvgPooling" refers to Average Pooling, and "2*2AvgPooling" means an average pooling layer with a 2*2 kernel.

[0058] Step 04: During the training phase, the cross-entropy loss function is used to measure the error between the prediction result output by the fully connected layer and the label. The loss is obtained by multiplying the error calculated by the two fully connected layers by different weight parameters and adding them together. Using different weight parameters can balance the training difficulty of different tasks. During backpropagation, the SGD optimizer is used to obtain the multi-label classification model with the lowest loss. The SGD optimizer has low requirements for gradients, so the training speed is very fast when applied to large datasets.

[0059] The cross-entropy loss function used in this step is as follows:

[0060]

[0061] Where N is the number of samples, M is the number of types, and y ic This indicates whether the i-th sample is the true label of the c-th class; if so, then y ic p is 1 if it is not 0 otherwise ic It is the probability that the i-th sample is predicted to be the c-th label.

[0062] The overall loss function is shown below:

[0063] Loss=α×L type+β×L angle

[0064] Where L type It is the cross-entropy loss function for type classification, L angle It is the cross-entropy loss function for directional classification. The total loss function is the weighted sum of these values. It is recommended that the value of α be 1.25 and the value of β be 1.0.

[0065] Step 05: Use the trained multi-label classification model to classify the ticket image to obtain the ticket type and text direction of the image, and then rotate and correct the ticket image according to the obtained text direction.

[0066] Step 06: Collect various types of receipts, including thermal receipts, electronic receipts, VAT invoices, train tickets, and ID cards, captured in natural scenes. On the collected images, manually mark the main areas of the receipts using polygons.

[0067] Step 07: Clean the labeled images to reduce labeling errors; randomly add image enhancement operations to the labeled images to obtain more training data; and convert the labeled images into mask images based on the coordinates of the labels to serve as training labels.

[0068] This section includes the following sub-steps:

[0069] Step 07-1: The collected tickets themselves may be deformed. Manually clean the labeled polygons, and especially check whether the deformed tickets are labeled correctly to reduce labeling errors.

[0070] Step 07-2: Randomly select one or more image enhancement methods such as image scaling, histogram transformation, brightness change, Gaussian blur, orientation rotation, and salt-and-pepper noise to add to the labeled image. First, the number of image operations to be added is selected randomly between [1, 3]. Then, the type of image enhancement operation to be added is also randomly selected. Among them, the orientation rotation angle range is [-10°, 10°]. When the selected image enhancement operation is orientation rotation, the inverse matrix of the rotation matrix needs to be calculated, and the coordinates of the labeled points are multiplied by the inverse matrix to obtain the coordinate mapping on the rotated image. The Gaussian blur convolution kernel size is randomly selected as 3 or 5, and the γ value of brightness change is randomly selected between [0.8, 1.8].

[0071] Step 07-3: Converting image enhancement data into training labels requires using manually annotated key points of the main body area of ​​the ticket. The annotated main body area and background area are then grayscaled and binarized to obtain a mask image of the annotated image, which serves as the training label.

[0072] Step 08: The U2-Net deep learning salient region detection network is used to process ticket images in natural scenes and extract the main body regions of various types of tickets. This neural network consists of nested U-Net networks. Each layer of the U-Net outputs the feature map predicted by that layer, and these output feature maps are fused to obtain the probability map of the main body region of the ticket.

[0073] Depend on Figure 4 As shown, the U2-Net network adopts an Encode-Decode architecture. The Encode module consists of a six-layer downsampling network. In En_1-4, the number of U-Net downsampling and upsampling modules gradually decreases according to the model depth. En_5-6 layers use convolution + dilated convolution to preserve deep image features. The Decode module consists of a five-layer upsampling network. De_1-5 has the same network structure as En_1-5 layers, and En_6 connects the Encode and Decode modules. The result of Encode downsampling is used as the input to Decode, and features are fused with the downsampling result to obtain the output feature map of this encode-decode layer. Finally, the feature maps of the six layers of encode-decode are fused to obtain the final output probability map.

[0074] like Figure 5 As shown, the residual U-module of U-Net also uses an Encode and Decode structure. The Encode module uses convolutional modules to achieve downsampling, and then uses dilated convolutions to connect to the upsampled Decode module. The layers corresponding to Encode and Decode also use the results of the Encode module as input to Decode, and finally, the feature maps are summed to obtain the output of the residual U-module of U-Net.

[0075] Step 09: The loss function of U2-Net during training consists of two parts. The first part is the sum of the cross-entropy loss of the feature map output by each layer and the labeled region mask. The second part is the cross-entropy loss of the probability map obtained after fusing the feature maps of each layer. Adam is used for optimization during training, and the model with the lowest loss is saved at the end.

[0076] The cross-entropy loss function used in this step is as follows:

[0077]

[0078] Where (h,w) represents the height and width of the image, and (x,y) represents the position of a point in the image. GT (x,y) PR (x,y) These represent the result marked at that point and the prediction result, respectively.

[0079] The loss for each layer's feature map is calculated as follows:

[0080]

[0081] Where M represents the number of layers in module U, and here M = 6, f m lossw represents the cross-entropy loss of the feature map output by this layer, F fuse loss fuse This represents the cross-entropy loss of the probability map obtained after feature layer fusion.

[0082] The Adam optimizer is characterized by its simple implementation, high computational efficiency, low memory requirements, and parameter updates that are unaffected by gradient scaling transformations. This makes the hyperparameters highly interpretable, allowing the model to achieve good results with little or no adjustment.

[0083] Step 10: Input the image of the ticket from the natural scene into the trained model to obtain a probability map of the main body region of the ticket; map the output probability map back to the region in the original image to obtain the main body region map of the ticket; detailed implementation steps are as follows:

[0084] Step 10-1: Read in the image and convert it to RGB color space. Scale the shorter side of the image to 320 and scale the longer side proportionally. Then, use normalization to convert the image pixel information to the range [0,1] to obtain preprocessed data for subsequent neural network calculations.

[0085] Step 10-2: The preprocessed data is input into the U2-Net network structure. Downsampling is performed in the Encode module. The initial input is a 3D image. The dimension changes as En_1-6: 3->64->128->256->512->512->512. The dimension of each layer corresponds to the De_5-1 part. That is, the dimension changes as De_5-1 changes as De_5-1: 512->256->128->64. Finally, the features output by the Decode module are fused and sigmoid is used to calculate the feature map of the final prediction result.

[0086] Step 10-3: After normalizing the output feature map, the predicted probability map is obtained. First, the maximum and minimum values ​​of the feature map are calculated. Then, the minimum value is subtracted from each feature information and the difference between the maximum and minimum values ​​is divided to obtain the normalized probability map.

[0087] Step 11: Based on the document type obtained in step S05 and the main body area of ​​the document obtained in step S10, use post-processing methods to fine-tune the viewing angle and small angle of the text on the document to obtain a main body area of ​​the document with no background interference and the text direction is positive. The detailed post-processing steps are as follows:

[0088] Step 11-1: After obtaining the normalized probability map predicted by the U2-Net model, multiply each value of the probability map by 255 and restore the probability map to the size of the original ticket image. Then, convert the color space of the image from RGB to BGR mode to obtain the input image for post-processing. Fill the obtained image with black borders of fixed length on the top, bottom, left, and right sides. The purpose of filling the borders is to prevent the minimum bounding rectangle obtained in subsequent steps from exceeding the boundary. Next, convert the image to a grayscale image, and then use the OTSU algorithm to obtain the binarization segmentation threshold of the grayscale image. Binarize the grayscale image according to the threshold to obtain a binary image. Perform contour detection on the binary image, retaining only the inflection point information of the contours. Then, among all the detected contours, find the contour with the largest area and calculate the minimum bounding rectangle of the largest contour.

[0089] Step 11-2: Based on the ticket type classification result in step S05, determine whether to take the two longest or two shortest parallel sides of the smallest bounding rectangle. If the ticket type is longer than wide, take the two shortest parallel sides; otherwise, take the two longest parallel sides. The purpose is to select the two sides parallel to the direction of the ticket text. This is because, due to the shooting angle, tickets photographed in natural scenes will undergo a certain degree of perspective distortion. The sides parallel to the direction of the ticket text remain basically parallel, while the other two sides are no longer parallel due to perspective. Considering this situation, retaining the two basically parallel sides during distortion correction ensures that the text will not be distorted. Excessive deformation; at the same time, on the extracted maximum contour, the Douglas-Peucker algorithm is first used to approximate the contour to obtain the vertex information of the approximate contour. Then, using the vertex information, points with corresponding interior angles close to 180° in the approximate contour are removed. Then, the length of each line segment formed by the retained points is calculated, and the first six longest line segments are retained. Here, six line segments are selected to find a quadrilateral that is more suitable for the document text area with large perspective changes. If the document image in the application scenario does not have a large perspective change, the number of retained line segments can be reduced as appropriate to speed up the processing, but it should not be less than four.

[0090] Step 11-3: Calculate the intersection points of the two parallel sides of the calculated minimum bounding rectangle with the six line segments after extending them into straight lines. Then calculate the distance between each intersection point and the four vertices of the minimum bounding rectangle. Each intersection point belongs to the region of the vertex closest to that intersection point, thus dividing the intersection points into four regions. Next, take one point from each of the four regions to form a quadrilateral. Calculate the intersection-union ratio (IUR) of the quadrilateral with the minimum bounding rectangle. Compare all combinations. If the difference between the two upper or two lower interior angles of the quadrilateral is greater than 10°, exclude this quadrilateral (because such a quadrilateral will cause significant distortion of the text in the original document image during later perspective transformation; the quadrilateral with the largest IUR with the minimum bounding rectangle will be obtained). If the ratio of the area of ​​this quadrilateral to the area of ​​the maximum outline obtained in Step 11-1 is less than a fixed value, then take the minimum bounding rectangle as the final quadrilateral. This fixed value can be set to 0.9.

[0091] Step 11-4: Fill the original ticket image with black borders of the same fixed length as described in Step 11-1, covering the top, bottom, left, and right sides. Based on the quadrilateral obtained in Step 11-3, perform a perspective transformation on the ticket image with the borders filled, ultimately obtaining an image of the main body area of ​​the ticket where the text and viewing direction are positive and background interference has been removed.

[0092] As a further improvement to the above scheme, the data types processed are not limited to invoices; they can also be applied to any task in scene text recognition where background interference occurs and text direction and perspective correction are needed. Multi-label classification networks can effectively distinguish different types of information about the same object within the same network, and can be used as an independent part to extract different detailed information about the same type of object. The post-processing method for invoice extraction can serve as an example of salient region detection. Based on this, subsequent operations for extracting salient region information from different types of objects can be implemented, further improving the accuracy of salient region detection.

[0093] In summary, this invention proposes a method for extracting and correcting invoice images from natural scenes. Compared to existing technologies, it effectively solves the problems of text tilt and perspective changes in invoice scene text recognition, effectively eliminates background interference, and simultaneously obtains the invoice type. The processed invoice image and the obtained invoice type can reduce interference and provide more information for subsequent tasks, improving the accuracy of scene text recognition and key information extraction.

[0094] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made as part of the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly within the patent protection scope of the present invention.

Claims

1. A method for extracting and correcting images of tickets in natural scenes, characterized in that, Includes the following steps: Step S01: Collect various ticket images taken in natural scenes, and manually classify them according to the type of ticket and the direction of the text on the ticket to obtain the type label and direction label of each ticket image; Step S02: Preprocess the ticket image by scaling it to a fixed size and performing random cropping, rotation, and image enhancement operations on the ticket image. Step S03: Construct a deep neural network for multi-label classification to classify the ticket image by type and text direction. This neural network consists of a backbone network DenseNet-169 and two fully connected layers. DenseNet-169 is responsible for extracting features from the image, and each fully connected layer shares the extracted feature information and is responsible for different classification tasks. Step S04: During the training phase, the cross-entropy loss function is used to measure the error between the prediction result output by the fully connected layer and the label. The loss is obtained by multiplying the error calculated by the two fully connected layers by different weight parameters and adding them together. During backpropagation, the SGD optimizer is used to obtain the multi-label classification model with the lowest loss. Step S05: Use the trained multi-label classification model to classify the ticket image to obtain the ticket type and text direction of the image, and then rotate and correct the ticket image according to the obtained text direction. Step S06: Collect various types of tickets photographed in natural scenes, and manually mark the main area of ​​the tickets using polygons on the collected images; Step S07: Perform data cleaning on the labeled images to reduce data labeling errors. Randomly add image enhancement operations to the labeled images and convert them into mask images based on the coordinates of the labels as training labels. Step S08: The deep learning salient region extraction network U²-Net is used to process the ticket images in natural scenes and extract the main body regions of various types of tickets. This neural network consists of nested U-Net networks. Each layer of U-Net outputs the feature map predicted by that layer, and these output feature maps are fused to obtain the probability map of the main body region of the ticket. Step S09: The loss function of U²-Net during the training phase consists of the sum of the cross-entropy loss of the feature map output by each layer and the mask of the labeled region. The Adam optimizer is used during training, and the model with the lowest loss is saved at the end. Step S10: Input the ticket image from the natural scene into the trained model to obtain the probability map of the main body region of the ticket. Map the output probability map to the region in the original image to obtain the main body region map of the ticket. Step S11: Based on the bill type obtained in step S05 and the main body area of ​​the bill obtained in step S10, post-processing methods are used to fine-tune the viewpoint and small angle of the text on the bill to obtain a main body area of ​​the bill with no background interference and the text direction is positive.

2. The method for extracting and correcting natural scene ticket images according to claim 1, characterized in that, The deep neural network for multi-label classification described in step S03 consists of a backbone network DenseNet-169 and two fully connected branch layers, with the following main structure: DenseNet-169 is mainly composed of four dense blocks with tight connections, and there is a transition layer between two adjacent dense blocks. Two fully connected layers are connected to the feature map output by DenseNet-169 to share feature map information and handle different classification tasks.

3. The method for extracting and correcting natural scene ticket images according to claim 1, characterized in that, The data augmentation and training label generation described in step S07 are as follows: Step S07-1: Check whether the key points of the main body area of ​​the invoice are correct on the labeled data to form the initial dataset; Step S07-2: Randomly select one or more image enhancement methods such as image scaling, histogram transformation, brightness change, Gaussian blur, orientation rotation, and salt-and-pepper noise to be added as part of the training dataset; Step S07-3: The key points of the main body area of ​​the ticket, which are manually labeled in the training dataset, are divided and extracted from the labeled main body area of ​​the ticket and the natural background area, and a mask of the original image is generated as the training label.

4. The method for extracting and correcting natural scene ticket images according to claim 1, characterized in that, The deep learning model U²-Net in step S08 consists of nested U-Net networks, including: The first layer U-Net module consists of an encoding module and a decoding module. The encoding module downsamples the input image layer by layer and extracts multi-scale features. The decoding module upsamples the corresponding multi-scale features layer by layer and fuses them with the features of the encoding module. The deepest encoded features serve as the connection between the encoding module and the decoding module. The decoding module outputs a feature map of the corresponding scale for loss calculation while completing feature fusion and upsampling. The second-layer U-Net module is nested within the first-layer U-Net module. The second-layer U-Net module also includes an encoding module and a decoding module, which are used to further extract and fuse multi-scale features. As the network progresses layer by layer, the number of convolutional operation units and upsampling units in the encoding and decoding modules of the second-layer U-Net module are adjusted with the network depth.

5. The method for extracting and correcting natural scene ticket images according to claim 1, characterized in that, Step S10 involves inputting the image of the ticket from the natural scene into the trained model to obtain a probability map of the main body region of the ticket. The specific steps are as follows: Step S10-1: After the input image is scaled proportionally, the scaled probability map is normalized. Step S10-2: The preprocessed data is processed through a network with an Encode and Decode structure to obtain the feature map output of each layer, and these feature maps are fused to obtain the final output feature map; Step S10-3: After obtaining the maximum and minimum values ​​of the output feature map, the probability map of the main body region of the bill is obtained by normalizing the output feature map by dividing the difference between the output feature map and the minimum value by the difference between the maximum and minimum values.

6. The method for extracting and correcting natural scene ticket images according to claim 1, characterized in that, The post-processing method described in step S11 for fine-tuning the perspective and text angle of the ticket refers to performing image post-processing operations based on the probability map output by U²-Net and the predicted ticket type. The specific steps are as follows: Step S11-1: Restore the predicted probability map to the size of the original image, perform preprocessing such as filling the boundary, grayscale conversion and binarization, detect contours on the binary map, find the contour with the largest area, and calculate its minimum bounding rectangle. Step S11-2: Based on the input ticket type, take the long or short side of the smallest bounding rectangle as the extracted parallel side. At the same time, after merging and filtering on the largest contour, take a group of line segments with longer lengths and calculate the intersection point of the extracted parallel side and the contour line segment. Step S11-3: Combine different intersection points and calculate the area of ​​the quadrilaterals respectively. Exclude quadrilaterals whose shapes are too different from rectangles. Take the quadrilateral with the largest intersection ratio of the smallest bounding rectangle and the quadrilateral. At the same time, it is necessary to verify that if the area of ​​the quadrilateral is too small, then take the smallest bounding rectangle as the final quadrilateral. Step S11-4: Fill the original ticket image with the boundary described in step S11-1, and perform perspective transformation on the ticket image with the boundary filled according to the quadrilateral obtained in step S11-3, finally obtaining the main body area of ​​the ticket with the text and the viewpoint direction being positive.

Images