Financial voucher image cutting correction method and system based on deep learning

Abstract

The invention belongs to the technical field of financial voucher image processing, and relates to a financial voucher image cutting correction method and system based on deep learning, and the method comprises the steps: converting an initial image into a to-be-predicted image with a preset size, and the initial image comprises a financial voucher; performing standardization processing on the to-be-predicted image; the to-be-predicted image is input into a pre-trained deep learning model to obtain prediction information, and the prediction information comprises position information of a financial voucher; processing the prediction information to obtain prediction position information, the prediction position information being position information of the financial voucher in the to-be-predicted image; the predicted position information is processed to obtain initial position information, and the initial position information is position information of the financial voucher in the initial image; and outputting a corrected financial voucher image according to the initial position information and the initial image. The bill correction method and device are suitable for a multi-bill mixed pasting scene, and bills at different angles can be corrected.

Description

Technical Field

[0001] This disclosure belongs to the field of financial document image processing technology, and relates to a method and system for cutting and correcting financial document images based on deep learning. Background Technology

[0002] As enterprises deepen their digital transformation, information technology is gradually being introduced into the financial management field to improve efficiency and accuracy. In the reimbursement and financial auditing processes, traditional paper vouchers are increasingly being replaced by electronic images, with companies typically archiving receipts, invoices, and other documents as images. This trend has spurred the demand for automated processing of this image data.

[0003] Optical Character Recognition (OCR) technology has been widely adopted in this context. OCR technology can convert scanned or photographed document images into editable text, greatly improving the efficiency of data entry and processing. Some enterprises have also introduced multimodal large models, combining image and text information to further enhance the intelligence level of document information extraction and verification.

[0004] Although OCR and multimodal large models have made some progress in the field of invoice recognition, the following technical bottlenecks and shortcomings still exist in practical applications: 1. Images containing multiple receipts or vouchers are difficult to identify accurately. During reimbursement or financial auditing, users often take photos of multiple receipts, invoices, or vouchers in the same image. Traditional OCR relies on the features of a single receipt for text extraction, which is ineffective for images with multiple receipts mixed together, easily leading to text errors or misidentification. 2. Non-standard document orientation affects the processing effectiveness of OCR and large-scale models. In practice, documents may be placed horizontally, upside down, or at an angle. OCR has a weaker ability to recognize documents with these non-standard orientations, easily leading to character recognition errors. Furthermore, multimodal large-scale models may also be misled when processing documents with non-standard orientations, reducing the accuracy of verification. 3. Irregular ticket boundaries affect segmentation and classification. During ticket photography, uneven lighting, complex backgrounds, and blurred edges can make boundary recognition difficult. Traditional OCR technology often relies on fixed text box positioning methods, making it difficult to accurately segment the effective area of ​​the ticket, thus affecting subsequent structured processing.

[0005] 4. Lack of efficient document correction algorithms leads to misidentification. Current OCR solutions mainly rely on rule matching or simple image processing methods to correct document orientation and shape, but lack intelligent correction technology based on deep learning. When faced with complex document scenarios (such as oblique shots, curled documents, uneven lighting, etc.), existing correction methods have poor adaptability, resulting in a decrease in the accuracy of the final OCR recognition.

[0006] 5. Poor Recognition Capability for Specialized Documents: In the process of recognizing documents in specialized fields, some OCR tools or multimodal large models typically employ image correction and segmentation techniques beforehand to improve recognition accuracy. However, due to the unique characteristics of some documents in the financial field (such as complex formats, irregular structures, low-quality scans, etc.), these general methods perform poorly. Summary of the Invention

[0007] To overcome the above problems, this disclosure provides a method and system for cutting and correcting financial voucher images based on deep learning.

[0008] The technical solutions disclosed in this disclosure are as follows: Firstly, this disclosure provides a deep learning-based method for segmenting and correcting financial document images, including: The initial image is converted into a prediction image of a preset size, and the initial image contains financial documents; Standardize the image to be predicted; The image to be predicted is input into a pre-trained deep learning model to obtain prediction information, which includes the location information of the financial voucher. The prediction information is processed to obtain the prediction location information, which is the location information of the financial voucher in the image to be predicted; The predicted location information is processed to obtain initial location information, which is the location information of the financial voucher in the initial image; Output the corrected financial document image based on the initial location information and the initial image.

[0009] Further, the initial image is converted into a prediction image of a preset size, including: Stretch the initial image to the same aspect ratio as the preset size; An image to be predicted of a preset size is obtained through affine transformation.

[0010] Furthermore, the image to be predicted is standardized, including: Obtain the RGB channel values ​​of the image to be predicted; The RGB channel values ​​are standardized based on the mean vector and standard deviation vector to eliminate the effects of one or more factors caused by differences in brightness, exposure, and contrast. The mean vector is the channel mean of the RGB three channels of the images in the training set used by the deep learning model; The standard deviation vector is the channel standard deviation of the RGB three channels of the images in the training set used by the deep learning model.

[0011] Furthermore, the deep learning model is a ResNet-18-FPN model, specifically: The ResNet-18 model consists of five convolutional modules connected in sequence, with each convolutional module outputting feature maps of varying levels from low to high. The FPN model fuses the feature maps output by different convolutional modules to obtain multi-scale feature maps.

[0012] Furthermore, the ResNet-18-FPN model also includes a multi-task prediction head; The multi-task prediction head makes the following predictions based on the multi-scale feature map: The center point heatmap branch yields the center point probability heatmap and the predicted center of the financial voucher; The offset branch yields the distance offsets from the prediction center to the four vertices of the predicted position of the financial voucher. The angle classification branch yields the predicted rotation angle of the financial voucher.

[0013] Further, the predicted information is processed to obtain the predicted location information, specifically as follows: The prediction information includes a probability heatmap, prediction center, distance offset, and prediction rotation angle. Non-maximum suppression is applied to the probability heatmap of center points to retain the predicted centers corresponding to local maxima, and the K centers with the highest confidence are retained as center point candidates. Obtain the distance offset and predicted rotation angle of the center point candidate; The vertex coordinates corresponding to the candidate center point are obtained based on the candidate center point position, distance offset, and predicted rotation angle. Candidate regions for fiscal vouchers are obtained based on vertex coordinates; The candidate fiscal voucher area is corrected for reflection rotation based on the predicted rotation angle. Non-maximum suppression is applied to the candidate fiscal document regions, and the candidate fiscal document regions corresponding to the local maximum values ​​are used as the fiscal document prediction regions.

[0014] Further, the predicted location information is processed to obtain initial location information, specifically as follows: Based on the predicted rotation angle and the downsampling ratio of the deep learning model, the vertex coordinates of the predicted region of the financial document are restored to the position information of the financial document in the image to be predicted. Based on the inverse transformation of the initial image into a predicted image of a preset size, the position information of the financial voucher in the predicted image is restored to the position information of the financial voucher in the initial image.

[0015] Furthermore, based on the initial location information and the initial image, the corrected image of the financial document is output, specifically as follows: Select the fiscal document prediction area in the initial image based on the initial location information; The predicted area of ​​the financial document is rotated according to the predicted rotation angle to obtain the predicted image of the financial document.

[0016] Furthermore, it also includes: Calculate the overlap of fiscal document forecast areas. For fiscal document forecast areas with an overlap exceeding the preset value, eliminate fiscal document forecast areas with lower confidence based on the center point confidence level.

[0017] Secondly, this disclosure provides a deep learning-based system for cutting and correcting fiscal voucher images, which cuts and corrects fiscal vouchers in an initial image according to the deep learning-based method for cutting and correcting fiscal voucher images described in the first aspect.

[0018] This disclosure has the following beneficial effects: This disclosure supports the detection of multiple invoices or vouchers at arbitrary angles within a single image, making it suitable for scenarios involving multiple invoices and vouchers. It can correct the angles of detected invoices, adjusting them to a standard orientation for easier subsequent OCR and large-scale model recognition. Based on the ResNet18 architecture, it balances model accuracy and inference speed, making it suitable for practical business needs. The model is fine-tuned and trained using vouchers and vouchers in the financial sector, demonstrating good performance on vertical data. Attached Figure Description

[0019] Figure 1 This is a diagram of the structure of the deep learning model disclosed in this publication.

[0020] Figure 2 This is a diagram of the ResNet-18 model architecture.

[0021] Figure 3 This is a structural diagram of the FPN model disclosed in this publication.

[0022] Figure 4 This is a schematic diagram showing the correction and cutting process before and after the present invention. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0024] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms "first," "second," and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described object changes. To keep the following description of the embodiments of this disclosure clear and concise, detailed descriptions of some known functions and components are omitted.

[0025] The present disclosure will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0026] Firstly, this disclosure provides a deep learning-based method for segmenting and correcting financial document images, including: The initial image is converted into a prediction image of a preset size, and the initial image contains financial documents; Standardize the image to be predicted; The image to be predicted is input into a pre-trained deep learning model to obtain prediction information, which includes the location information of the financial voucher. The prediction information is processed to obtain the prediction location information, which is the location information of the financial voucher in the image to be predicted; The predicted location information is processed to obtain initial location information, which is the location information of the financial voucher in the initial image; Output the corrected financial document image based on the initial location information and the initial image.

[0027] In one embodiment of this disclosure, converting an initial image into a predicted image of a preset size includes: Stretch the initial image to the same aspect ratio as the preset size; An image to be predicted of a preset size is obtained through affine transformation.

[0028] If the width and height of the image are 480 and 640 respectively, and the expected input size of the model is 512*512, then the longer side, i.e., 640, is selected as the reference and scaled proportionally to become 640*640. Then, an affine transformation is performed to map it to the 512×512 input. This ensures that the image will not be "flattened" or "cropped" after scaling, thus retaining the most effective information.

[0029] In one embodiment of this disclosure, the image to be predicted is standardized, including: Obtain the RGB channel values ​​of the image to be predicted; The RGB channel values ​​are standardized based on the mean vector and standard deviation vector to eliminate the effects of one or more factors caused by differences in brightness, exposure, and contrast. The mean vector is the channel mean of the RGB three channels of the images in the training set used by the deep learning model; The standard deviation vector is the channel standard deviation of the RGB three channels of the images in the training set used by the deep learning model.

[0030] The mathematical formula is: ; `original_pixel` is the RGB value of the original image (range: 0~255), `mean` is the channel mean (e.g., 0.408 for the R channel), and `std` is the channel standard deviation (e.g., 0.274 for the G channel).

[0031] In one embodiment of this disclosure, the deep learning model is a ResNet-18-FPN model, specifically: like Figure 1 As shown, the ResNet-18 model consists of five convolutional modules connected in sequence, with each convolutional module outputting feature maps of varying levels from low to high. like Figure 2 As shown, the FPN model fuses the feature maps output by different convolutional modules to obtain multi-scale feature maps.

[0032] The input image first passes through five convolutional modules (conv1, conv2_x, conv3_x, conv4_x, and conv5_x) of ResNet-18. Each stage outputs feature maps at different levels, capturing information from low-level texture to high-level semantics. The output features of ResNet-18 are then fused top-down using FPN, reducing the output resolution to 1 / 4 of the input size. FPN (Feature Pyramid Network) is a network architecture for multi-scale object detection, designed to extract information from feature maps at different levels, thereby enhancing the network's performance in object detection tasks at various scales. The core idea of ​​FPN is to effectively combine feature maps from different levels through top-down feature fusion, thus improving the ability to recognize objects of different sizes.

[0033] like Figure 1-3 As shown, in one embodiment of this disclosure, the ResNet-18-FPN model further includes a multi-task prediction head; The multi-task prediction head makes the following predictions based on the multi-scale feature map: The center point heatmap branch yields the center point probability heatmap and the predicted center of the financial voucher; The offset branch yields the distance offsets from the prediction center to the four vertices of the predicted position of the financial voucher. The angle classification branch yields the predicted rotation angle of the financial voucher.

[0034] In one embodiment of this disclosure, the predicted information is processed to obtain predicted location information, specifically as follows: The prediction information includes a probability heatmap, prediction center, distance offset, and prediction rotation angle. Non-maximum suppression is applied to the probability heatmap of center points to retain the predicted centers corresponding to local maxima, and the K centers with the highest confidence are retained as center point candidates. Obtain the distance offset and predicted rotation angle of the center point candidate; The vertex coordinates corresponding to the candidate center point are obtained based on the candidate center point position, distance offset, and predicted rotation angle. Candidate regions for fiscal vouchers are obtained based on vertex coordinates; The candidate fiscal voucher area is corrected for reflection rotation based on the predicted rotation angle. Non-maximum suppression is applied to the candidate fiscal document regions, and the candidate fiscal document regions corresponding to the local maximum values ​​are used as the fiscal document prediction regions.

[0035] like Figure 4As shown, in one embodiment of this disclosure, the predicted location information is processed to obtain initial location information, specifically as follows: Based on the predicted rotation angle and the downsampling ratio of the deep learning model, the vertex coordinates of the predicted region of the financial document are restored to the position information of the financial document in the image to be predicted. Based on the inverse transformation of the initial image into a predicted image of a preset size, the position information of the financial voucher in the predicted image is restored to the position information of the financial voucher in the initial image.

[0036] In one embodiment of this disclosure, a corrected image of the financial document is output based on the initial location information and the initial image, specifically as follows: Select the fiscal document prediction area in the initial image based on the initial location information; The predicted area of ​​the financial document is rotated according to the predicted rotation angle to obtain the predicted image of the financial document.

[0037] In one embodiment of this disclosure, it further includes: Calculate the overlap of fiscal document forecast areas. For fiscal document forecast areas with an overlap exceeding the preset value, eliminate fiscal document forecast areas with lower confidence based on the center point confidence level.

[0038] Secondly, this disclosure provides a deep learning-based system for cutting and correcting fiscal voucher images, which cuts and corrects fiscal vouchers in an initial image according to the deep learning-based method for cutting and correcting fiscal voucher images described in the first aspect.

[0039] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0040] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0041] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0042] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0043] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0044] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

[0045] The following points should be noted regarding this disclosure: (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0046] (2) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0047] The above description is merely an embodiment of this disclosure and does not limit the patent scope of this disclosure. Any equivalent structure made using the content of this disclosure and its drawings, or directly or indirectly applied to other related technical fields, is similarly included within the patent protection scope of this disclosure.

Claims

1. A deep learning-based method for segmenting and correcting financial voucher images, characterized in that, include: The initial image is converted into a prediction image of a preset size, and the initial image contains financial documents; Standardize the image to be predicted; The image to be predicted is input into a pre-trained deep learning model to obtain prediction information, which includes the location information of the financial voucher. The prediction information is processed to obtain the prediction location information, which is the location information of the financial voucher in the image to be predicted; The predicted location information is processed to obtain initial location information, which is the location information of the financial voucher in the initial image; Output the corrected financial document image based on the initial location information and the initial image.

2. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 1, characterized in that, Converting the initial image into a predicted image of a preset size includes: Stretch the initial image to the same aspect ratio as the preset size; An image to be predicted of a preset size is obtained through affine transformation.

3. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 1, characterized in that, The image to be predicted is standardized, including: Obtain the RGB channel values ​​of the image to be predicted; The RGB channel values ​​are standardized based on the mean vector and standard deviation vector to eliminate the effects of one or more factors caused by differences in brightness, exposure, and contrast. The mean vector is the channel mean of the RGB three channels of the images in the training set used by the deep learning model; The standard deviation vector is the channel standard deviation of the RGB three channels of the images in the training set used by the deep learning model.

4. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 1, characterized in that, The deep learning model is the ResNet-18-FPN model, specifically: The ResNet-18 model consists of five convolutional modules connected in sequence, with each convolutional module outputting feature maps of varying levels from low to high. The FPN model fuses the feature maps output by different convolutional modules to obtain multi-scale feature maps.

5. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 4, characterized in that, The ResNet-18-FPN model also includes a multi-task prediction head; The multi-task prediction head makes the following predictions based on the multi-scale feature map: The center point heatmap branch yields the center point probability heatmap and the predicted center of the financial voucher; The offset branch yields the distance offsets from the prediction center to the four vertices of the predicted position of the financial voucher. The angle classification branch yields the predicted rotation angle of the financial voucher.

6. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 5, characterized in that, The predicted information is processed to obtain the predicted location information, specifically as follows: The prediction information includes a probability heatmap, prediction center, distance offset, and prediction rotation angle. Non-maximum suppression is applied to the probability heatmap of center points to retain the predicted centers corresponding to local maxima, and the K centers with the highest confidence are retained as center point candidates. Obtain the distance offset and predicted rotation angle of the center point candidate; The vertex coordinates corresponding to the candidate center point are obtained based on the candidate center point position, distance offset, and predicted rotation angle. Candidate regions for fiscal vouchers are obtained based on vertex coordinates; The candidate fiscal voucher area is corrected for reflection rotation based on the predicted rotation angle. Non-maximum suppression is applied to the candidate fiscal document regions, and the candidate fiscal document regions corresponding to the local maximum values ​​are used as the fiscal document prediction regions.

7. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 6, characterized in that, The predicted location information is processed to obtain initial location information, specifically as follows: Based on the predicted rotation angle and the downsampling ratio of the deep learning model, the vertex coordinates of the predicted region of the financial document are restored to the position information of the financial document in the image to be predicted. Based on the inverse transformation of the initial image into a predicted image of a preset size, the position information of the financial voucher in the predicted image is restored to the position information of the financial voucher in the initial image.

8. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 7, characterized in that, Based on the initial location information and the initial image, the corrected image of the financial document is output, specifically: Select the fiscal document prediction area in the initial image based on the initial location information; The predicted area of ​​the financial document is rotated according to the predicted rotation angle to obtain the predicted image of the financial document.

9. The deep learning-based method for segmenting and correcting fiscal voucher images according to claim 7, characterized in that, Also includes: Calculate the overlap of fiscal document forecast areas. For fiscal document forecast areas with an overlap exceeding the preset value, eliminate fiscal document forecast areas with lower confidence based on the center point confidence level.

10. A deep learning-based image segmentation and correction system for financial documents, characterized in that, The deep learning-based fiscal voucher image cutting and correction method according to any one of claims 1-9 cuts and corrects fiscal vouchers in the initial image.

Images