Invoice information identification method and device, and electronic device

Abstract

The application discloses an invoice information recognition method and device and electronic equipment. Firstly, the multi-scale feature map of the to-be-recognized invoice image is determined; the position interaction processing of the pixel points is performed on each scale feature map, and the scale fusion processing between the various scale feature maps is performed, so that the category label of each field in the to-be-recognized invoice image and the boundary box position information of each field are obtained; according to the category label of each field and the boundary box position information of each field, the corresponding to-be-detected areas in the to-be-recognized invoice image are determined; the convolution processing is respectively performed on each to-be-detected area, so that the corresponding offset of each to-be-detected area is obtained; and the corresponding to-be-detected area is adjusted based on each offset; the text recognition is performed according to the adjusted to-be-detected areas, so that the recognition result of the to-be-recognized invoice image is obtained, and then the to-be-recognized invoice image can be accurately recognized.

Description

Technical Field

[0001] This invention relates to the field of computer vision processing technology, and in particular to a method, apparatus and electronic device for recognizing invoice information. Background Technology

[0002] With the accelerated advancement of enterprise digital transformation, the demand for automated processing of invoices, as important financial vouchers and expense reimbursement documents, is growing. Traditional invoice processing requires manual data entry, which is inefficient, error-prone, and difficult to scale, failing to meet the needs of modern enterprise financial management. Therefore, automatic invoice recognition methods based on Optical Character Recognition (OCR) have gained considerable attention.

[0003] However, there are currently many types of invoices, and the field layouts of different types vary greatly. The vertical and horizontal ratios of the text areas on the invoices also differ. OCR-based invoice recognition cannot accurately locate different fields on different types of invoices, and therefore cannot accurately recognize the invoices. Summary of the Invention

[0004] This application provides a method, apparatus, and electronic device for recognizing invoice information. By performing scale processing on the multi-scale feature map of the invoice image to be recognized, accurate bounding box position information of each field is obtained. Then, convolution processing is used to adjust each detection region determined based on the bounding box position information of each field. Text recognition is performed on each adjusted detection region to obtain accurate invoice recognition results.

[0005] In a first aspect, embodiments of this application provide a method for identifying invoice information, the method comprising:

[0006] A multi-scale feature map of the invoice image to be identified is determined; and pixel position interaction processing is performed on each scale feature map, and scale fusion processing is performed between the various scale feature maps to obtain the category label of each field in the invoice image to be identified and the bounding box position information of each field; wherein, the category label includes at least the invoice title, invoice code, invoice number, invoice date, and tax amount;

[0007] Based on the category labels of each field and the bounding box location information of each field, the corresponding detection regions in the invoice image to be identified are determined.

[0008] Each region to be detected is convolved to obtain the offset of each region; and the corresponding region to be detected is adjusted based on each offset.

[0009] Text recognition is performed on each of the adjusted detection areas to obtain the recognition result of the invoice image to be recognized.

[0010] Compared to existing technologies, this application first determines multi-scale feature maps of the invoice image to be recognized. Then, it performs pixel position processing on each scale feature map and scale fusion processing between the various scale feature maps. This ensures that each pixel in each scale feature map can perceive the positional information of other pixels within the same scale feature map, as well as the relative positional relationships and contextual associations between different fields under the same scale feature map. Furthermore, by combining the positional information from different scale feature maps, the bounding box position information and category label of each field in the invoice image to be recognized can be accurately determined. Then, based on the category label and bounding box position information of each field, the corresponding detection region can be accurately determined. The offset obtained by convolution processing can be used to adjust the bounding box position information of the detection region, thereby obtaining an accurate invoice recognition result.

[0011] As an optional implementation, the step of performing pixel position interaction processing on each scale feature map includes:

[0012] The target encoder encodes each scale feature map separately, and an attention mechanism is used to process each encoding result so that the position information of each pixel in each scale feature map includes the position information of the remaining pixels in the scale feature map other than its own pixel.

[0013] This application uses a target encoder to encode each scale feature map separately, which enables each pixel in each scale feature map to perceive the position information of other pixels in the same scale feature map, as well as the relative positional relationship and contextual association between different fields in the same scale feature map. This allows for accurate determination of the bounding box position information of each field in the invoice image to be recognized.

[0014] As an optional implementation, the scale fusion processing between feature maps of different scales includes:

[0015] Determine the resolution value of each scale feature map; and based on the time series corresponding to each scale feature map, perform scale fusion processing on two adjacent scale feature maps with different resolution values.

[0016] This application performs scale fusion processing on two adjacent scale feature maps according to different time series and resolution values, so as to accurately locate the bounding box position information of small-sized text in the low-level details, and thus accurately determine the bounding box position information of each field in the invoice image to be recognized.

[0017] As an optional implementation, after determining the corresponding detection regions in the invoice image to be recognized, and before performing text recognition based on the adjusted detection regions, the method further includes:

[0018] If an occluded area to be detected is detected, then the occluded area to be detected is converted to a different color space.

[0019] A mask region is generated using a target hue threshold, and the mask region is removed from the color space conversion result.

[0020] The text recognition based on the adjusted detection regions includes:

[0021] Text recognition is performed based on each detection region after adjustment and removal of the masked area.

[0022] This application addresses the situation where the text area is partially covered by a red stamp. After detecting the obscured area to be detected, it performs color space conversion and then uses a mask area to remove the area to be detected, thereby enabling more accurate identification of invoice information.

[0023] As an optional implementation, after performing text recognition based on the adjusted detection regions to obtain the recognition result of the invoice image to be recognized, the method further includes:

[0024] The recognition results are validated based on preset field formats and inter-field logic.

[0025] If the verification result is unsuccessful, a manual review reminder message will be generated.

[0026] After determining the recognition result, this application can further verify the accuracy of the invoice information recognition by checking the recognition result.

[0027] Secondly, embodiments of this application provide an invoice information recognition device, the device comprising:

[0028] The scale feature processing module is used to determine the multi-scale feature map of the invoice image to be identified; and to perform pixel position interaction processing on each scale feature map and scale fusion processing between the various scale feature maps to obtain the category label of each field in the invoice image to be identified and the bounding box position information of each field; wherein, the category label includes at least the invoice title, invoice code, invoice number, invoice date, and tax amount;

[0029] The region determination module is used to determine the corresponding detection regions in the invoice image to be identified based on the category labels of each field and the bounding box position information of each field.

[0030] The region adjustment module is used to perform convolution processing on each region to be detected to obtain the offset of each region; and adjust the corresponding region to be detected based on each offset.

[0031] The recognition result module is used to perform text recognition based on the adjusted detection areas to obtain the recognition result of the invoice image to be recognized.

[0032] As an optional implementation, the scale feature processing module is specifically used for:

[0033] The target encoder encodes each scale feature map separately, and an attention mechanism is used to process each encoding result so that the position information of each pixel in each scale feature map includes the position information of the remaining pixels in the scale feature map other than its own pixel.

[0034] As an optional implementation, the scale feature processing module is specifically used for:

[0035] Determine the resolution value of each scale feature map; and based on the time series corresponding to each scale feature map, perform scale fusion processing on two adjacent scale feature maps with different resolution values.

[0036] As an optional implementation, the region adjustment module is further configured to:

[0037] If an occluded area to be detected is detected, then the occluded area to be detected is converted to a different color space.

[0038] A mask region is generated using a target hue threshold, and the mask region is removed from the color space conversion result.

[0039] The recognition result module is specifically used for:

[0040] Text recognition is performed based on each detection region after adjustment and removal of the masked area.

[0041] As an optional implementation, the recognition result module is further used for:

[0042] The recognition results are validated based on preset field formats and inter-field logic.

[0043] If the verification result is unsuccessful, a manual review reminder message will be generated.

[0044] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it causes the processor to implement the method described in any of the first aspects above.

[0045] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method described in any of the first aspects.

[0046] Fifthly, embodiments of this application also provide a computer program product, including a computer program that is executed by a processor to implement the method as described in any of the first aspects above.

[0047] The technical effects of any of the implementation methods in aspects two through five can be found in the technical effects of the corresponding implementation methods in aspect one, and will not be repeated here. Attached Figure Description

[0048] Figure 1 This is a schematic diagram illustrating an application scenario of an invoice information recognition method provided in an embodiment of this application.

[0049] Figure 2 This is a schematic flowchart of an invoice information recognition method provided in an embodiment of this application;

[0050] Figure 3 This is a schematic diagram of the structure of an invoice information recognition device provided in an embodiment of this application;

[0051] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0053] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data used can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0054] With the accelerated advancement of enterprise digital transformation, the demand for automated processing of invoices, as important financial vouchers and expense reimbursement documents, is growing. Traditional invoice processing requires manual data entry, which is inefficient, error-prone, and difficult to scale, failing to meet the needs of modern enterprise financial management. Therefore, OCR-based automatic invoice recognition methods have gained considerable attention.

[0055] However, there are currently many types of invoices, and the field layouts of different types vary greatly. The vertical and horizontal ratios of the text areas on the invoices also differ. OCR-based invoice recognition cannot accurately locate different fields on different types of invoices, and therefore cannot accurately recognize the invoices.

[0056] To address this, this application provides a method, apparatus, and electronic device for recognizing invoice information. First, a multi-scale feature map of the invoice image to be recognized is determined. Then, pixel position processing is performed on each scale feature map, and scale fusion processing is applied between the various scale feature maps. This ensures that each pixel in each scale feature map can perceive the positional information of other pixels within the same scale feature map, as well as the relative positional relationships and contextual associations between different fields within the same scale feature map. Furthermore, by combining the positional information from different scale feature maps, the bounding box position information and category label of each field in the invoice image to be recognized can be accurately determined. Then, based on the category label and bounding box position information of each field, the corresponding detection region can be accurately determined. The offset obtained through convolution processing can be used to adjust the bounding box position information of the detection region, thereby obtaining an accurate invoice recognition result.

[0057] After introducing the design concept of the embodiments of this application, the following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only for illustrating the embodiments of this application and are not intended to limit the scope. In specific implementation, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0058] refer to Figure 1This diagram illustrates an application scenario for the invoice information recognition method provided in this application. The application scenario includes multiple terminal devices 101 (including terminal device 101-1, terminal device 101-2, ..., terminal device 101-n) and a server 102. The terminal devices 101 and the server 102 are connected via a wireless or wired network. The terminal devices 101 include, but are not limited to, desktop computers, mobile phones, mobile computers, tablets, media players, smart wearable devices, smart TVs, and other electronic devices. The server 102 can be a single server, a server cluster consisting of several servers, or a cloud computing center. The server 102 can be an independent physical server, a server cluster consisting of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0059] For example, terminal device 101-1 sends the invoice image to be recognized to server 102. In server 102, firstly, multi-scale feature maps of the invoice image to be recognized are determined; then, pixel position interaction processing is performed on each scale feature map, and scale fusion processing is performed between the various scale feature maps to obtain the category labels of each field in the invoice image to be recognized and the bounding box position information of each field; then, based on the category labels of each field and the bounding box position information of each field, the corresponding detection regions in the invoice image to be recognized are determined; convolution processing is performed on each detection region to obtain the offset of each detection region; and the corresponding detection regions are adjusted based on each offset; finally, text recognition is performed on the adjusted detection regions to obtain the recognition result of the invoice image to be recognized; the recognition result is sent to terminal device 101-1, and the recognition result is displayed to the user through terminal device 101-1.

[0060] Here, text recognition can also be performed on each of the adjusted detection areas in the terminal device 101-1. This is just an example, and this application does not limit the specific processing procedures of the terminal device 101-1 and the server 102.

[0061] Of course, the methods provided in the embodiments of this application are not limited to... Figure 1 The application scenarios shown can also be used in other possible scenarios, and this application does not impose any limitations. Figure 1 The functions that each device in the application scenario shown can achieve will be described in subsequent method embodiments, and will not be elaborated on here.

[0062] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.

[0063] The following is combined with Figure 1 The application scenarios shown are as follows: Figure 2 As shown, the method for recognizing invoice information provided in this application embodiment will be described in detail, specifically including the following steps:

[0064] S201, determine the multi-scale feature map of the invoice image to be identified; and perform pixel position interaction processing on each scale feature map, as well as scale fusion processing between the various scale feature maps, to obtain the category label of each field in the invoice image to be identified and the bounding box position information of each field.

[0065] For example, the invoice image to be recognized can be any one of the following: a scanned image of a paper invoice, a photo of the invoice taken with a mobile phone, or an image generated after rendering a PDF electronic invoice. For PDF electronic invoices, the pages can be rendered as high-resolution images, for example, the rendering resolution configuration (default value 300 DPI) can be set to meet the image clarity requirements for OCR recognition. Specifically, the PyMuPDF (fitz) or pdf2image library can be used to render each page of the PDF file as a high-resolution image, with the rendering DPI set to 300. For multi-page PDFs, each page is rendered and processed separately.

[0066] Optionally, before extracting scale features from the invoice image to be recognized, image preprocessing can be performed. For example, image enhancement, denoising, and tilt correction can be applied. Specifically, image enhancement involves using Contrast-Limited Adaptive Histogram Equalization (CLAHE) to enhance the contrast of the invoice image, highlighting the difference between the text and background areas. Denoising involves using Gaussian filtering or bilateral filtering to remove noise from the invoice image, smoothing the noise while preserving text edge information. Tilt correction involves detecting the tilt angle of the invoice image based on Hough Transform or the minimum area bounding rectangle algorithm, and correcting the tilted image to the positive orientation through affine or perspective transformation.

[0067] In one embodiment of this application, the image of the invoice to be identified is input into an improved real-time detection Transformer (RT-DETRv2) model to determine the bounding box location information and category label of each field. For example, the RT-DETRv2 model uses a convolutional neural backbone network (such as ResNet50 or HGNet) to extract multi-scale feature maps.

[0068] Then, each scale feature map is encoded separately according to the target encoder (e.g., a standalone Transformer encoder), and an attention mechanism is used to process each encoding result so that the position information of each pixel in each scale feature map includes the position information of the remaining pixels in the scale feature map excluding its own pixel. That is, each pixel in each scale feature map can perceive the position information of other pixels in the same scale feature map, as well as the relative positional relationships and contextual associations between different fields in the same scale feature map. For example, the invoice number is usually adjacent to the invoice code.

[0069] Simultaneously, the resolution values ​​of each scale feature map can be determined, and based on the corresponding time series of each scale feature map, scale fusion processing can be performed on adjacent scale feature maps with different resolution values. For example, according to the time series, two adjacent scale feature maps are, in sequence, a low-resolution upsampled scale feature map containing high-level semantic information and a high-resolution scale feature map containing detailed location information. Fusing the two scale feature maps allows the model to accurately locate salient regions such as "invoice amount" using high-level semantics, and accurately locate the bounding box location information of small-sized text such as "invoice number" using low-level details. This fusion method enhances feature representation, providing more accurate input for the subsequent Transformer decoder, and achieving high-precision detection.

[0070] After performing pixel-level position interaction processing on each scale feature map and scale fusion processing between the feature maps, a Transformer-based decoder and deformable attention mechanism generate category labels and bounding box location information for each field in the invoice image to be identified. Confidence scores for each field can also be obtained. The category labels include at least the following: Invoice Title, Invoice Code, Invoice Number, Invoice Date, Buyer Name, Buyer Tax ID, Seller Name, Seller Tax ID, Total Amount (including tax), Subtotal Amount (excluding tax), and Tax Amount.

[0071] It should be noted that during the training of the RT-DETRv2 model, the Hungarian matching algorithm can be used to achieve a one-to-one matching between the predicted bounding box and the ground truth bounding box, so that redundant detection boxes are not required when applying the RT-DETRv2 model.

[0072] Optionally, during the training of the RT-DETRv2 model, noisy bounding boxes can be added to the input queries of the decoder, enabling the model to accurately reconstruct the target location even under noisy conditions, thus enhancing the model's robustness. Furthermore, query selection with uncertainty minimization can be employed. By evaluating the uncertainty of the encoder's output features, high-quality initial object queries can be provided to the decoder, accelerating model convergence and improving detection accuracy. Additionally, for the detection of small targets (such as invoice numbers, invoice codes, and other small font fields), the ProgLoss progressive loss function and the Spatial-Temporal Alignment Loss (STAL) function can be optimized to improve the recognition capability of small targets.

[0073] This is merely an example to illustrate the training of the RT-DETRv2 model. This application does not limit the specific training process of the RT-DETRv2 model and can be adjusted according to the actual situation.

[0074] After determining the category labels of each field and the bounding box location information of each field in the invoice image to be recognized, step S202 is executed to determine the corresponding detection regions in the invoice image to be recognized based on the category labels of each field and the bounding box location information of each field; step S203 is executed to perform convolution processing on each detection region to obtain the offset of each detection region; and the corresponding detection regions are adjusted based on each offset; and step S204 is executed to perform text recognition based on the adjusted detection regions to obtain the recognition result of the invoice image to be recognized.

[0075] For example, the invoice image to be recognized is cropped according to the bounding box position information of each field, and the cropped image is input into the PaddleOCR model for text recognition.

[0076] Optionally, a text detection model within the PaddleOCR framework (such as DBNet++) can be used to locate the region to be detected and output the coordinates of the text's boundary polygons. The DBNet++ model embeds a variable convolution module within its feature pyramid network. This module, based on standard convolution, learns a set of two-dimensional offsets for each sampling point. During feature extraction, the sampling points of the convolution kernel adaptively deform according to the learned offsets, thus more accurately fitting the irregularly shaped or tilted / curved text edges. For example, in invoices with narrow, long text lines, such as invoice codes and numbers with significant aspect ratios, and when the text area is covered by a red stamp, deformable convolution can guide sampling points to focus on the unoccluded strokes, providing cleaner text region features for subsequent recognition.

[0077] Then, text recognition models in the PaddleOCR model (such as SVTR or PP-OCRv5) can be used for character recognition to output text strings. PP-OCRv5 adopts a dynamic super-resolution perceptual architecture (DSANet) and maintains high recognition accuracy in different resolution scenarios through an adaptive super-resolution adjustment strategy.

[0078] In one embodiment of this application, if an occluded detection area is detected, a color space conversion is performed on the occluded detection area; then, a mask area is generated using a target hue threshold, and the mask area is removed from the color space conversion result; text recognition is then performed on each detection area after adjustment and removal of the mask area.

[0079] Exemplarily, when a to-be-detected area is covered by a red seal, the RGB image is converted to the HSV color space, threshold segmentation is performed using the hue range of red (H∈[0,10]∪[156,180]) to generate a seal mask area, the seal mask area is removed from the text area, and then text recognition is performed, so as to effectively reduce the interference of the seal on text recognition.

[0080] In an embodiment of the present application, after text recognition is performed on each adjusted to-be-detected area to obtain the recognition result of the to-be-recognized invoice image, the recognition result may further be subjected to verification of preset field formats and inter-field logic; if the verification result is not passed, a manual review reminder message is generated. If the verification result is passed, or is passed after post-processing, the result is output according to a preset data structure (such as JSON format).

[0081] Exemplarily, regular expressions are used to perform format verification and error correction on each field. For example, the invoice code is usually a 10-digit or 12-digit numeric code; the invoice number is an 8-digit numeric code; the taxpayer identification number is a 15-20 digit alphanumeric mixed code; the date format is "YYYY年MM月DD日" or "YYYY-MM-DD"; the amount field shall conform to the format of "¥XXX.XX" or pure numbers. For recognition results that do not conform to the preset field formats, an edit distance algorithm or post-processing based on a language model is adopted for automatic error correction, or a manual review reminder message is generated.

[0082] A numerical consistency check may also be performed on the recognized tax-inclusive amount, tax-exclusive amount and tax amount, to verify whether the logical relationship of "tax-inclusive amount = tax-exclusive amount + tax amount" is valid. If the logical relationship is not valid, the result is marked as a low-confidence result, and a manual review reminder message is prompted.

[0083] This application employs RT-DETRv2 as the field localization model, fully leveraging its Transformer architecture's global contextual understanding capabilities to accurately identify field regions of different sizes and positions on invoices. Simultaneously, it combines the PaddleOCR model, utilizing its text detection and recognition algorithms, to effectively handle the diverse fonts and sizes of text in the unique scenario of invoices. Seal separation preprocessing further reduces the impact of seal interference on the recognition results. Furthermore, by uniformly supporting image and PDF invoice input, and utilizing an automatic PDF rendering module to achieve format compatibility, it avoids users deploying separate recognition processes for different formats, simplifying the system architecture. Moreover, by performing pre-defined field format validation, inter-field logic validation, and structured organization on the recognition results, it ensures that the output invoice recognition results can be directly used for subsequent financial audits, reimbursements, and other business processes, reducing the workload of manual review. The RT-DETRv2 model's Transformer architecture inherently possesses strong layout generalization capabilities, and combined with PaddleOCR's industrial-grade text recognition capabilities, it can adapt to various invoice layout changes without requiring the training of independent models for different layouts.

[0084] For example, suppose we acquire a scanned image of a paper VAT invoice (resolution 200-300 DPI, JPG or PNG format). Perform the following preprocessing operations on the image:

[0085] Using OpenCV's createCLAHE function, with a cropping limit of 2.0 and a grid size of 8×8, adaptive histogram equalization is performed on the image's luminance channel to enhance the contrast between text and background.

[0086] Using a bilateral filter with parameters set to d=9, sigmaColor=75, and sigmaSpace=75, text edges are preserved while smoothing background noise.

[0087] The main straight lines in the invoice image are detected using Canny edge detection and Hough transform, and the tilt angle is calculated. If the tilt angle exceeds a preset threshold (e.g., ±5°), it is corrected by rotation.

[0088] Then perform RT-DETRv2 target detection:

[0089] Load the pre-trained RT-DETRv2 model. The model's backbone network is ResNet50, the input image size is 640×640 pixels, and the decoder uses 300 object queries. The model defines 11 detection categories, covering invoice title, invoice code, invoice number, invoice date, buyer name, buyer identification number, seller name, seller identification number, amount including tax, amount excluding tax, and tax.

[0090] The preprocessed invoice image is scaled to 640×640 and input into the model. A CNN backbone network extracts multi-scale features (e.g., feature maps at three scales with strides of 8, 16, and 32). A hybrid encoder performs intra-scale feature interaction and cross-scale feature fusion. Finally, a Transformer decoder based on deformable attention generates the detection results. The model outputs the class label, bounding box location information, and confidence score for each field.

[0091] Detection results with a confidence score below 0.5 are discarded, while high-confidence detection boxes are retained. Duplicate detection boxes of the same category (IoU greater than a preset threshold) are implicitly removed during the training phase using the Hungarian matching algorithm, and no additional NMS step is required during model application.

[0092] Then perform PaddleOCR text recognition:

[0093] Based on the detected bounding box position information, sub-images corresponding to each field are cropped from the invoice image. These sub-images are then sequentially input into the PaddleOCR engine.

[0094] Text detection stage: The DBNet++ model performs text line detection on the sub-image and outputs the polygon coordinates of the text lines. For single-line text such as invoice numbers, the detection result is a single text line region; for fields that may have multiple lines, such as buyer's name and seller's name, the detection result is multiple text line regions.

[0095] Text recognition stage: The SVTR model performs character recognition on each detected text line and outputs a text string. For multi-line text, the recognition results of each line are concatenated into a complete string in top-to-bottom order.

[0096] For areas covered by the detected red stamp, stamp separation processing is performed: the sub-image is converted from the RGB color space to the HSV color space, and the red hue threshold range H∈[0, 10]∪[156, 180], saturation S∈[43,255], and brightness V∈[46, 255] are set for binarization to generate the stamp mask area; the pixels corresponding to the mask in the original sub-image are replaced with the local background color (taking the average value of the neighboring pixels of the stamp area), and the text area is restored before performing text recognition.

[0097] Finally, post-processing and output of recognition results are performed, that is, the following post-processing is applied to the recognition results of each field:

[0098] Invoice code: Verify whether it is a 10-digit or 12-digit numeric code; if it does not, mark it with a prompt.

[0099] Invoice number: Verify if it is an 8-digit number. If it is less than 8 digits, try to add zeros on the left to correct it.

[0100] Taxpayer Identification Number: Verify whether it conforms to a regular expression pattern of 15 to 20 characters of mixed alphanumeric encoding.

[0101] Dates: Convert to the standard format "YYYY-MM-DD".

[0102] Amount: Remove spaces and irrelevant characters from the recognition results, retain numbers and decimal points, and convert to floating-point format.

[0103] Logical verification: Verify whether "|tax-inclusive amount - (tax-exclusive amount + tax)| < 0.02" (rounding error of no more than 2 points is allowed). If it is not true, mark it as low confidence.

[0104] The final output is a structured JSON format result, as shown in the example below:

[0105] json

[0106] { "invoice_title": "General VAT Invoice", "invoice_code": "031002100611", "invoice_number": "12345678", "invoice_date": "2026-05-19", "buyer_name": "XX Technology Co., Ltd.", "buyer_tax_id": "91110108MA01XXXXX", "seller_name": "YY Trading Co., Ltd.", "seller_tax_id": "91110108MA02YYYYY", "total_amount": 11300.00, "subtotal_amount": 10000.00, "tax_amount": 1300.00, "confidence": 0.97}.

[0107] For the model training process described above, samples of VAT invoices in different formats (including ordinary VAT invoices, electronic VAT invoices, and special VAT invoices) can be collected. Each invoice image is manually labeled, including the category labels for each field and the bounding box position information. To improve the model's generalization ability, data augmentation is performed on the training data, including random rotation (±10°), random brightness adjustment, random contrast adjustment, and random Gaussian noise addition. Fine-tuning is performed based on the pre-trained RT-DETRv2 model: training parameters are set as follows: batch size of 16, initial learning rate of 1e-4, cosine annealing learning rate scheduling strategy, and 100 epochs of training. The optimizer is AdamW, with weight decay of 1e-4. Loss functions include classification loss (Focal Loss), bounding box regression loss (a combination of L1 Loss and GIoU Loss), and the ProgLoss + STAL loss function specifically for denoising tasks. Evaluation is performed on multiple validation sets, and the model with the highest mAP is selected as the final deployment model.

[0108] For businesses, this application can automatically identify expense invoices submitted by employees, extract field information, and automatically fill in expense forms, significantly reducing manual data entry. It can also automatically verify the logical relationship between invoice codes, numbers, and amounts, automatically identify and archive supplier invoices, and automate accounts receivable and payable management. Furthermore, it can batch process historical paper invoice archives, converting them into structured electronic data for easy retrieval and management.

[0109] like Figure 3 As shown in the figure, this application embodiment provides an invoice information recognition device 300, the device including:

[0110] The scale feature processing module 301 is used to determine the multi-scale feature map of the invoice image to be identified; and to perform pixel position interaction processing on each scale feature map and scale fusion processing between the various scale feature maps to obtain the category label of each field in the invoice image to be identified and the bounding box position information of each field; wherein, the category label includes at least the invoice title, invoice code, invoice number, invoice date, and tax amount;

[0111] The region determination module 302 is used to determine the corresponding regions to be detected in the invoice image to be identified based on the category labels of each field and the bounding box position information of each field.

[0112] The region adjustment module 303 is used to perform convolution processing on each region to be detected to obtain the offset of each region to be detected; and to adjust the corresponding region to be detected based on each offset.

[0113] The recognition result module 304 is used to perform text recognition based on the adjusted detection areas to obtain the recognition result of the invoice image to be recognized.

[0114] As an optional implementation, the scale feature processing module 301 is specifically used for:

[0115] The target encoder encodes each scale feature map separately, and an attention mechanism is used to process each encoding result so that the position information of each pixel in each scale feature map includes the position information of the remaining pixels in the scale feature map other than its own pixel.

[0116] As an optional implementation, the scale feature processing module 301 is specifically used for:

[0117] Determine the resolution value of each scale feature map; and based on the time series corresponding to each scale feature map, perform scale fusion processing on two adjacent scale feature maps with different resolution values.

[0118] As an optional implementation, the region adjustment module 303 is also used for:

[0119] If an occluded area to be detected is detected, then the color space of the occluded area to be detected is converted.

[0120] A mask region is generated using the target hue threshold, and the mask region is removed from the color space conversion result.

[0121] The recognition result module 304 is specifically used for:

[0122] Text recognition is performed on each detection area after adjustment and removal of the masked area.

[0123] As an optional implementation, the recognition result module 304 is also used for:

[0124] The recognition results are validated based on preset field formats and inter-field logic.

[0125] If the verification result is unsuccessful, a manual review reminder message will be generated.

[0126] Having introduced the invoice information identification method and apparatus according to exemplary embodiments of this application, we will now introduce an electronic device according to another exemplary embodiment of this application.

[0127] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0128] In some possible implementations, the electronic device according to this application may include at least one processor and at least one memory. The memory stores program code that, when executed by the processor, causes the processor to perform the steps in the invoice information identification method according to the various exemplary embodiments of this application described above. For example, the processor may perform steps such as those in the invoice information identification method.

[0129] The following reference Figure 4 To describe an electronic device 40 according to this embodiment of the present application. Figure 4 The electronic device 40 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0130] like Figure 4 As shown, the electronic device 40 is presented in the form of a general-purpose electronic device. The components of the electronic device 40 may include, but are not limited to: at least one processor 41, at least one memory 42, and a bus 43 connecting different system components (including memory 42 and processor 41).

[0131] Bus 43 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or a local bus using any of the multiple bus structures.

[0132] The memory 42 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 421 and / or cache memory 422, and may further include read-only memory (ROM) 423.

[0133] The memory 42 may also include a program / utility 425 having a set (at least one) of program modules 424, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0134] Electronic device 40 can also communicate with one or more external devices 44 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 40, and / or with any device that enables electronic device 40 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 45. Furthermore, electronic device 40 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 46. As shown, network adapter 46 communicates with other modules used in electronic device 40 via bus 43. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 40, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0135] In an exemplary embodiment, a computer-readable storage medium including instructions is also provided, such as a memory 42 including instructions, which can be executed by a processor 41 to perform the above-described method. Optionally, the computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0136] In an exemplary embodiment, a computer program product is also provided, including a computer program / instructions that, when executed by a processor 41, implement any of the methods for identifying invoice information as provided in this application.

[0137] In an exemplary embodiment, various aspects of the invoice information identification method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the invoice information identification method according to the various exemplary embodiments of this application described above.

[0138] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0139] The program product for image scaling according to the embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0140] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0141] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0142] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0143] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0144] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0145] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable image scaling device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable image scaling device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable image scaling device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0150] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for identifying invoice information, characterized in that, The method includes: A multi-scale feature map of the invoice image to be identified is determined; and pixel position interaction processing is performed on each scale feature map, and scale fusion processing is performed between the various scale feature maps to obtain the category label of each field in the invoice image to be identified and the bounding box position information of each field; wherein, the category label includes at least the invoice title, invoice code, invoice number, invoice date, and tax amount; Based on the category labels of each field and the bounding box location information of each field, the corresponding detection regions in the invoice image to be identified are determined. Each region to be detected is convolved to obtain the offset of each region; and the corresponding region to be detected is adjusted based on each offset. Text recognition is performed on each of the adjusted detection areas to obtain the recognition result of the invoice image to be recognized.

2. The method according to claim 1, characterized in that, The step of performing pixel position interaction processing on each scale feature map includes: The target encoder encodes each scale feature map separately, and an attention mechanism is used to process each encoding result so that the position information of each pixel in each scale feature map includes the position information of the remaining pixels in the scale feature map other than its own pixel.

3. The method according to claim 1, characterized in that, The scale fusion process between feature maps of different scales includes: Determine the resolution value of each scale feature map; and based on the time series corresponding to each scale feature map, perform scale fusion processing on two adjacent scale feature maps with different resolution values.

4. The method according to claim 1, characterized in that, After determining the corresponding detection regions in the invoice image to be recognized, and before performing text recognition based on the adjusted detection regions, the method further includes: If an occluded area to be detected is detected, then the occluded area to be detected is converted to a different color space. A mask region is generated using a target hue threshold, and the mask region is removed from the color space conversion result. The text recognition based on the adjusted detection regions includes: Text recognition is performed based on each detection region after adjustment and removal of the masked area.

5. The method according to claim 1, characterized in that, After performing text recognition based on the adjusted detection regions to obtain the recognition result of the invoice image to be recognized, the method further includes: The recognition results are validated based on preset field formats and inter-field logic. If the verification result is unsuccessful, a manual review reminder message will be generated.

6. A device for recognizing invoice information, characterized in that, The device includes: The scale feature processing module is used to determine the multi-scale feature map of the invoice image to be identified; and to perform pixel position interaction processing on each scale feature map and scale fusion processing between the various scale feature maps to obtain the category label of each field in the invoice image to be identified and the bounding box position information of each field; wherein, the category label includes at least the invoice title, invoice code, invoice number, invoice date, and tax amount; The region determination module is used to determine the corresponding detection regions in the invoice image to be identified based on the category labels of each field and the bounding box position information of each field. The region adjustment module is used to perform convolution processing on each region to be detected to obtain the offset of each region; and adjust the corresponding region to be detected based on each offset. The recognition result module is used to perform text recognition based on the adjusted detection areas to obtain the recognition result of the invoice image to be recognized.

7. The apparatus according to claim 6, characterized in that, The scale feature processing module is specifically used for: The target encoder encodes each scale feature map separately, and an attention mechanism is used to process each encoding result so that the position information of each pixel in each scale feature map includes the position information of the remaining pixels in the scale feature map other than its own pixel.

8. The apparatus according to claim 6, characterized in that, The scale feature processing module is specifically used for: Determine the resolution value of each scale feature map; and based on the time series corresponding to each scale feature map, perform scale fusion processing on two adjacent scale feature maps with different resolution values.

9. The apparatus according to claim 6, characterized in that, The region adjustment module is also used for: If an occluded area to be detected is detected, then the occluded area to be detected is converted to a different color space. A mask region is generated using a target hue threshold, and the mask region is removed from the color space conversion result. The recognition result module is specifically used for: Text recognition is performed based on each detection region after adjustment and removal of the masked area.

10. An electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1-5.

Images