Invoice image intelligent identification method and device, equipment and storage medium
Through the deep learning model, the problem of difficulty in identifying low-quality images is solved, and the automatic recognition and processing of invoice images is realized, and the recognition accuracy and efficiency are improved.
Patent Information
- Application Number
- CN202411210634.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-12-03
AI Technical Summary
The prior art is difficult to process low-quality, distorted and rotated invoice images, resulting in poor invoice recognition effect, increasing the cost of manual entry and data consistency problems, lacking automation means, making it difficult to achieve large-scale automatic extraction and processing of invoice information.
Deep learning model is used to detect invoice key points, split images, detect integrity, rotate text recognition boxes, input text recognition models, extract seal and QR code information, generate structured data, and realize automatic splitting, correction and invoice image Identify.
It improves the recognition accuracy and processing efficiency of low-quality invoice images, reduces manual intervention, realizes the automated processing of invoice images, and improves the accuracy and consistency of data.
Smart Images

Figure CN119068488A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to an invoice image intelligent recognition method, device, equipment and storage medium. Background Art
[0002] With the continuous development of artificial intelligence technology, deep learning has been widely used in the fields of image processing and text recognition. Traditional invoice recognition methods mainly rely on optical character recognition (OCR) technology, which requires high-quality input images and usually requires horizontal image data with less noise. However, in actual scenarios, the shooting conditions of invoice images are often poor, and most of the images have problems such as distortion, rotation, and blur, resulting in poor recognition effect of general OCR algorithms on invoice data, and ultimately unsatisfactory structured data extraction effect.
[0003] In the medical, insurance, and financial industries, invoice processing is a critical task. There are a large number of paper invoices in these industries. Manually entering these invoice information is time-consuming and error-prone, making it difficult to meet the needs of efficient and accurate information processing. The existing invoice processing system has the following shortcomings:
[0004] Image quality issues: Due to factors such as shooting angle and lighting conditions, the quality of invoice images varies, and there are problems such as distortion, rotation, and blur. Traditional OCR technology has difficulty processing these low-quality images.
[0005] High manual processing costs: Relying on manual invoice information entry is not only inefficient but also prone to errors, increasing operating costs.
[0006] Poor data consistency: During the manual data entry process, different operators may produce inconsistent data entry results, affecting the accuracy and reliability of the data.
[0007] Lack of automation: The existing system has a low level of automation, making it difficult to automatically extract and process invoice information on a large scale. Summary of the invention
[0008] The main purpose of the present invention is to provide a method, device, equipment and storage medium for intelligent recognition of invoice images, aiming to solve the technical problem of inaccurate recognition of low-quality invoice images in the prior art.
[0009] To achieve the above object, the present invention provides an invoice image intelligent recognition method, which comprises the following steps:
[0010] Acquire an image to be identified, and use a first deep learning model to detect invoice key points in the image to be identified. When it is detected that the image to be identified contains multiple invoice images, split the image to be identified according to the invoice key points to form multiple independent invoice images;
[0011] Detect the integrity of each invoice image. When all invoice images are detected to be complete, use the second deep learning model to detect the text position in each invoice image and generate a text recognition box.
[0012] Determine the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image, and rotate each invoice image and the text recognition frame of each invoice image to a horizontal direction;
[0013] A text image of a text recognition frame of each invoice image is intercepted, and the text image is input into a text recognition model to obtain a text recognition result of each invoice image.
[0014] Optionally, the detecting the integrity of each invoice image includes:
[0015] Detect whether all the preset key information of each invoice image exists, and use edge detection algorithm to check whether there is edge loss in each invoice image;
[0016] If all the preset key information of each invoice image exists and there is no edge loss, or all the preset key information of each invoice image exists and the edge loss does not affect the preset key information, then each invoice image is confirmed to be complete.
[0017] Optionally, determining the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image includes:
[0018] Selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and determining the rotation direction of each invoice image according to the text recognition confidence and aspect ratio of the target text recognition frames;
[0019] The rotation angle of each invoice image is determined from preset rotation angles according to the rotation direction of each invoice image.
[0020] Optionally, determining the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image includes:
[0021] Selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and calculating the main direction angle of each target text recognition frame using the boundary points of each target text recognition frame;
[0022] The rotation angle of each invoice image is obtained according to the main direction angle of each target text recognition frame and the text recognition confidence.
[0023] Optionally, after rotating each invoice image and the text recognition frame of each invoice image to a horizontal direction, the method further includes:
[0024] The third deep learning model is used to detect the seal image in each invoice image, and the seal image is enhanced. The enhanced seal image is recognized using the TROCR model, and the text information of the seal image is extracted, where the text information includes the unit name corresponding to each invoice.
[0025] Optionally, after extracting the text information in the seal image, the method further includes:
[0026] If there is an electronic invoice image in the invoice image, the fourth deep learning model is used to identify the invoice QR code of the electronic invoice image, the image contrast of the invoice QR code is enhanced, the enhanced invoice QR code is decoded using a QR code decoding algorithm, and the electronic bill detailed information of the electronic invoice is extracted.
[0027] Optionally, after extracting the electronic bill details of the electronic invoice, it also includes:
[0028] Use the information extraction model and the preset traditional extraction method to extract information from each invoice image to obtain structured data, which includes key fields and detailed items;
[0029] According to preset rules, the text recognition results of each invoice image, the text information in the seal image, the electronic invoice details and structured data are integrated to generate the final invoice result for each invoice image.
[0030] Furthermore, in order to achieve the above-mentioned purpose, the present invention also provides an invoice image intelligent recognition device, the invoice image intelligent recognition device comprising:
[0031] An invoice detection and splitting module obtains an image to be identified, and uses a first deep learning model to detect invoice key points in the image to be identified. When it is detected that the image to be identified contains multiple invoice images, the image to be identified is split according to the invoice key points to form multiple independent invoice images;
[0032] An invoice integrity detection module detects the integrity of each invoice image. When all invoice images are detected to be complete, a second deep learning model is used to detect the text position in each invoice image and generate a text recognition box.
[0033] An image direction correction module determines the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image, and rotates each invoice image and the text recognition frame of each invoice image to a horizontal direction;
[0034] The image text recognition module captures the text image of the text recognition frame of each invoice image, inputs the text image into the text recognition model, and obtains the text recognition result of each invoice image.
[0035] Furthermore, to achieve the above-mentioned purpose, the present invention also provides an invoice image intelligent recognition device, which includes a memory, a processor, and an invoice image intelligent recognition program stored in the memory and executable on the processor, and the invoice image intelligent recognition program, when executed by the processor, implements the steps of the invoice image intelligent recognition method as described above.
[0036] Furthermore, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which is stored an invoice image intelligent recognition program, and when the invoice image intelligent recognition program is executed by a processor, the steps of the invoice image intelligent recognition method described above are implemented.
[0037] The present invention relates to an intelligent invoice image recognition method, which obtains an image to be recognized, uses a first deep learning model to detect invoice key points in the image, and when the image contains multiple invoices, splits them according to the key points to form multiple independent invoice images; detects the integrity of each invoice image, uses a second deep learning model to detect the text position, and generates a text recognition frame; determines the rotation angle according to the text recognition confidence of the text recognition frame, rotates the invoice image and the text recognition frame to the horizontal direction; intercepts the text picture of the text recognition frame, inputs it into the text recognition model, and obtains the text recognition result. By using the deep learning model, the present invention can effectively process low-quality, rotated and distorted invoice images, improves the recognition accuracy and processing efficiency of low-quality invoice images, and realizes the automatic splitting, correction and recognition of invoice images, reduces manual intervention, and improves processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 This is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the invoice image intelligent recognition device of the present invention;
[0039] Figure 2 This is a flow chart of the first embodiment of the invoice image intelligent recognition method of the present invention;
[0040] Figure 3 This is a schematic diagram of the functional modules of a preferred embodiment of the invoice image intelligent recognition device of the present invention.
[0041] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0042] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0043] It should be noted that with the continuous development of artificial intelligence technology, deep learning has been widely used in the fields of image processing and text recognition. Traditional invoice recognition methods mainly rely on optical character recognition (OCR) technology, which requires high-quality input images and usually requires horizontal image data with less noise. However, in actual scenarios, the shooting conditions of invoice images are often poor, and most of the images have problems such as distortion, rotation, and blur, resulting in poor recognition effect of general OCR algorithms on invoice data, and ultimately unsatisfactory structured data extraction effect.
[0044] In the medical, insurance, and financial industries, invoice processing is a critical task. There are a large number of paper invoices in these industries. Manually entering these invoice information is time-consuming and error-prone, making it difficult to meet the needs of efficient and accurate information processing. The existing invoice processing system has the following shortcomings:
[0045] Image quality issues: Due to factors such as shooting angle and lighting conditions, the quality of invoice images varies, and there are problems such as distortion, rotation, and blur. Traditional OCR technology has difficulty processing these low-quality images.
[0046] High manual processing costs: Relying on manual invoice information entry is not only inefficient but also prone to errors, increasing operating costs.
[0047] Poor data consistency: During the manual data entry process, different operators may produce inconsistent data entry results, affecting the accuracy and reliability of the data.
[0048] Lack of automation: The existing system has a low level of automation, making it difficult to automatically extract and process invoice information on a large scale.
[0049] Based on the above-mentioned defects, the present invention provides an invoice image intelligent recognition device, referring to Figure 1 , Figure 1 This is a schematic diagram of the structure of the device hardware operating environment involved in the embodiment of the invoice image intelligent recognition device of the present invention.
[0050] like Figure 1As shown, the invoice image intelligent recognition device may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0051] Those skilled in the art will understand that Figure 1 The hardware structure of the invoice image intelligent recognition device shown in the figure does not constitute a limitation of the invoice image intelligent recognition device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0052] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and an invoice image intelligent recognition program. The operating system is a program for managing and controlling the invoice image intelligent recognition device and software resources, and supports the operation of the network communication module, the user interface module, the invoice image intelligent recognition program, and other programs or software; the network communication module is used to manage and control the network interface 1004; and the user interface module is used to manage and control the user interface 1003.
[0053] exist Figure 1 In the hardware structure of the invoice image intelligent recognition device shown, the network interface 1004 is mainly used to connect to the backend server and perform data communication with the backend server; the user interface 1003 is mainly used to connect to the client and perform data communication with the client; the processor 1001 can call the invoice image intelligent recognition program stored in the memory 1005 and perform the following operations:
[0054] Acquire an image to be identified, and use a first deep learning model to detect invoice key points in the image to be identified. When it is detected that the image to be identified contains multiple invoice images, split the image to be identified according to the invoice key points to form multiple independent invoice images;
[0055] Detect the integrity of each invoice image. When all invoice images are detected to be complete, use the second deep learning model to detect the text position in each invoice image and generate a text recognition box.
[0056] Determine the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image, and rotate each invoice image and the text recognition frame of each invoice image to a horizontal direction;
[0057] A text image of a text recognition frame of each invoice image is intercepted, and the text image is input into a text recognition model to obtain a text recognition result of each invoice image.
[0058] Furthermore, the detecting the integrity of each invoice image includes:
[0059] Detect whether all the preset key information of each invoice image exists, and use edge detection algorithm to check whether there is edge loss in each invoice image;
[0060] If all the preset key information of each invoice image exists and there is no edge loss, or all the preset key information of each invoice image exists and the edge loss does not affect the preset key information, then each invoice image is confirmed to be complete.
[0061] Furthermore, the step of determining the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image includes:
[0062] Selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and determining the rotation direction of each invoice image according to the text recognition confidence and aspect ratio of the target text recognition frames;
[0063] The rotation angle of each invoice image is determined from preset rotation angles according to the rotation direction of each invoice image.
[0064] Furthermore, the step of determining the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image includes:
[0065] Selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and calculating the main direction angle of each target text recognition frame using the boundary points of each target text recognition frame;
[0066] The rotation angle of each invoice image is obtained according to the main direction angle of each target text recognition frame and the text recognition confidence.
[0067] Furthermore, after rotating each invoice image and the text recognition frame of each invoice image to a horizontal direction, the method further includes:
[0068] The third deep learning model is used to detect the seal image in each invoice image, and the seal image is enhanced. The enhanced seal image is recognized using the TROCR model, and the text information of the seal image is extracted, where the text information includes the unit name corresponding to each invoice.
[0069] Furthermore, after extracting the text information in the seal image, the method further includes:
[0070] If there is an electronic invoice image in the invoice image, the fourth deep learning model is used to identify the invoice QR code of the electronic invoice image, the image contrast of the invoice QR code is enhanced, the enhanced invoice QR code is decoded using a QR code decoding algorithm, and the electronic bill detailed information of the electronic invoice is extracted.
[0071] Furthermore, after extracting the electronic bill details of the electronic invoice, it also includes:
[0072] Use the information extraction model and the preset traditional extraction method to extract information from each invoice image to obtain structured data, which includes key fields and detailed items;
[0073] According to preset rules, the text recognition results of each invoice image, the text information in the seal image, the electronic invoice details and structured data are integrated to generate the final invoice result for each invoice image.
[0074] The specific implementation of the invoice image intelligent recognition device of the present invention is basically the same as the embodiments of the invoice image intelligent recognition method described below, and will not be repeated here.
[0075] The present invention also provides an invoice image intelligent recognition method based on the invoice image intelligent recognition device.
[0076] Reference Figure 2 , Figure 2 This is a flow chart of the first embodiment of the invoice image intelligent recognition method of the present invention.
[0077] The embodiment of the present invention provides an embodiment of the method for intelligent recognition of invoice images. It should be noted that although the logical order is shown in the flow chart, in some cases, the steps shown or described may be executed in a different order from that shown here.
[0078] In various embodiments of the invoice image intelligent recognition method, the execution entity is an invoice image intelligent recognition device.
[0079] The invoice image intelligent recognition method is applied to an invoice image intelligent recognition device, and the invoice image intelligent recognition method includes:
[0080] Step S10, obtaining an image to be identified, using a first deep learning model to detect invoice key points in the image to be identified, and when it is detected that the image to be identified contains multiple invoice images, splitting the image to be identified according to the invoice key points to form multiple independent invoice images;
[0081] With the continuous development of artificial intelligence technology, deep learning has been widely used in the fields of image processing and text recognition. Traditional invoice recognition methods mainly rely on optical character recognition (OCR) technology, which requires high-quality input images and usually requires horizontal image data with less noise. However, in actual scenarios, the shooting conditions of invoice images are often poor, and most of the images have problems such as distortion, rotation, and blur, resulting in poor recognition effect of general OCR algorithms on invoice data, and ultimately unsatisfactory structured data extraction effect.
[0082] In the medical, insurance, and financial industries, invoice processing is a critical task. There are a large number of paper invoices in these industries. Manually entering these invoice information is time-consuming and error-prone, making it difficult to meet the needs of efficient and accurate information processing. The existing invoice processing system has the following shortcomings:
[0083] Image quality issues: Due to factors such as shooting angle and lighting conditions, the quality of invoice images varies, and there are problems such as distortion, rotation, and blur. Traditional OCR technology has difficulty processing these low-quality images.
[0084] High manual processing costs: Relying on manual invoice information entry is not only inefficient but also prone to errors, increasing operating costs.
[0085] Poor data consistency: During the manual data entry process, different operators may produce inconsistent data entry results, affecting the accuracy and reliability of the data.
[0086] Lack of automation: The existing system has a low level of automation, making it difficult to automatically extract and process invoice information on a large scale.
[0087] In this embodiment, a high-resolution camera or scanner may be used to obtain an image containing one or more invoices. The image should be as clear and evenly lit as possible during the acquisition.
[0088] The first deep learning model can be a variety of models for image detection and key point recognition, common ones include:
[0089] Convolutional Neural Networks (CNN): Such as ResNet, VGG, etc. These networks can extract features of images.
[0090] Object detection models: such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), Faster R-CNN, etc. These models can detect objects and their boundaries in images.
[0091] Keypoint detection models: such as OpenPose, Hourglass, keypoints-net, etc. These models are specifically used to detect key points in images.
[0092] These models can be trained with a training dataset to identify key points in invoice images.
[0093] The acquired images are fed into a pre-trained deep learning model. The model is able to automatically detect the key points of each invoice in the image.
[0094] Invoice key points refer to some important feature points in the invoice image. These key points help locate and identify the boundaries and content of the invoice. Common invoice key points include:
[0095] Four corner points: The four vertices of the invoice (upper left corner, upper right corner, lower left corner, lower right corner), used to determine the boundaries of the invoice.
[0096] Text area: The area containing important text information on the invoice, such as invoice number, date, amount, etc.
[0097] QR code area: The area on the invoice where the QR code is located, used to extract electronic information.
[0098] Seal area: The area on the invoice where the seal is located, used to verify the legitimacy of the invoice.
[0099] These key points help the system recognize and process invoice images, ensuring that subsequent text recognition and information extraction are accurate and effective.
[0100] The acquired image is input into a pre-trained deep learning model (such as YOLO, SSD, Faster R-CNN, etc.). The model extracts image features through a convolutional neural network and outputs the coordinates of the four corner points, text area, QR code area, and seal area of the invoice through a fully connected layer. Each invoice area in the image is cut according to the detected key points. When cutting, a certain amount of edge extension is considered to ensure the integrity of the invoice.
[0101] Use image processing algorithms (such as cropping and affine transformation) to segment each invoice area into independent images. Output the segmented independent invoice images and save them in a standard format (such as JPEG, PNG).
[0102] Each individual invoice image will be used as input for subsequent processing for further text recognition and information extraction.
[0103] If the image to be identified contains only one invoice image, the image to be identified is not split and the entire invoice image is directly processed.
[0104] Example description:
[0105] Assuming that the input image contains three invoices, the specific implementation steps can be as follows:
[0106] Image acquisition: Use a scanner to scan a paper document containing three invoices to obtain a high-resolution image.
[0107] Invoice key point detection: Input the image into a pre-trained deep learning model. The model detects the four corner points of each invoice and outputs the key point coordinates.
[0108] Image segmentation: Based on the key point coordinates, the image is cropped and affine transformed to obtain independent invoice images. The cropped area of each invoice is accurately located and segmented based on the detected key points.
[0109] Output result: Three independent invoice images are obtained as input for subsequent processing.
[0110] Through the above steps, the image containing multiple invoices can be effectively segmented into independent invoice images, which is convenient for subsequent recognition and information extraction. This method greatly improves the efficiency and accuracy of invoice image processing and overcomes the shortcomings of the existing technology in processing low-quality images.
[0111] Step S20, detecting the integrity of each invoice image. When it is detected that all invoice images are complete, using the second deep learning model to detect the text position in each invoice image and generate a text recognition frame;
[0112] In this embodiment, each invoice image is checked to ensure that all necessary key information (such as invoice number, date, amount, etc.) exists. Only when all invoice images are confirmed to be intact, the next step of processing is carried out. The edge detection algorithm is used to check whether the invoice image has edge loss. If the key information is complete and there is no edge loss, the invoice image is determined to be complete.
[0113] Each invoice image is fed into the second deep learning model. The model identifies the text location in the invoice image and generates a text recognition box to mark the text area in preparation for text recognition.
[0114] The second deep learning model is mainly used to detect the text position in the invoice image and generate the text recognition box. The following are some commonly used deep learning models that can be used for text position detection:
[0115] EAST (Efficient and Accurate Scene Text Detector): EAST is an efficient and accurate scene text detector that can directly output the coordinates of the four vertices of the text box and is suitable for detecting text in any direction.
[0116] CTPN (Connectionist Text Proposal Network): CTPN is a text detection method based on convolutional neural networks that can generate high-quality text candidate regions and construct text lines by connecting text blocks.
[0117] DBNet (Differentiable Binarization Network): DBNet separates the text area and the background area through a differentiable binarization module, which can effectively detect the text area and is particularly suitable for text detection in complex scenes.
[0118] PSENet (Progressive Scale Expansion Network): PSENet can detect text of different scales by gradually expanding the text area, which is very suitable for processing long text and multi-line text.
[0119] CRAFT (Character Region Awareness for Text detection): CRAFT can detect character regions and predict the connectivity of each character, which is particularly suitable for detecting small characters in complex backgrounds.
[0120] TextBoxes++: TextBoxes++ is an improved text detection model that performs well on text with high aspect ratios and is suitable for detecting text in natural scenes.
[0121] A specific implementation method may be: Check each invoice image to ensure that all necessary key information (such as invoice number, date, amount, etc.) is present. Use edge detection algorithm to check whether the invoice image has edge loss. If the key information is complete and there is no edge loss, the invoice image is determined to be complete. Input the complete invoice image into a pre-trained second deep learning model (such as EAST, CTPN, DBNet, etc.). The model automatically detects the text position in the invoice image and generates a text recognition box.
[0122] When an incomplete invoice image is detected, the following measures can be taken to process the incomplete invoice image to ensure that the subsequent recognition and extracted information is accurate and reliable:
[0123] Record incomplete invoices: Mark and record incomplete invoice images for subsequent manual review and supplementation.
[0124] Re-capture image: Prompts the user to retake or rescan an incomplete invoice image, if possible, to ensure the image is complete.
[0125] Partial identification and extraction: For invoices that are incomplete but still have some information available, the system can try to extract the available key information and prompt the user to supplement the missing information.
[0126] Completion algorithm: Try to repair and complete incomplete invoice images through image processing technology, such as using image restoration algorithms to fill in missing parts.
[0127] Notify users: When the system detects an incomplete invoice, it automatically notifies users that the invoice is incomplete and provides solutions or suggestions.
[0128] Example description:
[0129] Assuming the input image contains two invoices, the specific implementation steps are as follows:
[0130] Completeness Check: Perform a completeness check on each invoice image to confirm that the invoice image contains all necessary information without any edge loss.
[0131] Text location detection: The confirmed complete invoice image is input into the pre-trained DBNet model.
[0132] The model detects the location of text in the invoice and generates a text recognition box:
[0133] Invoice 1 text recognition box: [(60,110,400,80),(60,200,400,50),...]
[0134] Invoice 2 text recognition box: [(520,160,380,70),(520,260,380,50),...]
[0135] Through the above steps, the integrity of each invoice image is ensured, and a text recognition frame for text recognition is generated, providing a reliable basis for subsequent text recognition and information extraction.
[0136] Step S30, determining the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image, and rotating each invoice image and the text recognition frame of each invoice image to a horizontal direction;
[0137] In this embodiment, the text recognition confidence in the text recognition frame is used to determine the rotation direction and angle of the invoice image. Text with high confidence usually indicates that its direction is correct. The rotation angle of the invoice image is determined by analyzing the aspect ratio and confidence of the text recognition frame. The identified rotation angle is applied to the invoice image and the text recognition frame to rotate them to the correct horizontal position, which is convenient for subsequent text recognition and information extraction.
[0138] A specific implementation method may be: using a deep learning model to detect the text position in the invoice image and generate a text recognition frame that marks the position and size of all text areas in the image.
[0139] Recognize the text in the text recognition box and calculate the recognition confidence of each text. The confidence indicates the reliability of the recognition result. The direction of text with high confidence is usually correct.
[0140] Analyze the aspect ratio and recognition confidence of the text recognition frame to determine the rotation angle of the image. If the angle between the long side of the text recognition frame and the horizontal line is small and the confidence is high, the direction of the text is correct. By counting the directions of most text recognition frames, the rotation angle of the entire invoice image is determined.
[0141] According to the determined rotation angle, the invoice image and the text recognition frame are rotated to a horizontal direction. The rotation operation makes the text in the invoice image horizontal, which is convenient for subsequent text recognition and information extraction.
[0142] Example description:
[0143] Assuming the input image contains an invoice with an unknown rotation angle, the specific implementation steps are as follows:
[0144] Text recognition box detection: Use the deep learning model to detect the text position in the invoice image and generate multiple text recognition boxes: Text recognition box: [(60,110,400,80),(60,200,400,50),...].
[0145] Text recognition and confidence calculation: Recognize the text in each text recognition box and calculate the recognition confidence: Text recognition results and confidence: [("invoice number", 0.95), ("date", 0.90), ...].
[0146] Determine the rotation angle: Analyze the aspect ratio and recognition confidence of the text recognition frame to determine the image rotation angle: The angle between the long side of the text recognition frame and the horizontal line is small, and the confidence is high, indicating that the text direction is correct. Count the directions of most text recognition frames and determine that the invoice image rotation angle is 90°.
[0147] Image rotation: Rotate the invoice image and text recognition frame to a horizontal direction according to the determined 90° rotation angle. After rotation, the text in the invoice image becomes horizontal, which is convenient for subsequent text recognition and information extraction.
[0148] Through the above steps, the rotation angle of the invoice image can be effectively determined, and the image can be rotated to the correct horizontal position, providing an accurate basis for subsequent recognition and information extraction.
[0149] Step S40, capturing a text image of a text recognition frame of each invoice image, inputting the text image into a text recognition model, and obtaining a text recognition result of each invoice image.
[0150] In this embodiment, the text recognition frame generated by the deep learning model in each invoice image is cropped to capture the text area within the frame. The captured text image is used as input and passed to the pre-trained text recognition model. The text recognition model processes the input text image and outputs the recognition result, that is, the text information in the invoice.
[0151] A specific implementation method may be: using a deep learning model to detect the text position in the invoice image and generate a text recognition frame. The text recognition frame marks the position and size of all text areas in the image. According to the coordinates of the text recognition frame, the invoice image is cropped, the text area in the frame is intercepted, and multiple text images are generated. The intercepted text images are input into a pre-trained text recognition model, and the text recognition model processes each text image and outputs a recognition result. The recognition results of all text images are summarized to obtain the text information of the entire invoice.
[0152] Example description:
[0153] Assuming the input image contains an invoice, the implementation steps are as follows:
[0154] Text recognition box detection: Use the deep learning model to detect the text position in the invoice image and generate multiple text recognition boxes: Text recognition box: [(60,110,400,80), (60,200,400,50),...]
[0155] Capture text images: Crop the invoice image according to the coordinates of the text recognition box, capture the text area within the box, and generate multiple text images.
[0156] Text recognition: Input the captured text image into a pre-trained text recognition model (such as CRNN, Tesseract, Attention-OCR, etc.).
[0157] The text recognition model processes each text image and outputs the recognition result:
[0158] Text recognition result: ["Invoice number: 12345678", "Date: 2023-01-01",...]
[0159] Summarize recognition results: Summarize the recognition results of all text images to obtain the text information of the entire invoice:
[0160] Invoice text recognition results:
[0161] Invoice number:12345678
[0162] Date:2023-01-01
[0163] Amount:¥100.00 ...
[0165] Through the above steps, the text information in the invoice image can be accurately recognized, and a complete text recognition result can be output, thereby improving the accuracy of text recognition.
[0166] The present invention relates to an intelligent invoice image recognition method, which obtains an image to be recognized, uses a first deep learning model to detect invoice key points in the image, and when the image contains multiple invoices, splits them according to the key points to form multiple independent invoice images; detects the integrity of each invoice image, uses a second deep learning model to detect the text position, and generates a text recognition frame; determines the rotation angle according to the text recognition confidence of the text recognition frame, rotates the invoice image and the text recognition frame to the horizontal direction; intercepts the text picture of the text recognition frame, inputs it into the text recognition model, and obtains the text recognition result. By using the deep learning model, the present invention can effectively process low-quality, rotated and distorted invoice images, improves the recognition accuracy and processing efficiency of low-quality invoice images, and realizes the automatic splitting, correction and recognition of invoice images, reduces manual intervention, and improves processing efficiency.
[0167] Furthermore, a second embodiment of the invoice image intelligent recognition method of the present invention is proposed. In the above step S20, the integrity of each invoice image is detected, including:
[0168] Step a1, detecting whether all the preset key information of each invoice image exists, and using an edge detection algorithm to check whether there is edge loss in each invoice image;
[0169] Step a2: If all the preset key information of each invoice image exists and there is no edge loss, or if all the preset key information of each invoice image exists and the edge loss does not affect the preset key information, then each invoice image is confirmed to be complete.
[0170] In this embodiment, a key information check is performed on each invoice image to ensure that preset key information such as invoice number, date, amount, etc. exists.
[0171] Preset key information refers to important fields and information that must be present on an invoice. This information is used to verify the legitimacy and integrity of the invoice. Specifically, it may include:
[0172] Invoice Number: A number that uniquely identifies an invoice for record keeping and tracking purposes.
[0173] Date: The date the invoice was issued, recording when the transaction occurred.
[0174] Amount: The transaction amount recorded on the invoice, usually including the total amount and tax.
[0175] Taxpayer Identification Number: The taxpayer identification number of the buyer and seller for tax identification purposes.
[0176] Invoicing unit name: the name of the unit or company that issues the invoice.
[0177] Purchaser’s name: The name of the organization or individual purchasing the goods or services.
[0178] Product or service details: including product name, quantity, unit price, total price and other specific transaction details.
[0179] QR code: The QR code on the electronic invoice is used to verify and read electronic information.
[0180] Seal: The seal of the issuing unit, used to verify the authenticity of the invoice.
[0181] Use edge detection algorithms to perform edge detection on each invoice image to identify if there is edge loss or damage.
[0182] If all the preset type key information of each invoice image is present and there is no edge loss, the invoice image is confirmed to be complete.
[0183] If all the key information of the preset type of each invoice image exists but there is edge loss, it is necessary to further determine whether the edge loss affects the key information. If not, the invoice image is confirmed to be complete.
[0184] Example description:
[0185] Assuming the input image contains two invoices, the specific implementation steps are as follows:
[0186] Check that the preset type key information of each invoice image is complete:
[0187] Invoice 1: Complete information including invoice number, date, amount, etc.
[0188] Invoice 2: Invoice number, date, amount and other information are complete.
[0189] Edge detection: Perform edge detection on each invoice image to check if there is any loss on the edge:
[0190] Invoice 1: No loss in edge detection results;
[0191] Invoice 2: The edge detection result has slight edge loss.
[0192] Completeness confirmation:
[0193] For invoice 1, the key information is complete and there is no edge loss, confirming that the invoice image is complete.
[0194] For invoice 2, the key information is complete and the edge loss does not affect the key information, confirming that the invoice image is complete.
[0195] Through the above steps, this embodiment can effectively detect and confirm the integrity of the invoice image and ensure the accuracy and completeness of key information.
[0196] Furthermore, a third embodiment of the invoice image intelligent recognition method of the present invention is proposed. In the above step S30, the rotation angle of each invoice image is determined according to the text recognition confidence of the text recognition frame of each invoice image, including:
[0197] Step b1, selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and determining the rotation direction of each invoice image according to the text recognition confidence and aspect ratio of the target text recognition frames;
[0198] Step b2: determining the rotation angle of each invoice image from preset rotation angles according to the rotation direction of each invoice image.
[0199] In this embodiment, among the text recognition frames of each invoice image, several target text recognition frames with the highest text recognition confidence are selected. The rotation direction of the invoice image is determined based on the text recognition confidence and aspect ratio of these target text recognition frames. Based on the determined rotation direction, the specific rotation angle of the invoice image is determined from the preset rotation angles.
[0200] A specific implementation method may be: using a deep learning model to detect the text position in the invoice image and generate a text recognition frame that marks the position and size of all text areas in the image.
[0201] Recognize the text in the text recognition box and calculate the recognition confidence of each text. The confidence indicates the reliability of the recognition result. Text with high confidence usually indicates that its direction is correct.
[0202] From the text recognition frames of each invoice image, select the first several target text recognition frames with the highest text recognition confidence.
[0203] The rotation direction of each invoice image is determined based on the aspect ratio of the selected target text recognition box and its text recognition confidence. If the angle between the long side of the text recognition box and the horizontal line is small and the confidence is high, the direction of the text is correct.
[0204] According to the determined rotation direction, the specific rotation angle of each invoice image is determined from preset rotation angles (such as 0°, 90°, 180°, 270°).
[0205] Example description:
[0206] Assuming the input image contains an invoice with an unknown rotation angle, the specific implementation steps are as follows:
[0207] Text recognition box detection: Use the deep learning model to detect the text position in the invoice image and generate multiple text recognition boxes: Text recognition box: [(60,110,400,80),(60,200,400,50),...].
[0208] Text recognition and confidence calculation: Recognize the text in each text recognition box and calculate the recognition confidence: Text recognition results and confidence: [("invoice number", 0.95), ("date", 0.90), ...].
[0209] Select target text recognition boxes: Select the first several target text recognition boxes with the highest text recognition confidence from the text recognition boxes: Selected target text recognition boxes: [(60,110,400,80),(60,200,400,50)].
[0210] Determine the rotation direction: Determine the rotation direction of the invoice image based on the aspect ratio of the selected target text recognition box and its text recognition confidence: the angle between the long side of the text recognition box and the horizontal line is small, and the confidence is high, indicating that the text direction is correct. Determine that the invoice image needs to be rotated 90° clockwise.
[0211] Determine the rotation angle: According to the determined rotation direction, determine the rotation angle of the invoice image as 90° from the preset rotation angles.
[0212] Through the above steps, this embodiment can accurately determine the rotation angle of the invoice image and rotate the image to a correct horizontal position to ensure the accuracy of text recognition.
[0213] Furthermore, a fourth embodiment of the invoice image intelligent recognition method of the present invention is proposed. In the above step S30, the rotation angle of each invoice image is determined according to the text recognition confidence of the text recognition frame of each invoice image, including:
[0214] Step c1, selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and calculating the main direction angle of each target text recognition frame using the boundary points of each target text recognition frame;
[0215] Step c2, obtaining the rotation angle of each invoice image according to the main direction angle of each target text recognition frame and the text recognition confidence.
[0216] In this embodiment, among the text recognition frames of each invoice image, several target text recognition frames with the highest text recognition confidence are selected. The main direction angle of each target text recognition frame is calculated using the boundary points of each target text recognition frame. The rotation angle of each invoice image is determined based on the main direction angle and text recognition confidence of each target text recognition frame.
[0217] A specific implementation method may be: using a deep learning model to detect the text position in the invoice image and generate a text recognition frame that marks the position and size of all text areas in the image.
[0218] Recognize the text in the text recognition box and calculate the recognition confidence of each text. The confidence indicates the reliability of the recognition result. Text with high confidence usually indicates that its direction is correct.
[0219] From the text recognition frames of each invoice image, select the first several target text recognition frames with the highest text recognition confidence.
[0220] Using the boundary points of each target text recognition box, the main direction angle of each target text recognition box is calculated. The main direction angle refers to the angle between the long side of the text recognition box and the horizontal line.
[0221] The rotation angle of the invoice image is comprehensively determined based on the main direction angle of each target text recognition frame and its text recognition confidence. The final rotation angle is determined and applied to the invoice image and text recognition frame to rotate them to a horizontal position.
[0222] To calculate the main direction angle of each target text recognition box, the coordinates of the boundary points can be used to determine the rotation angle of the text box. Assume that the coordinates of the boundary points of each text recognition box are (x 1 ,y 1 )、(x 2 ,y 2 )、(x 3 ,y 3 ) and (x4 ,y 4 ), where (x 1 ,y 1 )、(x 2 ,y 2 ) is the upper boundary of the box, (x 3 ,y 3 ) and (x 4 ,y 4 ) is the lower border of the box.
[0223] The main direction angle θ can be determined by calculating the direction angle of the upper boundary segment:
[0224]
[0225] Where: θ is the main direction angle of the text recognition frame; (x 1 ,y 1 )、(x 2 ,y 2 ) are the coordinates of the two vertices on the upper boundary of the text recognition box.
[0226] The first several (eg, 10) target text recognition frames with the highest text recognition confidence are selected from the text recognition frames.
[0227] For each target text recognition box, calculate its main direction angle:
[0228]
[0229] θ i represents the main direction angle of the i-th target text recognition box, (x 1i ,y 1i )(x 2i ,y 2i ) are the coordinates of the two vertices on the upper boundary of the i-th target text recognition frame. By combining the main direction angles of all target text recognition frames, the final rotation angle of the invoice image is determined:
[0230]
[0231] Where N is the number of selected target text recognition boxes.
[0232] Example description:
[0233] Suppose there is an invoice image with two target text recognition boxes, with upper boundary point coordinates of (60,110) and (460,160), and (60,200) and (460,250). This means that the invoice image needs to be rotated.
[0234] Specific implementation steps may include:
[0235] Text recognition box detection: Use the deep learning model to detect the text position in the invoice image and generate multiple text recognition boxes: Text recognition box: [(60,110,400,50),(60,200,400,50),...].
[0236] Text recognition and confidence calculation: Recognize the text in each text recognition box and calculate the recognition confidence: Text recognition results and confidence: [("invoice number", 0.95), ("date", 0.90), ...].
[0237] Select the target text recognition box: Select the first several target text recognition boxes with the highest text recognition confidence from the text recognition boxes: Selected target text recognition boxes: [(60,110,460,160), (60,200,460,250)].
[0238] For each target text recognition box, calculate its main direction angle:
[0239] The first text recognition box:
[0240]
[0241] The second text recognition box:
[0242]
[0243] Determine the rotation angle of the invoice image based on the main direction angles of the two target text recognition frames:
[0244]
[0245] According to the calculated rotation angle of 7.13°, the invoice image and text recognition frame are rotated to a horizontal position, so that they are rotated 7.13° to correctly recognize the text.
[0246] Through the above steps, this embodiment can accurately determine the rotation angle of the invoice image that needs to be rotated, and rotate it to the correct horizontal position to ensure the accuracy of text recognition.
[0247] Furthermore, a fifth embodiment of the invoice image intelligent recognition method of the present invention is proposed. In the above step S30, after rotating each invoice image and the text recognition frame of each invoice image to the horizontal direction, the method further includes:
[0248] Step d1, using the third deep learning model to detect the seal image in each invoice image, performing image enhancement processing on the seal image, using the TROCR model to recognize the enhanced seal image, and extracting the text information of the seal image, wherein the text information includes the unit name corresponding to each invoice.
[0249] In this embodiment, a third deep learning model (such as YOLO, Faster R-CNN, etc.) is used to detect the seal position in each invoice image and generate a seal recognition frame. The seal recognition frame marks the position and size of the seal area in the image.
[0250] The third deep learning model is mainly used to detect the seal position in the invoice image. Commonly used models include:
[0251] YOLO (You Only Look Once): YOLO is a real-time object detection system that can quickly and accurately detect the location of objects in an image.
[0252] Faster R-CNN (Region-based Convolutional Neural Networks): Faster R-CNN is a classic target detection model with high detection accuracy and is suitable for detecting targets in complex images.
[0253] RetinaNet: RetinaNet combines speed and accuracy and is an efficient model suitable for various object detection tasks.
[0254] EfficientDet: EfficientDet is an efficient and accurate object detection model that can reduce computational costs while ensuring accuracy.
[0255] Perform image enhancement processing on the detected seal image area to improve the clarity and contrast of the image. Commonly used image enhancement techniques include histogram equalization, contrast enhancement, denoising, etc.
[0256] The enhanced seal image is input into the pre-trained TROCR model for text recognition. The TROCR model is based on the Transformer structure and recognizes the text in the seal image.
[0257] Extract and output the text information in the seal image, usually including the company name corresponding to the invoice.
[0258] Example description:
[0259] Assuming the input image contains an invoice, the implementation steps are as follows:
[0260] Seal image detection: Use the YOLO model to detect the seal position in the invoice image and generate a seal recognition box: Seal recognition box: [(150, 300, 350, 500)].
[0261] Image enhancement processing: Perform image enhancement processing on the detected seal image area to improve the image clarity and contrast.
[0262] Seal image recognition: The enhanced seal image is input into the TROCR model for text recognition.
[0263] The TROCR model recognizes the text in the seal image and outputs the recognition result:
[0264] Seal text recognition result: "XX Company\nAddress: No. XX, XX Road\nRegistration Number: 12345678".
[0265] Extract text information: Extract and output text information from the seal image: "XX Company", "Address: No. XX, XX Road", "Registration Number: 12345678".
[0266] Through the above steps, this embodiment can accurately detect and identify the seal information in the invoice image, and extract key information such as the company name, thereby ensuring the integrity and accuracy of the invoice information.
[0267] Furthermore, a sixth embodiment of the invoice image intelligent recognition method of the present invention is proposed. In the above step d1, after extracting the text information in the seal image, the method further includes:
[0268] Step e1: If there is an electronic invoice image in the invoice image, use the fourth deep learning model to identify the invoice QR code of the electronic invoice image, enhance the image contrast of the invoice QR code, use the QR code decoding algorithm to decode the enhanced invoice QR code, and extract the electronic bill detailed information of the electronic invoice.
[0269] In this embodiment, a deep learning model (such as YOLO, Faster R-CNN, etc.) is used to detect whether the invoice image is an electronic invoice. If it is an electronic invoice, the following steps are continued.
[0270] The fourth deep learning model is mainly used to recognize the QR code in the invoice image, especially the QR code of the electronic invoice. Commonly used models include:
[0271] YOLO (You Only Look Once): YOLO is a real-time target detection system that can quickly and accurately detect the location of targets in images. It is suitable for detecting QR codes in invoices.
[0272] Faster R-CNN (Region-based Convolutional Neural Networks): Faster R-CNN is a classic object detection model with high detection accuracy and is suitable for detecting QR codes in complex images.
[0273] RetinaNet: RetinaNet combines speed and accuracy and is an efficient model suitable for various object detection tasks, capable of detecting QR codes in invoice images.
[0274] EfficientDet: EfficientDet is an efficient and accurate object detection model that can reduce computational costs while ensuring accuracy. It is suitable for detecting QR codes in invoices.
[0275] Use the fourth deep learning model (such as YOLO, Faster R-CNN, etc.) to detect the QR code position in the electronic invoice image and generate a QR code recognition frame. Perform image contrast enhancement processing on the detected QR code area to improve the clarity and contrast of the QR code for subsequent decoding. Use a QR code decoding algorithm (such as Zxing, ZBar, etc.) to decode the enhanced QR code and extract the electronic bill details of the electronic invoice.
[0276] Example description:
[0277] Assuming the input image contains an invoice, the implementation steps are as follows:
[0278] Electronic invoice recognition: Use the YOLO model to detect whether the invoice image is an electronic invoice. If it is an electronic invoice, proceed with the following steps:
[0279] Test results: Electronic invoice
[0280] Electronic invoice QR code detection: Use the YOLO model to detect the QR code position in the invoice image and generate a QR code recognition box: QR code recognition box: [(500, 300, 650, 450)].
[0281] Image contrast enhancement: Perform image contrast enhancement on the detected QR code area to improve the clarity and contrast of the QR code.
[0282] QR code decoding: Input the enhanced QR code image into the QR code decoding algorithm for decoding:
[0283] Decoding result: "Electronic invoice information: {invoice number: 12345678, date: 2023-01-01, amount: 100.00,...}".
[0284] Through the above steps, this embodiment can accurately identify the electronic invoice QR code in the invoice image, decode the QR code information, extract the detailed information of the electronic invoice, and ensure the integrity and accuracy of the invoice data.
[0285] Furthermore, a seventh embodiment of the method for intelligently identifying invoice images of the present invention is proposed. In the above step e1, after extracting the electronic bill detailed information of the electronic invoice, the method further includes:
[0286] Step f1, extract information from each invoice image using an information extraction model and a preset traditional extraction method to obtain structured data, where the structured data includes key fields and detailed items;
[0287] Step f2, according to preset rules, the text recognition results of each invoice image, the text information in the seal image, the electronic invoice details and the structured data are integrated to generate the final invoice result of each invoice image.
[0288] In this embodiment, information extraction models (such as models based on natural language processing) and traditional extraction methods (such as regular expressions, template matching, etc.) are used to extract information from invoice images. Key fields (such as invoice number, date, amount, etc.) and detailed items (such as product name, quantity, unit price, etc.) in the invoice image are extracted.
[0289] The extracted information is structured to generate structured data, ensuring that the data format is unified and easy to process.
[0290] According to the preset rules, the text recognition results, seal recognition information, electronic receipt information and structured data are processed comprehensively to ensure the integrity and accuracy of the information. The preset rules may include data verification, field matching, information integration and other steps.
[0291] All information is combined to generate the final bill result for each invoice image, ensuring that all key fields and detailed items are included.
[0292] Example description:
[0293] Assuming the input image contains an invoice, the implementation steps are as follows:
[0294] Information extraction: Use information extraction models and traditional extraction methods to extract key information from invoice images:
[0295] Key fields: invoice number, date, amount, etc.;
[0296] Detailed items: product name, quantity, unit price, etc.
[0297] Generate structured data: Structuring the extracted information:
[0298] Structured data:
[0299] {
[0300] "Invoice Number":"12345678",
[0301] "Date":"2023-01-01",
[0302] "Amount":"100.00",
[0303] "Details":[
[0304] {"Product Name":"Product A","Quantity":"1","Unit Price":"50.00"},
[0305] {"Product Name":"Product B","Quantity":"1","Unit Price":"50.00"} ]
[0307] }.
[0308] Comprehensive processing: According to preset rules, the text recognition results, seal recognition information, electronic receipt information and structured data are comprehensively processed:
[0309] Text recognition results: invoice number, date, amount
[0310] Seal identification information: XX Company
[0311] Electronic invoice information: invoice number, date, amount, taxpayer identification number, invoicing unit
[0312] Structured data: invoice number, date, amount, details
[0313] Combining all the information, the final bill result is generated:
[0314] Final ticket results:
[0315] {
[0316] "Invoice Number":"12345678",
[0317] "Date":"2023-01-01",
[0318] "Amount":"100.00",
[0319] "Invoicing Unit":"XX Company",
[0320] "Taxpayer Identification Number":"9876543210",
[0321] "Details":[
[0322] {"Product Name":"Product A","Quantity":"1","Unit Price":"50.00"},
[0323] {"Product Name":"Product B","Quantity":"1","Unit Price":"50.00"} ]
[0325] }.
[0326] Through the above steps, this embodiment can accurately extract and integrate the information in the invoice image, generate a complete and structured bill result, and ensure the integrity and accuracy of the data.
[0327] The invention also provides an invoice image intelligent recognition device.
[0328] Reference Figure 3 , Figure 3 This is a functional module diagram of the first embodiment of the invoice image intelligent recognition device of the present invention. The invoice image intelligent recognition device comprises:
[0329] An invoice detection and splitting module obtains an image to be identified, and uses a first deep learning model to detect invoice key points in the image to be identified. When it is detected that the image to be identified contains multiple invoice images, the image to be identified is split according to the invoice key points to form multiple independent invoice images;
[0330] An invoice integrity detection module detects the integrity of each invoice image. When all invoice images are detected to be complete, a second deep learning model is used to detect the text position in each invoice image and generate a text recognition box.
[0331] An image direction correction module determines the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image, and rotates each invoice image and the text recognition frame of each invoice image to a horizontal direction;
[0332] The image text recognition module captures the text image of the text recognition frame of each invoice image, inputs the text image into the text recognition model, and obtains the text recognition result of each invoice image.
[0333] In addition, an embodiment of the present invention further provides a computer storage medium.
[0334] The storage medium stores an invoice image intelligent recognition program, and when the invoice image intelligent recognition program is executed by the processor, the steps of the invoice image intelligent recognition method described above are implemented.
[0335] The specific implementation of the storage medium of the present invention is basically the same as the embodiments of the above-mentioned invoice image intelligent recognition method, and will not be repeated here.
[0336] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims. All equivalent structures or equivalent process changes made using the contents of the specification and drawings of the present invention, or directly or indirectly used in other related technical fields, are protected by the present invention.
[0337] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of boxes in the block diagram and / or the flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0338] In addition, each functional module in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part. If the function is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk.
[0339] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0340] The above are only various implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. An intelligent invoice image recognition method, characterized in that: The following steps are involved: Acquire an image to be identified, and use a first deep learning model to detect invoice key points in the image to be identified. When it is detected that the image to be identified contains multiple invoice images, split the image to be identified according to the invoice key points to form multiple independent invoice images; Detect the integrity of each invoice image. When all invoice images are detected to be complete, use the second deep learning model to detect the text position in each invoice image and generate a text recognition box. Determine the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image, and rotate each invoice image and the text recognition frame of each invoice image to a horizontal direction; A text image of a text recognition frame of each invoice image is intercepted, and the text image is input into a text recognition model to obtain a text recognition result of each invoice image.
2. The method for intelligent invoice image recognition according to claim 1, characterized in that: The detection of the integrity of each invoice image includes: Detect whether all the preset key information of each invoice image exists, and use edge detection algorithm to check whether there is edge loss in each invoice image; If all the preset key information of each invoice image exists and there is no edge loss, or all the preset key information of each invoice image exists and the edge loss does not affect the preset key information, then each invoice image is confirmed to be complete.
3. The method for intelligent invoice image recognition according to claim 1, characterized in that: The step of determining the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image includes: Selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and determining the rotation direction of each invoice image according to the text recognition confidence and aspect ratio of the target text recognition frames; The rotation angle of each invoice image is determined from preset rotation angles according to the rotation direction of each invoice image.
4. The method for intelligent invoice image recognition according to claim 1, characterized in that: The step of determining the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image includes: Selecting a preset number of target text recognition frames with the highest text recognition confidence from the text recognition frames of each invoice image, and calculating the main direction angle of each target text recognition frame using the boundary points of each target text recognition frame; The rotation angle of each invoice image is obtained according to the main direction angle of each target text recognition frame and the text recognition confidence.
5. The method for intelligent invoice image recognition according to claim 1, characterized in that: After rotating each invoice image and the text recognition frame of each invoice image to a horizontal direction, the method further includes: The third deep learning model is used to detect the seal image in each invoice image, and the seal image is enhanced. The enhanced seal image is recognized using the TROCR model, and the text information of the seal image is extracted, where the text information includes the unit name corresponding to each invoice.
6. The method for intelligent invoice image recognition according to claim 5, characterized in that: After extracting the text information in the seal image, the method further includes: If there is an electronic invoice image in the invoice image, the fourth deep learning model is used to identify the invoice QR code of the electronic invoice image, the image contrast of the invoice QR code is enhanced, the enhanced invoice QR code is decoded using a QR code decoding algorithm, and the electronic bill detailed information of the electronic invoice is extracted.
7. The method for intelligent recognition of invoice images as claimed in claim 6, characterized in that: After extracting the electronic bill details of the electronic invoice, it also includes: Use the information extraction model and the preset traditional extraction method to extract information from each invoice image to obtain structured data, which includes key fields and detailed items; According to preset rules, the text recognition results of each invoice image, the text information in the seal image, the electronic invoice details and structured data are integrated to generate the final invoice result for each invoice image.
8. An intelligent invoice image recognition device, characterized in that: The invoice image intelligent recognition device comprises: An invoice detection and splitting module obtains an image to be identified, and uses a first deep learning model to detect invoice key points in the image to be identified. When it is detected that the image to be identified contains multiple invoice images, the image to be identified is split according to the invoice key points to form multiple independent invoice images; An invoice integrity detection module detects the integrity of each invoice image. When all invoice images are detected to be complete, a second deep learning model is used to detect the text position in each invoice image and generate a text recognition box. An image direction correction module determines the rotation angle of each invoice image according to the text recognition confidence of the text recognition frame of each invoice image, and rotates each invoice image and the text recognition frame of each invoice image to a horizontal direction; The image text recognition module captures the text image of the text recognition frame of each invoice image, inputs the text image into the text recognition model, and obtains the text recognition result of each invoice image.
9. An intelligent invoice image recognition device, characterized in that: The invoice image intelligent recognition device includes a memory, a processor, and an invoice image intelligent recognition program stored in the memory and executable on the processor. When the invoice image intelligent recognition program is executed by the processor, the steps of the invoice image intelligent recognition method as described in any one of claims 1 to 7 are implemented.
10. A computer storage medium, characterized in that: The storage medium stores an invoice image intelligent recognition program, and when the invoice image intelligent recognition program is executed by the processor, the steps of the invoice image intelligent recognition method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Image processing apparatus, method and device, and readable storage medium
CN110399851A
Mixed-pasting bill image processing method, device, computer equipment and storage medium
CN111931664A
OCR-based case text recognition method and electronic equipment
CN112861842A
Value-added tax invoice identification method based on template matching
CN116912868A
Cited By
Table data intelligent identification analysis and articulation system and method based on large model
CN121768023A
