Bill identification method and device, computer readable storage medium and electronic equipment

Abstract

The invention relates to the technical field of artificial intelligence, and provides a bill recognition method, a bill recognition device, a medium and equipment, and the method comprises the steps: recognizing the orientation of a bill through a bill direction judgment model after a bill image is obtained, and carrying out the direction correction of the bill image according to an orientation recognition result; detecting a character area from the bill image after direction correction through a bill character detection model, and cutting the bill image based on the position coordinates of the textboxes to obtain a plurality of textbox images; recognizing text content corresponding to each textbox image through a bill character recognition model; inputting the position coordinates and the text content of each textbox into a bill field structured model for structured semantic analysis, and generating a structured recognition result of the bill; wherein the at least one model is obtained by training a training sample subjected to data enhancement processing. According to the invention, high-precision identification of bill information can be realized in a real scene with low image quality.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a bill recognition method, a bill recognition device, a computer-readable storage medium, and an electronic device. Background Technology

[0002] In many business processes across industries such as finance, healthcare, and insurance, it is often necessary to extract key information from various invoice images submitted by users for subsequent review, settlement, or risk control analysis. These invoices typically contain structured data fields such as date, amount, institution name, and serial number, and accurate acquisition of these data is a crucial prerequisite for business automation. Traditionally, this information has relied primarily on manual review and entry, which is not only slow and labor-intensive but also prone to errors due to subjective negligence or visual fatigue, making it difficult to meet the demands of today's large-scale, time-sensitive business processing.

[0003] In recent years, with the development of artificial intelligence technology, information extraction methods based on image recognition have gradually become a research and application hotspot. In particular, optical character recognition (OCR) technology has shown good application potential in various document processing scenarios. However, in actual use, the ticket images uploaded by users are often taken by handheld devices. Affected by factors such as shooting environment, device performance, and operating habits, the image quality varies greatly, and problems such as blurriness, uneven lighting, distortion, occlusion, and reflection often occur, which brings great difficulties to the accurate positioning and recognition of text.

[0004] Existing automated recognition methods often rely on fixed formats or preset rules for invoices. For example, some schemes locate corresponding fields by recognizing specific keywords, but the entire field extraction process may fail when the keyword region is damaged or recognition errors occur. Other methods rely on pre-established templates, matching fields based on their relative positions in the image; however, the accuracy of template matching significantly decreases once the image is rotated, scaled, or locally deformed. Therefore, how to achieve stable and accurate recognition of invoice content even when the input image quality is unstable and the layout varies remains a pressing problem in current technological development. In light of this, there is an urgent need in this field to develop a new invoice recognition method and apparatus.

[0005] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of this disclosure. Summary of the Invention

[0006] The purpose of this disclosure is to provide a ticket recognition method, ticket recognition device, computer-readable storage medium, and electronic device, thereby overcoming, to at least a certain extent, the technical problem of being unable to achieve accurate ticket recognition under conditions of low image quality due to limitations in related technologies.

[0007] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part by practice of this disclosure.

[0008] According to a first aspect of this disclosure, a method for identifying invoices is provided, comprising: After obtaining the ticket image, the orientation of the ticket is identified by the ticket orientation determination model, and the orientation of the ticket image is corrected according to the orientation identification result; The text region is detected from the direction-corrected ticket image using a ticket text detection model, and multiple text box images are obtained by cropping the ticket image based on the position coordinates of the text boxes. The text content corresponding to each of the text box images is identified using a document text recognition model. The position coordinates and text content of each text box are input into the bill field structured model for structured semantic parsing to generate the structured recognition result of the bill; At least one model is trained using data-augmented training samples.

[0009] In an exemplary embodiment of this disclosure, the ticket image is obtained in the following manner: The image to be identified is input into the ticket detection model, and the position coordinates of the ticket area in the image to be identified are detected. The ticket image is obtained by cropping based on the position coordinates of the ticket area.

[0010] In an exemplary embodiment of this disclosure, the ticket detection model is trained in the following manner: Obtain the first training image set containing the annotations for the ticket regions; Data augmentation processing is performed on the training images in the first training image set to generate a second training image set; the data augmentation processing includes at least one of random cropping, image flipping, mirror flipping, random occlusion, and random distortion; The ticket detection model is trained based on the second training image set.

[0011] In an exemplary embodiment of this disclosure, the bill direction determination model is trained in the following manner: Obtain the third training image set containing ticket orientation labels; Data augmentation processing is performed on the images in the third training image set to generate a fourth training image set; the data augmentation processing includes at least one of image flipping, mirror flipping, and random occlusion. The ticket orientation judgment model is trained based on the fourth training image set.

[0012] In an exemplary embodiment of this disclosure, the document text detection model is trained in the following manner: Obtain the ticket image containing the text box annotations as the fifth training image set; The document text detection model is obtained by training the fifth training image set.

[0013] In an exemplary embodiment of this disclosure, the invoice text recognition model is trained in the following manner: Construct a sixth training image set; the sixth training image set contains synthetic images, which are generated by combining text and background. The invoice text recognition model is trained based on the sixth training image set.

[0014] In an exemplary embodiment of this disclosure, the synthesized image is obtained in the following manner: Construct a corpus containing text content related to multiple target domains; the frequency of rare characters in the corpus is greater than a preset frequency threshold; A background image is cropped from a real ticket image; the background image encompasses various imaging degradation modes; The text content from the corpus is randomly inserted into the background image to generate the first image; The first image is degraded to obtain the synthesized image.

[0015] In an exemplary embodiment of this disclosure, the imaging degradation mode includes at least one of the following: Electronically displayed images, distorted images from photographs, uneven lighting, and text obstruction.

[0016] In an exemplary embodiment of this disclosure, the step of degrading the first image to obtain the synthesized image includes: The first image is processed by random text offsetting to generate the second image; The second image is blurred to generate the composite image; the blurring process includes at least one of Gaussian blur, motion blur, salt-and-pepper noise, and image compression.

[0017] In an exemplary embodiment of this disclosure, the step of inputting the position coordinates and text content of each text box into a document field structured model for structured semantic parsing to generate a structured recognition result for the document includes: The position coordinates and text content of each text box are identified using the structured model of the ticket fields to determine their respective field categories. Multiple text contents belonging to the same field category are aggregated to generate complete field values, thereby generating the structured recognition result of the invoice.

[0018] In an exemplary embodiment of this disclosure, the invoice field structured model is a multi-classification model, which includes a text semantic extraction module, a text box position extraction module, a feature fusion module, and a classification head module. The text semantic extraction module is used to embed the input text content at the character level and extract its semantic features; The text box position extraction module is used to extract the spatial position features of the text box through a multilayer perceptron network. The feature fusion module is used to splice and fuse the semantic features and spatial location features; The classification header module is used to output the probability distribution of each field category through a fully connected network.

[0019] In an exemplary embodiment of this disclosure, the bill field structure model is trained in the following manner: Obtain the seventh training dataset, which includes the coordinates of the text boxes and the labels of the field categories; The seventh training dataset is augmented to generate the eighth training dataset. The data augmentation includes at least one of the following: randomly deleting, replacing or inserting text content, slightly shifting the coordinates of the text boxes, randomly deleting some text boxes, and randomly shuffling the input order of the text boxes. The structured model of the invoice fields was trained based on the eighth training dataset.

[0020] According to a second aspect of this disclosure, a ticket recognition device is provided, comprising: The image processing module is used to identify the orientation of the ticket through a ticket orientation determination model after obtaining the ticket image, and to perform orientation correction on the ticket image based on the orientation identification result; The text detection module is used to detect text regions from the direction-corrected ticket image using a ticket text detection model, and to crop the ticket image based on the position coordinates of the text boxes to obtain multiple text box images. The text recognition module is used to identify the text content corresponding to each text box image through the invoice text recognition model; The structured recognition module is used to input the position coordinates and text content of each text box into the bill field structured model for structured semantic parsing, and generate the structured recognition result of the bill. At least one model is trained using data-augmented training samples.

[0021] According to a third aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the ticket recognition method described in the first aspect above.

[0022] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the ticket recognition method described in the first aspect by executing the executable instructions.

[0023] As can be seen from the above technical solutions, the bill recognition method, bill recognition device, computer-readable storage medium, and electronic device in the exemplary embodiments of this disclosure have at least the following advantages and positive effects: In the technical solutions provided by some embodiments of this disclosure, on the one hand, by introducing a ticket orientation judgment model and automatically correcting the image, problems such as rotation and inversion caused by arbitrary shooting angles by users can be effectively corrected, ensuring that the subsequent text detection and recognition modules process the image content under a unified direction, avoiding text region omissions or field positioning errors caused by directional deviations, thereby enhancing the stability and usability of the system in practical applications; furthermore, since at least one model is trained using data-augmented training samples, the model is exposed to a more diverse sample distribution during training, enabling it to better cope with common problems in the image such as quality reduction, slight deformation, noise interference, and complex backgrounds during the inference stage, thereby improving the system's adaptability to complex inputs and recognition stability in real business scenarios; on the other hand, by inputting the text content and text box position coordinates into the ticket field structured model for structured semantic parsing, and using multimodal information to jointly determine the field category, the reliance on keyword matching or fixed layout in traditional methods is avoided. Even in cases of inconsistent field naming, slight differences in layout, or missing information, key fields can still be accurately identified based on semantic and spatial relationships, improving the intelligence level and applicability of ticket information extraction.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0025] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0026] Figure 1A flowchart illustrating the invoice recognition method in an embodiment of this disclosure is shown; Figure 2 This diagram illustrates a process for obtaining a ticket image in an embodiment of the present disclosure. Figure 3 A schematic diagram illustrating the training process of the invoice detection model in this embodiment of the present disclosure is shown. Figure 4 A schematic diagram illustrating the training process of the bill direction determination model in an embodiment of this disclosure is shown. Figure 5 A schematic diagram illustrating the training process of the document text recognition model in this embodiment of the present disclosure is shown. Figure 6 This diagram illustrates the training process of the bill field structured model in an embodiment of this disclosure. Figure 7 This diagram illustrates the overall flow of the invoice recognition method in this embodiment of the present disclosure. Figure 8 This diagram illustrates the structure of the document recognition device in an exemplary embodiment of this disclosure. Figure 9 A schematic diagram of the structure of an electronic device in an exemplary embodiment of this disclosure is shown. Detailed Implementation

[0027] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0028] The terms “a,” “an,” “the,” and “the” are used in this specification to indicate the presence of one or more elements / components / etc.; the terms “including” and “having” are used to indicate an open-ended inclusion and to mean that there may be other elements / components / etc. in addition to the listed elements / components / etc.; the terms “first” and “second” are used only as markings and are not a limitation on the number of objects.

[0029] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0030] In the embodiments of this disclosure, a ticket recognition method is first provided, which at least to some extent overcomes the shortcomings of related technologies that cannot achieve accurate recognition of tickets when the image quality is low.

[0031] Figure 1 The diagram shows a flowchart of a bill recognition method in an embodiment of this disclosure. The execution entity of the bill recognition method can be a server that recognizes bills.

[0032] refer to Figure 1 A ticket recognition method according to an embodiment of the present disclosure includes the following steps: Step S110: After obtaining the ticket image, the orientation of the ticket is identified by the ticket orientation judgment model, and the orientation of the ticket image is corrected according to the orientation recognition result. Step S120: Detect text regions from the orientation-corrected ticket image using a ticket text detection model, and crop the ticket image based on the position coordinates of the text boxes to obtain multiple text box images; Step S130: The text content corresponding to each text box image is identified by the invoice text recognition model; Step S140: Input the position coordinates and text content of each text box into the bill field structured model for structured semantic parsing to generate the structured recognition result of the bill; wherein, at least one model is trained using training samples that have undergone data augmentation.

[0033] exist Figure 1The technical solution provided in the illustrated embodiment, on the one hand, by introducing a ticket orientation judgment model and automatically correcting the image, can effectively correct problems such as rotation and inversion caused by the user's arbitrary shooting angle, ensuring that the subsequent text detection and recognition modules process the image content under a unified direction, avoiding text region omissions or field positioning errors caused by directional deviations, thereby enhancing the stability and usability of the system in practical applications; furthermore, since at least one model is trained using data-augmented training samples, the model is exposed to a more diverse sample distribution during training, enabling it to better cope with common problems in the image such as quality degradation, slight deformation, noise interference, and complex backgrounds during the inference stage, thereby improving the system's adaptability to complex inputs and recognition stability in real business scenarios; on the other hand, by inputting the text content and text box position coordinates into the ticket field structured model for structured semantic parsing, and using multimodal information to jointly determine the field category, the reliance on keyword matching or fixed layout in traditional methods is avoided. Even in cases of inconsistent field naming, slight differences in layout, or missing information, key fields can still be accurately identified based on semantic and spatial relationships, improving the intelligence level and applicability of ticket information extraction.

[0034] The following are Figure 1 The specific implementation process of each step in the process will be explained in detail: First, in this embodiment of the disclosure, "document" refers to a written voucher used to record transaction information and having a specific format and anti-counterfeiting features. Such documents may include, but are not limited to: invoices (such as general VAT invoices, special VAT invoices, and electronic invoices), receipts, train tickets, airline tickets, taxi receipts, bank drafts (such as checks and drafts), insurance policies, bills of lading, etc., which can be set according to actual circumstances, and this disclosure does not impose any special limitations on them.

[0035] It should be noted that this disclosure can pre-train multiple models (invoice detection model, invoice direction judgment model, invoice text detection model, invoice text recognition model, and invoice field structuring model), and one or more of the above models are trained using data augmented training samples.

[0036] The aforementioned data augmentation processing can include one or more of the following: geometric and spatial transformation enhancement, image quality degradation enhancement, and semantic-level perturbation enhancement. Specifically, the geometric and spatial transformation enhancement can include one or more of the following: random cropping, rotation, scaling, flipping (horizontal / vertical), mirroring, affine transformation, perspective transformation, and random distortion. The image quality degradation enhancement can include one or more of the following: blurring (Gaussian blur, motion blur), noise injection (salt-and-pepper noise, Gaussian noise), image compression distortion (JPEG compression), uneven lighting, overexposure / underexposure, electronic screen display distortion, reflection simulation, and text area occlusion (such as stamping, regular lines crossing text areas, random text offset, and font deformation). The semantic-level perturbation enhancement can include one or more of the following: random deletion, replacement, and insertion of text content (such as perturbation of name and amount characters), small-range offset of text box coordinates, random deletion of some text boxes, and shuffling of the text box input order. These can all be set according to actual conditions, and this disclosure does not impose any special limitations on them.

[0037] The following explains how to obtain the ticket image in step S110. (Refer to...) Figure 2 , Figure 2 This diagram illustrates a process for obtaining a document image in an embodiment of the present disclosure, including steps S201-S202: In step S201, the image to be identified is input into the ticket detection model to detect the position coordinates of the ticket area in the image to be identified.

[0038] In this step, the image to be recognized can be input into the ticket detection model to detect the position coordinates of the ticket region in the image, for example: .in, This represents the x-coordinate of the leftmost side of the bounding box (i.e., the x-coordinate of the top left corner). This represents the y-coordinate of the top edge of the bounding box (i.e., the vertical coordinate of the top left corner). This represents the x-coordinate of the far right of the bounding box (i.e., the x-coordinate of the bottom right corner). This represents the y-coordinate of the bottom edge of the bounding box (i.e., the ordinate of the bottom right corner).

[0039] refer to Figure 3 , Figure 3 This diagram illustrates the training process of the above-described invoice detection model in this embodiment of the present disclosure, including steps S301-S303: In step S301, a first training image set containing the annotation of the ticket area is obtained.

[0040] In this step, a first training image set containing labeled invoice regions can be obtained. This first training image set can include real invoice images from various sources, covering different invoice types (such as VAT invoices, medical bills, transportation tickets, fixed-amount invoices, etc.), different shooting conditions (such as natural light, indoor lighting, mobile phone flash), different imaging devices (such as smartphones, scanners), and different image qualities (such as high-definition, blurry, reproduced images, electronic screen screenshots). The bounding box coordinates of the invoice region in each image are manually or semi-automatically labeled to supervise the learning process of the invoice detection model. By constructing a diverse initial dataset, basic data support is provided for subsequent model training.

[0041] In step S302, data augmentation processing is performed on the training images in the first training image set to generate a second training image set; the data augmentation processing includes at least one of random cropping, image flipping, mirror flipping, random occlusion, and random distortion.

[0042] In this step, various geometric transformations and local disturbances can be applied to the original image to simulate various complex situations that may occur in real-world scenes, thereby generating a second training image set.

[0043] Specifically, random cropping can simulate scenarios where the position of the paper document in the image is not fixed and the background is complex, enhancing the model's adaptability to changes in target scale and position. Image flipping and mirroring can increase the diversity of image orientation, improving the model's robustness to rotation, inversion, and other situations. Random occlusion can introduce rectangular masks or noise blocks into the image to simulate real-world interference factors such as finger occlusion, binding holes, and background interference. Random distortion based on affine transformation or elastic deformation can simulate non-rigid deformations of paper documents such as bending, folding, and perspective distortion, enabling the model to cope with geometric distortions caused by improper shooting angles. The combined use of these data augmentation techniques significantly improves the diversity of training samples and the richness of the dataset without increasing additional annotation costs, effectively alleviating the overfitting problem in small sample scenarios.

[0044] In step S303, a ticket detection model is trained based on the second training image set.

[0045] In this step, a deep learning object detection architecture (such as YOLO, Faster R-CNN, or PP-YOLOE) can be used as the base network for the ticket detection model. The second training image set, augmented with data, is input into the model, and end-to-end training is performed by minimizing the bounding box regression loss and classification loss. Since the training data already covers various degradation patterns and geometric variations, the trained ticket detection model possesses strong generalization ability, accurately identifying and locating ticket regions under complex backgrounds, low-quality images, and non-standard shooting conditions. During the inference phase, the model can quickly extract the complete ticket image from the image to be recognized, providing high-quality input for subsequent orientation correction, text recognition, and structured parsing, thereby improving the overall stability and accuracy of the ticket recognition system.

[0046] Next, refer to Figure 2 In step S202, the ticket image is obtained by cropping based on the position coordinates of the ticket area.

[0047] In this step, after obtaining the location coordinates of the ticket area, the ticket image can be cropped from the image to be recognized based on these location coordinates.

[0048] Next, refer to Figure 1 In step S110, after obtaining the ticket image, the orientation of the ticket is identified by the ticket orientation judgment model, and the orientation of the ticket image is corrected according to the orientation recognition result.

[0049] In this step, after obtaining the ticket image, the ticket orientation can be identified using a ticket orientation determination model, and the ticket image can be corrected for orientation based on the orientation identification result. (Reference) Figure 4 , Figure 4 This diagram illustrates the training process of the above-described bill direction determination model in this embodiment of the present disclosure, including steps S401-S403: In step S401, a third training image set containing ticket orientation labels is obtained.

[0050] In this step, the third training image set can include a large number of real or synthetic ticket images, each labeled with its correct reading direction, typically categorized into four types: 0 degrees (forward), 90 degrees (clockwise rotation), 180 degrees (inverted), and 270 degrees (counter-clockwise rotation). These images originate from user-uploaded data in real-world business scenarios, covering different ticket types, different shooting devices, different lighting conditions, and various imaging degradation modes (such as blurring, overexposure, and distortion from re-photographing). By constructing a labeled dataset covering multiple angles and quality levels, supervisory signals are provided for the model to learn visual features under different orientations, ensuring its ability to recognize complex directional states.

[0051] In step S402, data augmentation processing is performed on the images in the third training image set to generate a fourth training image set; the data augmentation processing includes at least one of image flipping, mirror flipping, and random occlusion.

[0052] In this step, various data augmentation strategies can be introduced to simulate different interference situations that may occur in real-world scenarios, thereby improving the diversity of training data and the generalization ability of the model. For example: Image flipping (e.g., 90 degrees, 180 degrees, 270 degrees) is used to explicitly construct samples with different rotation angles, enhancing the model's sensitivity to changes in orientation. Mirror flip (horizontal or vertical flip) simulates mirror distortion caused by abnormal shooting posture, improving the model's adaptability to non-standard imaging; Random occlusion, by adding local masks or noise blocks to the image, simulates interference such as finger occlusion, binding holes, and background clutter, forcing the model to focus on global structure rather than local texture, thereby improving the robustness of orientation discrimination under partial occlusion.

[0053] The combination of the above enhancement methods significantly expands the number and distribution of effective training samples without increasing the cost of additional manual annotation, effectively alleviating the overfitting problem caused by imbalanced directional samples or scarce data.

[0054] In step S403, a ticket orientation judgment model is trained based on the fourth training image set.

[0055] In this step, a classification network architecture (such as ResNet, MobileNet, or EfficientNet) can be used as the basic structure of the document orientation determination model. The data-augmented fourth training image set is input into the model, and supervised training is performed using orientation categories (0 degrees / 90 degrees / 180 degrees / 270 degrees) as labels. The model parameters are optimized by minimizing the cross-entropy loss function. Because various geometric transformations and visual disturbances are fully incorporated during training, the trained model possesses strong anti-interference capabilities and orientation discrimination accuracy. It can stably output the correct rotation angle even under low-quality conditions such as image blur, uneven lighting, and partial occlusion. During the inference phase, the model can automatically identify the orientation of the input document and guide subsequent image rotation correction, ensuring that the text area is in the standard reading direction, providing a consistent and reliable input format for downstream text detection and structured parsing.

[0056] Next, refer to Figure 1 In step S120, the text region is detected from the orientation-corrected ticket image using the ticket text detection model, and multiple text box images are obtained by cropping the ticket image based on the position coordinates of the text boxes.

[0057] In this step, a text detection model can be used to detect text regions from the orientation-corrected document image, and multiple text box images can be obtained by cropping the document image based on the position coordinates of the text boxes.

[0058] The aforementioned document text detection model can be trained in the following way: First, images of tickets containing text box annotations can be obtained as the fifth training image set. This fifth training image set can include a large number of ticket images with precise text box annotations. Each field content in each image is represented by a four-point coordinate system (...). The polygonal bounding box annotations defined by the model only mark the actual location of the field content, excluding field names or other unstructured information. This annotation strategy helps the model focus on learning the positional features of the field content, avoiding interference from extraneous information, thereby improving the model's robustness and accuracy in complex scenarios. Furthermore, to cover various possible variations in document styles and layouts, the fifth training image set should encompass document samples of multiple types, formats, and layout designs to ensure the model has good generalization ability.

[0059] Secondly, a document text detection model can be trained using the aforementioned fifth training image set. For example, object detection algorithms within deep learning frameworks (such as Faster R-CNN, SSD, YOLO, etc.) can be used to construct the basic architecture of the document text detection model. The carefully labeled fifth training image set is input into the model for supervised learning, using the four coordinates corresponding to each field's content as labels to optimize the model parameters, enabling it to accurately predict the location of text boxes in unseen document images. To further improve model performance, data augmentation techniques (such as rotation, scaling, cropping, etc.) can be combined to generate more diverse training samples, simulating various situations that may occur in real-world applications. The finally trained document text detection model can accurately identify and locate the positions of all field contents in various document images, providing a solid foundation for subsequent structured analysis.

[0060] Next, refer to Figure 1 In step S130, the text content corresponding to each text box image is identified by the invoice text recognition model.

[0061] In this step, the text content corresponding to each text box image can be identified using a document text recognition model. (Reference) Figure 5 , Figure 5 This diagram illustrates the training process of the document text recognition model in this embodiment, including steps S501-S502: In step S501, a sixth training image set is constructed; the sixth training image set contains synthetic images, which are generated by combining text and background.

[0062] In this step, to improve the recognition accuracy of the document text recognition model in real and complex scenarios, especially its robustness to low-quality images such as blurry, occluded, unevenly lit, and distorted images, this disclosure employs a text-background synthesis method based on real backgrounds to construct high-fidelity training data. Specifically, this may include the following sub-steps: First, a corpus containing text content related to the target domain is constructed. The corpus covers common text information in medical invoices, such as drug names, hospital names, treatment items, patient names, dates, and amounts. Special emphasis is placed on the coverage of rare characters to ensure that their frequency of occurrence in the corpus is higher than a preset threshold, thereby ensuring the model's ability to recognize low-frequency characters. Secondly, representative background image areas are cropped from a large number of real ticket images. The background images cover a variety of imaging degradation modes, including but not limited to: electronic screen display images, distorted reproduction images, uneven lighting areas, text areas covered by seals, and regular lines (such as table lines) crossing text areas, to ensure that the synthesized images have realism and diversity. Then, the text content in the corpus is randomly inserted into the background image with variable font, font size, color and tilt angle to generate the first image, simulating the fusion effect of text and background in real tickets; Next, the first image undergoes multi-stage degradation processing: first, random text offset processing is performed to simulate the slight shift in text position caused by printing deviation or shooting shake; then, blurring processing is performed, including at least one method such as Gaussian blur, motion blur, salt and pepper noise, and image compression, to synthesize text with different degrees of blur, that is, to obtain a synthesized image, thereby enhancing the model's ability to recognize blurry characters.

[0063] The synthetic images generated through the above process constitute the sixth training image set, which closely resembles real user-uploaded images in terms of visual features, degradation patterns, and semantic distribution. This effectively compensates for problems such as the scarcity of real labeled data, insufficient samples of rare characters, and difficulty in collecting low-quality images.

[0064] In step S502, a ticket text recognition model is trained based on the sixth training image set.

[0065] In this step, a deep learning sequence recognition architecture (such as CRNN, Transformer-based OCR, or PP-OCR series models) can be used as the base network for the document text recognition model. Synthetic images and their corresponding text labels from the sixth training image set are used as input and supervision signals, and model parameters are optimized through end-to-end training. Since the training data fully simulates various degradation factors in real-world scenarios (such as blurring, occlusion, lighting changes, and font deformation), the trained text recognition model possesses strong anti-interference capabilities and generalization performance, accurately recognizing the characters within text boxes even with poor image quality. Especially when facing challenging scenarios such as rare characters, stamp overprints, electronic screen glare, and blurred images from reproductions, the model demonstrates superior recognition stability compared to traditional OCR systems. During the inference phase, the model can efficiently output the text content corresponding to each text box (e.g., "May 10, 2025"), providing high-precision semantic input for subsequent field structured parsing.

[0066] Next, refer to Figure 1 In step S140, the position coordinates and text content of each text box are input into the bill field structured model for structured semantic parsing to generate the structured recognition result of the bill.

[0067] In this step, the position coordinates and text content of each text box can be input into the bill field structured model for structured semantic parsing to generate the structured recognition result of the bill.

[0068] Specifically, the invoice field structured model can perform joint analysis on each text box based on multimodal input information. On one hand, it utilizes the semantic features of the text content; on the other hand, it combines the text's spatial location features within the invoice image. Through a built-in deep neural network, it performs context-aware classification reasoning, outputting the probability distribution of the field category to which each text box belongs. The model determines the category label based on the highest probability, such as "invoice number," "invoice date," "buyer's name," and "total amount." For composite fields composed of multiple text fragments (e.g., "total price including tax" consists of "amount" and "tax rate," or the amount field is split into integer and decimal parts), the model can identify multiple text boxes belonging to the same field category and aggregate their content according to their logical order in the invoice (e.g., from top to bottom, from left to right) or semantic relevance, concatenating them to generate complete and standardized field values. Finally, all field categories and their corresponding aggregated text content are organized into a structured data format (e.g., JSON or key-value pairs), forming a complete structured recognition result for the invoice. This method does not rely on fixed templates or keyword matching, and can adapt to different formats, different language habits, and ticket images with local distortions, significantly improving the accuracy of field recognition and the robustness of the system.

[0069] refer to Figure 6 , Figure 6 This diagram illustrates the training process of the above-described invoice field structured model in this embodiment of the present disclosure, including steps S601-S603: In step S601, the seventh training dataset containing the text box position coordinates and field category labels is obtained.

[0070] In this step, we collect and organize the text box coordinates and their corresponding field category labels from various invoice images to form the seventh training dataset. Field categories include, but are not limited to: "Invoice Number," "Invoice Date," "Buyer Name," "Buyer Credit Code," "Seller Name," "Seller Credit Code," "Total Amount," "Total Tax," and "Total Price Including Tax." We ensure that each data point contains accurate coordinates and category labels through manual or automatic annotation tools, laying the foundation for subsequent data augmentation and model training.

[0071] In step S602, data augmentation processing is performed on the seventh training dataset to generate the eighth training dataset. The data augmentation processing includes at least one of the following: randomly deleting, replacing, or inserting text content, slightly shifting the coordinates of the text box positions, randomly deleting some text boxes, and randomly shuffling the input order of the text boxes.

[0072] In this step, to improve the model's generalization ability and robustness, various data augmentation methods are used to process the seventh training dataset. Specifically, the following measures can be taken: Perform random operations on the text content, such as deleting, replacing, or inserting characters, to simulate spelling errors or missing information in real-world scenarios; Apply a small offset to the text box position coordinates to simulate changes in text position caused by different scanning or shooting angles; Randomly delete some text boxes to enhance the model's reasoning ability under conditions of incomplete information; By shuffling the input order of the text boxes, the model's ability to understand and reconstruct the logical relationships between them is enhanced. After this processing, the resulting eighth training dataset not only enriches the sample diversity but also improves the model's ability to cope with complex environments.

[0073] In step S603, a structured model of the invoice fields is trained based on the eighth training dataset.

[0074] In this step, the eighth training dataset can be used as input, and the model can be trained using an end-to-end supervised learning approach. Specifically, the text content and position coordinates of each text box can be input into the ticket field structured model. The predicted probability distribution of each field category is calculated through forward propagation, and the classification loss (such as cross-entropy loss) is calculated with the true labels. The model parameters are updated using the backpropagation algorithm, and iterative convergence to the optimal solution is achieved through an optimizer (such as Adam). The final ticket field structured model can integrate text semantics and spatial location information to accurately determine the field category of each text box and output a complete structured recognition result.

[0075] The following describes the specific network structure of the above-mentioned invoice field structuring model. For example, the above-mentioned invoice field structuring model may include: Text semantic extraction module: used to embed the input text content at the character level, capture the contextual semantic dependencies through the LSTM network, and then extract high-dimensional semantic features through the MLP network; Text box position extraction module: used to receive the position coordinates of the text box and extract its spatial layout features through a multilayer perceptron (MLP) network; Feature fusion module: used to concatenate and fuse semantic features and positional features, and further model the global relationship across fields through MLP network and Transformer network to achieve context-aware joint representation; Classification Header Module: Used to output the probability distribution of each field category through a fully connected network to complete multi-class classification tasks.

[0076] In the exemplary embodiments of this disclosure, the fields to be identified can include a total of 9 categories, plus the "other" category, constituting a 10-category task (i.e., invoice number, invoice date, buyer's name, buyer's credit code, seller's name, seller's credit code, total amount, total tax, total price including tax, and others). This model, by jointly modeling text content and spatial location, eliminates the reliance on fixed templates or keyword matching. Even when field names are obscured, text recognition is incorrect, or the layout is distorted, it can still accurately infer field types based on context and relative position, significantly improving the accuracy and robustness of structured invoice recognition.

[0077] refer to Figure 7 , Figure 7 This diagram illustrates the overall flow of the document recognition method in this embodiment, including steps S701-S710: In step S701, the image to be recognized is input.

[0078] In this step, the user uploads an original image containing the invoice (such as a photo of an invoice taken with a mobile phone) as the input source for the entire recognition process. This image may contain complex backgrounds, uneven lighting, rotation, blurring, or occlusion, and requires multiple subsequent processing stages to extract valid information.

[0079] In step S702, ticket detection and ticket trimming are performed.

[0080] In this step, the image to be recognized is input into the ticket detection model to detect the position coordinates of the ticket region in the image (e.g., ...). The original image is cropped based on these coordinates to obtain a ticket image containing only the ticket content, providing clean input for subsequent processing.

[0081] In step S703, the direction of the ticket is determined.

[0082] In this step, the orientation of the ticket image is identified by the ticket orientation judgment model, and its rotation angle (such as 0 degrees, 90 degrees, 180 degrees or 270 degrees) is output. The orientation of the ticket image is then corrected according to this angle to restore it to the standard reading orientation, ensuring the accuracy of subsequent text detection and recognition.

[0083] In step S704, the ticket rotation correction is performed.

[0084] In this step, based on the orientation judgment result obtained in step S703, the ticket image is subjected to geometric transformation (such as rotation) to complete the image orientation correction, so that the text content is in a horizontally readable state.

[0085] In step S705, text detection is performed.

[0086] In this step, a document text detection model is used to detect all text regions from the orientation-corrected document image and output the position coordinates (e.g., four-point coordinates) of each text box. This creates multiple text box images for subsequent recognition.

[0087] In step S706, text strip trimming is performed.

[0088] In this step, based on the text box position coordinates detected in step S705, the corresponding text region image (i.e., text box image) is cropped from the ticket image to provide independent input for subsequent text recognition.

[0089] In step S707, character recognition is performed.

[0090] In this step, the OCR recognition of each text box image is performed using a ticket text recognition model, and the text content corresponding to each text box is output (such as "May 10, 2025", "1234567890", etc.), realizing the conversion from image to text.

[0091] In step S708, text box classification is performed.

[0092] In this step, the position coordinates of each text box and its corresponding text content are input into the invoice field structured model for semantic parsing and field category identification. This model integrates text semantics and spatial location information to determine the field category (such as "invoice number", "invoice date", "total amount", etc.) of each text box.

[0093] In step S709, field merging processing is performed.

[0094] In this step, for multiple text boxes belonging to the same field category (such as amounts consisting of integer and decimal parts), the system aggregates the content according to their logical order or semantic relationship in the invoice to generate complete field values, avoiding information incompleteness caused by segmentation recognition.

[0095] In step S710, the structured recognition result is output.

[0096] In this step, all field categories and their corresponding aggregated text content are organized into a structured data format (such as JSON, key-value pairs, or XML) to form the final structured invoice recognition result, which can be directly used by downstream business systems (such as financial reconciliation and insurance claims).

[0097] Based on the above technical solutions, this disclosure has at least the following technical effects: First, it improves the accuracy and robustness of invoice recognition. By employing deep learning-based text detection technology, it directly locates and identifies the content areas of fields without relying on field names or keyword information. This method significantly enhances the system's adaptability to various complex situations (such as image distortion, perspective transformation, etc.), thereby greatly improving the accuracy of field extraction.

[0098] Secondly, it achieves efficient information extraction. Utilizing a self-designed deep learning text box classification network structure, combined with the position coordinates of the text boxes and multimodal information of the text content, it performs accurate field classification and information extraction. This approach not only improves the accuracy of field classification but also effectively handles complex scenarios containing multiple related text boxes, further optimizing the efficiency and quality of information extraction.

[0099] Third, it reduced data annotation costs and shortened model development cycles. The application of innovative data augmentation techniques greatly enriched the training dataset, reducing reliance on large amounts of manually labeled data. This not only helps reduce overall data annotation costs but also enables models to achieve ideal performance in a shorter time, accelerating the entire process from research to deployment.

[0100] Fourth, it empowers a wide range of business application scenarios. The high-precision invoice recognition method disclosed herein can be widely applied in multiple fields such as financial auditing, tax declaration, and corporate expense reimbursement. Through automated processing, it reduces human intervention, improves work efficiency, and lowers operating costs. Furthermore, the flexibility and scalability of this method also enable the recognition of other types of invoices, demonstrating broad application prospects.

[0101] This disclosure also provides a ticket recognition device. Figure 8 This diagram illustrates the structure of the document recognition device in an exemplary embodiment of this disclosure; as shown below. Figure 8 As shown, the ticket recognition device 800 may include an image processing module 810, a text detection module 820, a text recognition module 830, and a structured recognition module 840. Wherein: The image processing module 810 is used to identify the orientation of the ticket by a ticket orientation judgment model after obtaining the ticket image, and to perform orientation correction on the ticket image based on the orientation recognition result. The text detection module 820 is used to detect text regions from the direction-corrected ticket image using a ticket text detection model, and to crop the ticket image based on the position coordinates of the text boxes to obtain multiple text box images. The text recognition module 830 is used to recognize the text content corresponding to each of the text box images through the invoice text recognition model; The structured recognition module 840 is used to input the position coordinates and text content of each text box into the bill field structured model for structured semantic parsing, and generate the structured recognition result of the bill. At least one model is trained using data-augmented training samples.

[0102] In an exemplary embodiment of this disclosure, the image processing module 810 is configured to: The image to be identified is input into the ticket detection model, and the position coordinates of the ticket area in the image to be identified are detected. The ticket image is obtained by cropping based on the position coordinates of the ticket area.

[0103] In an exemplary embodiment of this disclosure, the image processing module 810 is configured to: Obtain the first training image set containing the annotations for the ticket regions; Data augmentation processing is performed on the training images in the first training image set to generate a second training image set; the data augmentation processing includes at least one of random cropping, image flipping, mirror flipping, random occlusion, and random distortion; The ticket detection model is trained based on the second training image set.

[0104] In an exemplary embodiment of this disclosure, the image processing module 810 is configured to: Obtain the third training image set containing ticket orientation labels; Data augmentation processing is performed on the images in the third training image set to generate a fourth training image set; the data augmentation processing includes at least one of image flipping, mirror flipping, and random occlusion. The ticket orientation judgment model is trained based on the fourth training image set.

[0105] In an exemplary embodiment of this disclosure, the text detection module 820 is configured as follows: Obtain the ticket image containing the text box annotations as the fifth training image set; The document text detection model is obtained by training the fifth training image set.

[0106] In an exemplary embodiment of this disclosure, the text recognition module 830 is configured as follows: Construct a sixth training image set; the sixth training image set contains synthetic images, which are generated by combining text and background. The invoice text recognition model is trained based on the sixth training image set.

[0107] In an exemplary embodiment of this disclosure, the text recognition module 830 is configured as follows: Construct a corpus containing text content related to multiple target domains; the frequency of rare characters in the corpus is greater than a preset frequency threshold; A background image is cropped from a real ticket image; the background image encompasses various imaging degradation modes; The text content from the corpus is randomly inserted into the background image to generate the first image; The first image is degraded to obtain the synthesized image.

[0108] In an exemplary embodiment of this disclosure, the imaging degradation mode includes at least one of the following: Electronically displayed images, distorted images from photographs, uneven lighting, and text obstruction.

[0109] In an exemplary embodiment of this disclosure, the text recognition module 830 is configured as follows: The first image is processed by random text offsetting to generate the second image; The second image is blurred to generate the composite image; the blurring process includes at least one of Gaussian blur, motion blur, salt-and-pepper noise, and image compression.

[0110] In an exemplary embodiment of this disclosure, the structured recognition module 840 inputs the position coordinates and text content of each text box into the document field structured model for structured semantic parsing, generating the structured recognition result of the document, including: The position coordinates and text content of each text box are identified using the structured model of the ticket fields to determine their respective field categories. Multiple text contents belonging to the same field category are aggregated to generate complete field values, thereby generating the structured recognition result of the invoice.

[0111] In an exemplary embodiment of this disclosure, the invoice field structured model is a multi-classification model, which includes a text semantic extraction module, a text box position extraction module, a feature fusion module, and a classification head module. The text semantic extraction module is used to embed the input text content at the character level and extract its semantic features; The text box position extraction module is used to extract the spatial position features of the text box through a multilayer perceptron network. The feature fusion module is used to splice and fuse the semantic features and spatial location features; The classification header module is used to output the probability distribution of each field category through a fully connected network.

[0112] In an exemplary embodiment of this disclosure, the structured recognition module 840 is configured to: Obtain the seventh training dataset, which includes the coordinates of the text boxes and the labels of the field categories; The seventh training dataset is augmented to generate the eighth training dataset. The data augmentation includes at least one of the following: randomly deleting, replacing or inserting text content, slightly shifting the coordinates of the text boxes, randomly deleting some text boxes, and randomly shuffling the input order of the text boxes. The structured model of the invoice fields was trained based on the eighth training dataset.

[0113] The specific details of each module in the aforementioned document recognition device have been described in detail in the corresponding document recognition method, so they will not be repeated here.

[0114] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0115] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0116] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0117] This disclosure also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device.

[0118] Computer-readable storage media can be, for example—but not limited to—electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0119] A computer-readable storage medium can be sent, propagated, or transmitted for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wireless, wireline, optical fiber, RF, etc., or any suitable combination thereof.

[0120] A computer-readable storage medium carries one or more programs that, when executed by an electronic device, cause the electronic device to perform the methods described in the above embodiments.

[0121] Furthermore, this disclosure also provides an electronic device capable of implementing the above-described method.

[0122] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0123] The following reference Figure 9 To describe an electronic device 900 according to such an embodiment of the present disclosure. Figure 9 The electronic device 900 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0124] like Figure 9 As shown, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processor 910, at least one memory 920, a bus 930 connecting different system components (including memory 920 and processor 910), and a display 940.

[0125] The memory stores program code that can be executed by the processor 910, causing the processor 910 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processor 910 can perform actions such as... Figure 1As shown: Step S110, after obtaining the ticket image, the orientation of the ticket is identified by the ticket orientation judgment model, and the orientation of the ticket image is corrected according to the orientation recognition result; Step S120, the text region is detected from the orientation-corrected ticket image by the ticket text detection model, and multiple text box images are obtained by cropping the ticket image based on the position coordinates of the text boxes; Step S130, the text content corresponding to each text box image is identified by the ticket text recognition model; Step S140, the position coordinates and text content of each text box are input into the ticket field structuring model for structured semantic parsing to generate the structured recognition result of the ticket; wherein, at least one model is trained using training samples that have undergone data augmentation processing.

[0126] The memory 920 may include a readable medium in the form of volatile storage, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0127] The memory 920 may also include a program / utility 9204 having a set (at least one) of program modules 9205, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0128] Bus 930 can represent one or more of several types of bus structures, including a memory bus or memory controller, peripheral bus, graphics acceleration port, processor, or a local bus using any of the various bus structures.

[0129] Electronic device 900 can also communicate with one or more external devices 1000 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0130] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the claims.

Claims

1. A method for recognizing invoices, characterized in that, include: After obtaining the ticket image, the orientation of the ticket is identified by the ticket orientation determination model, and the orientation of the ticket image is corrected according to the orientation identification result; The text region is detected from the direction-corrected ticket image using a ticket text detection model, and multiple text box images are obtained by cropping the ticket image based on the position coordinates of the text boxes. The text content corresponding to each of the text box images is identified using a document text recognition model. The position coordinates and text content of each text box are input into the bill field structured model for structured semantic parsing to generate the structured recognition result of the bill; At least one model is trained using data-augmented training samples.

2. The method according to claim 1, characterized in that, The ticket direction determination model is trained in the following way: Obtain the third training image set containing ticket orientation labels; Data augmentation processing is performed on the images in the third training image set to generate a fourth training image set; the data augmentation processing includes at least one of image flipping, mirror flipping, and random occlusion. The ticket orientation judgment model is trained based on the fourth training image set.

3. The method according to claim 1, characterized in that, The document text recognition model was trained in the following way: Construct a sixth training image set; the sixth training image set contains synthetic images, which are generated by combining text and background. The invoice text recognition model is trained based on the sixth training image set.

4. The method according to claim 3, characterized in that, The synthesized image is obtained through the following method: Construct a corpus containing text content related to multiple target domains; the frequency of rare characters in the corpus is greater than a preset frequency threshold; Cropping the background image from the actual ticket image; The background image covers a variety of imaging degradation modes; The text content from the corpus is randomly inserted into the background image to generate the first image; The first image is degraded to obtain the synthesized image.

5. The method according to claim 1, characterized in that, The step of inputting the position coordinates and text content of each text box into the document field structured model for structured semantic parsing to generate the structured recognition result of the document includes: The position coordinates and text content of each text box are identified using the structured model of the ticket fields to determine their respective field categories. Multiple text contents belonging to the same field category are aggregated to generate complete field values, thereby generating the structured recognition result of the invoice.

6. The method according to claim 5, characterized in that, The invoice field structured model is a multi-classification model, which includes a text semantic extraction module, a text box position extraction module, a feature fusion module, and a classification head module. The text semantic extraction module is used to embed the input text content at the character level and extract its semantic features; The text box position extraction module is used to extract the spatial position features of the text box through a multilayer perceptron network. The feature fusion module is used to splice and fuse the semantic features and spatial location features; The classification header module is used to output the probability distribution of each field category through a fully connected network.

7. The method according to claim 6, characterized in that, The structured model for the invoice fields was trained in the following way: Obtain the seventh training dataset, which includes the coordinates of the text boxes and the labels of the field categories; Data augmentation processing is performed on the seventh training dataset to generate the eighth training dataset; The data augmentation process includes at least one of the following: randomly deleting, replacing, or inserting text content, slightly shifting the coordinates of the text box, randomly deleting part of the text box, and randomly shuffling the input order of the text box. The structured model of the invoice fields was trained based on the eighth training dataset.

8. A ticket recognition device, characterized in that, include: The image processing module is used to identify the orientation of the ticket through a ticket orientation determination model after obtaining the ticket image, and to perform orientation correction on the ticket image based on the orientation identification result; The text detection module is used to detect text regions from the direction-corrected ticket image using a ticket text detection model, and to crop the ticket image based on the position coordinates of the text boxes to obtain multiple text box images. The text recognition module is used to identify the text content corresponding to each text box image through the invoice text recognition model; The structured recognition module is used to input the position coordinates and text content of each text box into the bill field structured model for structured semantic parsing, and generate the structured recognition result of the bill. At least one model is trained using data-augmented training samples.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the ticket recognition method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the ticket recognition method according to any one of claims 1 to 7 by executing the executable instructions.

Images