Bill information identification method and device, equipment, storage medium and program product

Abstract

The invention relates to a bill information identification method and device, equipment, a storage medium and a program product. The method comprises the steps of obtaining an image of a to-be-detected bill; inputting the image of the to-be-detected bill into the text detection model, and obtaining prediction boxes of a plurality of text fields of different character types in the image of the to-be-detected bill output by the text detection model; the text detection model is used for identifying character features of text fields, determining character types to which the corresponding text fields belong according to the character features, and determining prediction boxes of the text fields according to the character types of the text fields; identifying text information in the plurality of prediction frames through a text identification model; and according to a screening rule in the regularization judgment instruction, screening bill information of the to-be-detected bill from the text information in the plurality of prediction boxes. By adopting the method, the recognition accuracy of the bill information can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, storage medium and program product for identifying bill information. Background Art

[0002] At present, tickets are important reimbursement vouchers for related enterprises and units. During the review and entry of tickets, personnel need to first verify the authenticity and compliance of the tickets. After confirmation, they can obtain and enter the ticket information, which includes the invoice code, invoice number, license plate number, date of travel, boarding (disembarking) time, mileage, fare, etc. With the rapid development of artificial intelligence, optical character recognition (OCR) technology can assist in the automatic recognition of ticket information.

[0003] In the related art, the information in the bill image is usually recognized by the OCR technology using a cascade structure, that is, the text area in the bill image is first detected by a text detection method, and then the text area is recognized by a text recognition method.

[0004] However, paper bills may have problems with overlapping or sticking of the original text and printed text due to misalignment. OCR technology recognition using a cascade structure will output the entire line of overlapping or sticky text, thus affecting the recognition accuracy of the bill information. Summary of the invention

[0005] Based on this, it is necessary to provide a method, device, equipment, storage medium and program product for identifying bill information that can improve the accuracy of bill information recognition in order to solve the above technical problems.

[0006] In a first aspect, the present application provides a method for identifying bill information, comprising:

[0007] Acquire an image of the bill to be detected;

[0008] The image of the bill to be detected is input into a text detection model, and prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model are obtained; the text detection model is used to identify text features of the text field, determine the text type to which the corresponding text field belongs according to the text features, and determine the prediction box of the text field according to the text type of the text field;

[0009] Identify text information in multiple prediction boxes through a text recognition model;

[0010] According to the screening rules in the regularized judgment instruction, the bill information of the bill to be detected is screened out from the text information in the multiple prediction boxes.

[0011] In one embodiment, the ticket to be tested is a bus ticket; the text type includes the original font and the printed font on the ticket to be tested; the ticket information includes at least one of the following: ticket code, ticket number, license plate number, boarding date, boarding and alighting time, mileage, and fare.

[0012] In one embodiment, the step of filtering the bill information of the bill to be detected from the text information in the plurality of prediction boxes according to the filtering rule in the regularized judgment instruction includes:

[0013] If the text type of the first text information is the original font on the bill to be detected, all characters in the first text information are numbers and the number of characters in the first text information is a set number of digits, then the first text information is determined to be the bill information of the bill to be detected, and the first text information is any text information in the text information in the multiple prediction boxes.

[0014] In one embodiment, the step of filtering the bill information of the bill to be detected from the text information in the plurality of prediction boxes according to the filtering rule in the regularized judgment instruction includes:

[0015] If a keyword corresponding to the bill information is identified in the second text information, and the text format of the third text information is the text format corresponding to the identified keyword, then the third text information is determined to be the bill information of the bill to be detected;

[0016] Among them, the second text information and the third text information are any text information in the text information in the multiple prediction boxes, the text type of the second text information is the original font on the bill to be detected, the text type of the third text information is the printed font on the bill to be detected, and the horizontal displacement difference of the prediction boxes corresponding to the second text information and the third text information in the image of the bill to be detected is less than the target number of pixels.

[0017] In one embodiment, the method further comprises:

[0018] capturing images of a plurality of first sample bills;

[0019] Annotating the original fonts and the printed fonts in the images of the plurality of first sample bills respectively to form a first training set for the text detection model;

[0020] The pre-trained weights of the text detection model are loaded, and the text detection model is trained using the first training set until the loss function of the text detection model converges.

[0021] In one embodiment, the text recognition model includes a scene text recognition model; the method further includes:

[0022] Acquire multiple images of second sample bills, wherein the images of the second sample bills include a marking box and marking information of a text field in the marking box;

[0023] According to the marking frame, the images of the plurality of second sample bills are cropped to obtain a plurality of marking frame images;

[0024] Generate a second training set for the text recognition model according to the multiple annotation frame images and the annotation information respectively corresponding to the multiple annotation frame images;

[0025] The text recognition model is trained using the second training set until the loss function of the text recognition model converges.

[0026] In a second aspect, the present application provides a device for identifying bill information, comprising:

[0027] An acquisition module, used for acquiring an image of a bill to be detected;

[0028] A positioning module, used for inputting the image of the bill to be detected into a text detection model, and obtaining prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model; the text detection model is used for identifying text features of the text field, determining the text type to which the corresponding text field belongs according to the text features, and determining the prediction box of the text field according to the text type of the text field;

[0029] A recognition module, used to recognize text information in multiple prediction boxes through a text recognition model;

[0030] The screening module is used to screen out the bill information of the bill to be detected from the text information in the multiple prediction boxes according to the screening rules in the regularized judgment instruction.

[0031] In one embodiment, the ticket to be tested is a bus ticket; the text type includes the original font and the printed font on the ticket to be tested; the ticket information includes at least one of the following: ticket code, ticket number, license plate number, boarding date, boarding and alighting time, mileage, and fare.

[0032] In one embodiment, the screening module is also used to determine that the first text information is the bill information of the bill to be detected if the text type of the first text information is the original font on the bill to be detected, all characters in the first text information are numbers, and the number of characters in the first text information is a set number of digits, and the first text information is any text information in the text information in the multiple prediction boxes.

[0033] In one embodiment, the screening module is further configured to determine that the third text information is the bill information of the bill to be detected if a keyword corresponding to the bill information is identified in the second text information and the text format of the third text information is the text format corresponding to the identified keyword;

[0034] Among them, the second text information and the third text information are any text information in the text information in the multiple prediction boxes, the text type of the second text information is the original font on the bill to be detected, the text type of the third text information is the printed font on the bill to be detected, and the horizontal displacement difference of the prediction boxes corresponding to the second text information and the third text information in the image of the bill to be detected is less than the target number of pixels.

[0035] In one embodiment, the bill information recognition device further includes:

[0036] A training module is used to collect images of multiple first sample bills; annotate the original fonts and printed fonts in the images of the multiple first sample bills respectively to form a first training set for the text detection model; load the pre-trained weights of the text detection model, and use the first training set to train the text detection model until the loss function of the text detection model converges.

[0037] In one embodiment, the training module is further used to collect images of multiple second sample bills, the images of the second sample bills include a labeling box and labeling information of a text field in the labeling box; according to the labeling box, the images of the multiple second sample bills are cropped to obtain multiple labeling box images; according to the multiple labeling box images and the labeling information respectively corresponding to the multiple labeling box images, a second training set of the text recognition model is generated; and the text recognition model is trained using the second training set until the loss function of the text recognition model converges.

[0038] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the above-mentioned method for identifying bill information of the first aspect when executing the computer program.

[0039] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for identifying bill information of the first aspect described above is implemented.

[0040] In a fifth aspect, the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for identifying bill information of the first aspect mentioned above.

[0041] The above-mentioned bill information recognition method, device, equipment, storage medium and program product first obtain an image of the bill to be detected; secondly, input the image of the bill to be detected into a text detection model, and obtain the prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model; the text detection model is used to identify the text features of the text field, determine the text type to which the corresponding text field belongs according to the text features, and determine the prediction box of the text field according to the text type of the text field; thirdly, identify the text information in the multiple prediction boxes through the text recognition model; finally, according to the screening rules in the regularized judgment instruction, screen the bill information of the bill to be detected from the text information in the multiple prediction boxes. Since the text detection model can determine the text type to which the text field belongs based on the text features, and then determine the prediction box of the text field based on the text type to which the text field belongs, when two different types of text fields overlap or stick together, they will not be recognized as the prediction box of the same text field, but will be recognized as the prediction boxes of two text fields respectively, so as to perform subsequent text recognition and bill information screening, thereby improving the recognition accuracy of bill information. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 An application environment diagram of a bill information recognition method provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of a method for identifying bill information provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a prediction box of a text field in the related art;

[0046] Figure 4A schematic diagram of a prediction box for a text field provided in an embodiment of the present application;

[0047] Figure 5 A flowchart of another method for identifying bill information provided in an embodiment of the present application;

[0048] Figure 6 A structural block diagram of a bill information recognition device provided in an embodiment of the present application;

[0049] Figure 7 An internal structure diagram of a computer device provided for an application embodiment. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] The method for identifying bill information provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0052] The terminal 102 sends the image of the bill to be detected to the server 104. The server 104 inputs the image of the bill to be detected into the text detection model, and obtains the prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model; the text detection model is used to identify the text features of the text field, determine the text type to which the corresponding text field belongs according to the text features, and determine the prediction box of the text field according to the text type of the text field. Subsequently, the server 104 identifies the text information in the multiple prediction boxes through the text recognition model. Finally, the server 104 filters out the bill information of the bill to be detected from the text information in the multiple prediction boxes according to the screening rules in the regularized judgment instruction.

[0053] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. Head-mounted devices may be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. The server 104 may be an independent physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0054] In an exemplary embodiment, Figure 2 As shown, a method for identifying bill information is provided, and the method is applied to Figure 1 Taking the server in as an example, the method for identifying the ticket information includes S201-S204:

[0055] S201, obtaining an image of a bill to be detected.

[0056] In the present application, when bill detection is required, the server may first obtain an image of the bill to be detected.

[0057] The above-mentioned tickets to be detected may include bus tickets, restaurant tickets, hotel tickets, etc., which are not limited in the present embodiment. In order to ensure the accuracy of ticket information recognition, the image of the ticket to be detected can be an image captured when the front of the ticket to be detected is placed vertically and flat, and the font of the text in the image of the detected ticket needs to be clear and visible.

[0058] It should be understood that the embodiments of the present application do not limit how to obtain the image of the bill to be detected. In some embodiments, the user can take an image of the bill to be detected through a terminal device, and then the terminal device sends the image of the bill to be detected to the server. The terminal device may include a mobile phone, a camera, a scanner, etc. In other embodiments, the server may also directly obtain the image of the bill to be detected from the relevant reimbursement system.

[0059] S202: Input the image of the bill to be detected into a text detection model, and obtain prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model.

[0060] In this step, after the server obtains the image of the bill to be detected, the image of the bill to be detected can be input into the text detection model, and the prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model can be obtained.

[0061] Among them, the above-mentioned prediction box is used to identify the position of the text field in the image of the bill to be detected. It should be understood that the embodiment of the present application does not limit the type of prediction box of the text field. For example, it can include a rectangular box, a circular box, etc.

[0062] Exemplarily, if the predicted box of the text field is a rectangular box, the rectangular box can be identified by the coordinates of the two diagonal corners of the rectangular box. For example, the rectangular box is identified as [x1, y1, x2, y2], (x1, y1) is the coordinate of the upper left corner of the rectangular box, and (x2, y2) is the coordinate of the lower right corner of the rectangular box.

[0063] The text type of the text field includes the original font and the printed font on the bill to be detected. It should be understood that some fonts may be printed on the original bill, and when a consumption behavior occurs, information related to the consumption behavior may be printed on the original bill. The font on the original bill is the original font. The font of the printed information related to the consumption behavior is the printed font.

[0064] The text detection model provided in the embodiments of the present application is described below.

[0065] In some embodiments, the text detection model is used to identify text features of a text field, determine the text type to which the corresponding text field belongs based on the text features, and determine a prediction box for the text field based on the text type of the text field.

[0066] Among them, the above-mentioned text detection model can be any text target detection model, for example, a lightweight yoLov8-seg model, and the text features of the above-mentioned text field can include text color features and shape features, etc.

[0067] It should be understood that the yoLov8-seg model is a variant of the yolov8 model that focuses on instance segmentation. Instance segmentation is a computer vision task that requires the model to not only identify and localize each object in an image, but also generate an accurate pixel-level mask of each object in order to distinguish between instances of the same category but different instances in the image.

[0068] Exemplarily, the text detection model can identify the text paragraphs on the bill to be detected, and then the text detection model extracts the text color features and shape features of the text fields in each text paragraph, classifies the text fields by the color features and shape features, and determines the text fields belonging to the original font and the text fields belonging to the printed font. Finally, the text detection model calibrates the prediction box of the text field belonging to the original font, and calibrates the prediction box of the text field belonging to the printed font, thereby outputting the prediction boxes of multiple text fields of different text types.

[0069] In some embodiments, the server may also collect multiple images of the first sample bills, and then the server respectively annotates the original fonts and printed fonts in the images of the multiple first sample bills to form a first training set of the text detection model. Finally, the server loads the pre-trained weights of the text detection model and trains the text detection model using the first training set until the loss function of the text detection model converges.

[0070] Exemplarily, if the ticket to be detected is a bus ticket, and the text detection model is a lightweight yoLov8-seg model, images of bus tickets (for example, car rental invoices) from multiple regions can be collected as the first sample ticket images, and then the text paragraphs in the bus ticket images can be annotated. The original font on the bus ticket can be marked as text type "0", and the printed font on the bus ticket can be marked as text type "1". After the annotation is completed, the first training set of the lightweight yoLov8-seg model can be formed. Subsequently, the pre-trained weights of the Common Objects in Context are loaded, and the lightweight yoLov8-seg model is trained using the first training set until the loss function (Loss) converges, so that the model has text detection capabilities and text type classification capabilities. Among them, the input size of the first training set is 640x640.

[0071] For example, Figure 3 is a schematic diagram of a prediction box of a text field in the related art, Figure 4 A schematic diagram of a prediction box of a text field provided in an embodiment of the present application. Figure 3 As shown in the figure, the Differentiable Binarization Network (DBNet) model is used for text detection. Since the DBNet model does not have the ability to classify text, it cannot distinguish the original font and the printed font in the bill to be detected. When the printing is misaligned and text overlaps or sticks together, the two text fields will be detected as the same text field by default. This text field is used as the input of the text recognition model, which will directly affect the accuracy of text recognition. Figure 4 As shown, using the text detection model provided in the embodiment of the present application, the original font and the printed copybook of the bill to be detected can be effectively distinguished, providing a guarantee for the recognition accuracy of the text recognition model.

[0072] Among them, the DBNet model is a deep learning model for text detection tasks, which mainly locates text areas in document images. The core of the DBNet model is to use binarization technology to simplify network prediction and can directly generate prediction boxes for text fields.

[0073] S203: Identify text information in multiple prediction boxes using a text recognition model.

[0074] In this step, after the server obtains the prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model, the text information in the multiple prediction boxes can be recognized by the text recognition model.

[0075] It should be understood that the embodiments of the present application do not limit the text recognition model. For example, it can include a visual language (Swin Transformer for Text Recognition, SVTR) model, a deep text detector (‌DeepTextSpotter‌) model, a fast oriented text recognition (Fast Oriented Text Spotting, FOTS) model, etc.

[0076] For example, the embodiment of the present application adopts the SVTR model as a text recognition model. The SVTR model is a single visual model designed for scene text recognition, which is mainly used to recognize the text in the scene. Compared with the traditional two-stage model of feature extraction plus sequence modeling, the SVTR model abandons the reliance on sequential modeling, which makes it more efficient and accurate when dealing with text recognition in complex scenes.

[0077] In some embodiments, the server may also collect multiple images of the second sample bills, wherein the images of the second sample bills include a labeling box and labeling information of the text field in the labeling box. Secondly, the server crops the images of the multiple second sample bills according to the labeling box to obtain multiple labeling box images. Thirdly, the server generates a second training set of the text recognition model according to the multiple labeling box images and the labeling information corresponding to the multiple labeling box images. Finally, the server uses the second training set to train the text recognition model until the loss function of the text recognition model converges.

[0078] The second sample note and the first sample note may be the same sample note or different sample notes, which is not limited in the embodiment of the present application.

[0079] In some embodiments, the annotation box of the image of the second sample bill may be a prediction box output by the text detection model, and the annotation information of the image of the second sample bill may be manually output annotation information, which is used to annotate the text content of the text paragraph in the annotation box. The server may crop along the edge of the annotation box to obtain multiple annotation box images, thereby generating a second training set of the text recognition model with the corresponding annotation information, and then using the second training set to train the text recognition model until the loss function of the text recognition model converges, so that the text recognition model has text recognition capabilities.

[0080] S204: Filter the bill information of the bill to be detected from the text information in the multiple prediction boxes according to the screening rules in the regularized judgment instruction.

[0081] In this step, after the server recognizes the text information in multiple prediction boxes through the text recognition model, the bill information of the bill to be detected can be filtered out from the text information in the multiple prediction boxes according to the filtering rules in the regularized judgment instruction.

[0082] If the ticket to be detected is a bus ticket, the ticket information includes at least one of the following: ticket code, ticket number, license plate number, boarding date, boarding and alighting time, mileage, and bus fare.

[0083] It should be understood that in the present application, different ticket information may correspond to different screening rules.

[0084] In some embodiments, if the text type of the first text information is the original font on the bill to be detected, all characters in the first text information are numbers and the number of characters in the first text information is a set number of digits, then the first text information is determined to be the bill information of the bill to be detected, and the first text information is any text information in the text information in multiple prediction boxes.

[0085] The number of set digits corresponding to different bill information is different. For example, the number of set digits corresponding to the bill code is 12, and the number of set digits corresponding to the bill number is 8.

[0086] Exemplarily, the text field in which all characters of the first text information are numbers and the text type predicted by the text detection model is "0" can be filtered out to determine whether the number of digits of the first text information in the text field is 12. If so, it is determined that the first text information is the invoice code in the bill information.

[0087] Exemplarily, the text field in which all characters of the first text information are numbers and the text detection model predicts the text type as "0" can be filtered out to determine whether the number of digits of the first text information in the text field is 8. If so, it is determined that the first text information is the invoice number in the bill information.

[0088] In other embodiments, if a keyword corresponding to the bill information is identified in the second text information, and the text format of the third text information is the text format corresponding to the identified keyword, the third text information is determined to be the bill information of the bill to be detected.

[0089] Among them, the second text information and the third text information are any text information in the text information in multiple prediction boxes, the text type of the second text information is the original font on the bill to be detected, the text type of the third text information is the printed font on the bill to be detected, and the horizontal displacement difference of the prediction boxes corresponding to the second text information and the third text information in the image of the bill to be detected is less than the target number of pixels.

[0090] It should be understood that the embodiments of the present application do not limit keywords. In some embodiments, the keyword corresponding to the license plate number is "vehicle number", the keyword corresponding to the date of travel is "date", the keywords corresponding to the boarding and alighting times are "boarding" and "getting off", the keywords corresponding to the mileage is "mileage", and the keyword corresponding to the fare is "amount".

[0091] It should be understood that the embodiments of the present application do not impose any restriction on the number of target pixels. In some embodiments, the number of target pixels can be 15, 12, 10, etc. By limiting the horizontal displacement difference of the prediction boxes corresponding to the second text information and the third text information in the image of the bill to be inspected by the number of target pixels, it can be ensured that the third text information is printed on the same line of the second text information.

[0092] Exemplarily, the second text information selected includes the keyword "vehicle number" and the text detection model predicts that the text category of the second text information is "0"; then the text format of the third text information is selected as 7 digits (numbers + letters) or 8 digits (province abbreviation + numbers + letters) and the text detection model predicts that the third text information is "1". At this time, it is determined whether the horizontal displacement difference of the prediction box corresponding to the second text information and the third text information in the image of the bill to be detected is less than 15 pixels. If it is satisfied, the third text information is determined to be the license plate number in the bill information.

[0093] Exemplarily, the second text information selected includes the keyword "date" and the text detection model predicts that the text category of the second text information is "0"; then the text format of the third text information selected is "xxxx year xx month xx day" or "xxxx-xx-xx" and the text detection model predicts that the third text information is "1". At this time, it is determined whether the horizontal displacement difference of the prediction box corresponding to the second text information and the third text information in the image of the ticket to be detected is less than 15 pixels. If it is satisfied, the third text information is determined to be the date of travel in the ticket information.

[0094] Exemplarily, the second text information selected includes the keywords "get on the bus" or "get off the bus", and the text detection model predicts that the text category of the second text information is "0"; then the text format of the third text information selected is "xx:xx", where xx in the text format is a number with range constraints at different positions, and the text detection model predicts that the third text information is "1". At this time, determine whether the horizontal displacement difference between the prediction boxes corresponding to the second text information and the third text information in the image of the ticket to be detected is less than 15 pixels. If so, the third text information is determined to be the boarding and alighting time in the ticket information.

[0095] Exemplarily, the second text information selected includes the keyword "mileage", and the text detection model predicts that the text category of the second text information is "0"; then the text format of the third text information selected is "xx.xxkg" or "xx.xx kilometers", where xx in the text format is a number, and the text detection model predicts that the third text information is "1". At this time, determine whether the horizontal displacement difference between the prediction boxes corresponding to the second text information and the third text information in the image of the bill to be detected is less than 15 pixels. If satisfied, the third text information is determined to be the mileage in the bill information.

[0096] Exemplarily, the second text information selected includes the keyword "amount", and the text detection model predicts that the text category of the second text information is "0"; then the text format of the third text information selected is "xx.xx yuan" or "¥xx.xx", where xx in the text format is a number with range constraints at different positions, and the text detection model predicts that the third text information is "1". At this time, determine whether the horizontal displacement difference of the prediction boxes corresponding to the second text information and the third text information in the image of the ticket to be detected is less than 15 pixels. If satisfied, the third text information is determined to be the fare in the ticket information.

[0097] In this application, through rich and rigorous regular judgment instructions, the prediction box of the text detection model is used to predict the text type, and the recognized text information output by the text recognition model is used. Through layer-by-layer screening of screening rules, the bill information on the bill to be detected is correctly screened out, making the entire system more rigorous and scientific.

[0098] In some embodiments, after the server obtains the bill information of the bill to be tested, the bill information of the bill to be tested can be entered into the reimbursement system, and the reimbursement system verifies the authenticity and compliance of the bill to be tested based on the bill information of the bill to be tested, and finally completes the reimbursement of the bill.

[0099] In this application, through the end-to-end cascade structure between the text detection model, the text recognition model and the regularized judgment instructions, the text detection model is first used to detect the text field, and then the text recognition model is used to recognize the text information. Finally, the text information is filtered through the regularized judgment instructions, the bill information is extracted, and the system is entered to automatically complete the entire process of bill authenticity verification and reimbursement. It is efficient, fast and saves manpower. Moreover, compared with the single-step structure, the recognition accuracy of the cascade mechanism is higher.

[0100] The method for identifying bill information provided by the embodiment of the present application first obtains an image of the bill to be detected; secondly, inputs the image of the bill to be detected into a text detection model, and obtains prediction frames of multiple text fields of different text types in the image of the bill to be detected output by the text detection model; the text detection model is used to identify the text features of the text field, determine the text type to which the corresponding text field belongs according to the text features, and determine the prediction frame of the text field according to the text type of the text field; thirdly, the text information in the multiple prediction frames is identified by the text recognition model; finally, the bill information of the bill to be detected is filtered out from the text information in the multiple prediction frames according to the screening rules in the regularized judgment instruction. Since the text detection model can determine the text type to which the text field belongs based on the text features, and then determine the prediction frame of the text field based on the text type to which the text field belongs, when two different types of text fields overlap or stick together, they will not be identified as the prediction frame of the same text field, but will be identified as the prediction frames of two text fields respectively, so as to perform subsequent text recognition and bill information screening, thereby improving the recognition accuracy of bill information.

[0101] The following describes how to train the text detection model and the text recognition model. Figure 5 A flow chart of another method for identifying bill information provided in an embodiment of the present application is shown as follows: Figure 5 As shown, the method for identifying the bill information includes S301-S311:

[0102] S301, collecting images of a plurality of first sample bills.

[0103] S302: annotate the original fonts and printed fonts in the images of a plurality of first sample bills respectively to form a first training set of a text detection model.

[0104] S303, loading the pre-trained weights of the text detection model, and using the first training set to train the text detection model until the loss function of the text detection model converges.

[0105] S304: Collect multiple images of second sample bills, where the images of the second sample bills include a marking box and marking information of a text field in the marking box.

[0106] S305 . Crop the images of the plurality of second sample bills according to the marked frames to obtain a plurality of marked frame images.

[0107] S306: Generate a second training set for the text recognition model according to the multiple annotation frame images and the annotation information respectively corresponding to the multiple annotation frame images.

[0108] S307 . Use the second training set to train the text recognition model until the loss function of the text recognition model converges.

[0109] S308: Acquire the image of the bill to be detected.

[0110] S309: Input the image of the bill to be detected into the text detection model, and obtain the prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model.

[0111] The text detection model is used to identify text features of a text field, determine the text type to which the corresponding text field belongs based on the text features, and determine the prediction box of the text field based on the text type of the text field.

[0112] S310: Identify text information in multiple prediction boxes using a text recognition model.

[0113] S311 , according to the screening rules in the regularized judgment instruction, screen out the bill information of the bill to be detected from the text information in the multiple prediction boxes.

[0114] The method for identifying bill information provided by the embodiment of the present application first obtains an image of the bill to be detected; secondly, inputs the image of the bill to be detected into a text detection model, and obtains prediction frames of multiple text fields of different text types in the image of the bill to be detected output by the text detection model; the text detection model is used to identify the text features of the text field, determine the text type to which the corresponding text field belongs according to the text features, and determine the prediction frame of the text field according to the text type of the text field; thirdly, the text information in the multiple prediction frames is identified by the text recognition model; finally, the bill information of the bill to be detected is filtered out from the text information in the multiple prediction frames according to the screening rules in the regularized judgment instruction. Since the text detection model can determine the text type to which the text field belongs based on the text features, and then determine the prediction frame of the text field based on the text type to which the text field belongs, when two different types of text fields overlap or stick together, they will not be identified as the prediction frame of the same text field, but will be identified as the prediction frames of two text fields respectively, so as to perform subsequent text recognition and bill information screening, thereby improving the recognition accuracy of bill information.

[0115] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0116] Based on the same inventive concept, the embodiment of the present application also provides a bill information recognition device for implementing the bill information recognition method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more bill information recognition devices provided below can refer to the limitations of the bill information recognition method above, and will not be repeated here.

[0117] In an exemplary embodiment, Figure 6 As shown, a bill information recognition device 400 is provided, comprising: an acquisition module 401, a positioning module 402, an identification module 403 and a screening module 404, wherein:

[0118] The acquisition module 401 is used to acquire the image of the bill to be detected.

[0119] The positioning module 402 is used to input the image of the bill to be detected into the text detection model, and obtain the prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model; the text detection model is used to identify the text features of the text field, determine the text type to which the corresponding text field belongs based on the text features, and determine the prediction box of the text field based on the text type of the text field.

[0120] The recognition module 403 is used to recognize text information in multiple prediction boxes through a text recognition model.

[0121] The screening module 404 is used to screen out the bill information of the bill to be detected from the text information in the multiple prediction boxes according to the screening rules in the regularized judgment instruction.

[0122] In one embodiment, the ticket to be tested is a bus ticket; the text type includes the original font and the printed font on the ticket to be tested; the ticket information includes at least one of the following: ticket code, ticket number, license plate number, boarding date, boarding and alighting time, mileage, and fare.

[0123] In one embodiment, the filtering module 404 is also used to determine that the first text information is the bill information of the bill to be detected if the text type of the first text information is the original font on the bill to be detected, all characters in the first text information are numbers, and the number of characters in the first text information is a set number of digits, and the first text information is any text information in the text information in multiple prediction boxes.

[0124] In one embodiment, the screening module 404 is further used to determine that the third text information is the bill information of the bill to be detected if a keyword corresponding to the bill information is identified in the second text information and the text format of the third text information is the text format corresponding to the identified keyword.

[0125] Among them, the second text information and the third text information are any text information in the text information in multiple prediction boxes, the text type of the second text information is the original font on the bill to be detected, the text type of the third text information is the printed font on the bill to be detected, and the horizontal displacement difference of the prediction boxes corresponding to the second text information and the third text information in the image of the bill to be detected is less than the target number of pixels.

[0126] In one embodiment, the bill information recognition device 400 further includes:

[0127] A training module is used to collect images of multiple first sample bills; annotate the original fonts and printed fonts in the images of the multiple first sample bills respectively to form a first training set for a text detection model; load the pre-trained weights of the text detection model, and use the first training set to train the text detection model until the loss function of the text detection model converges.

[0128] In one embodiment, the training module is further used to collect images of multiple second sample bills, the images of the second sample bills include a labeling box and labeling information of a text field in the labeling box; according to the labeling box, the images of the multiple second sample bills are cropped to obtain multiple labeling box images; according to the multiple labeling box images and the labeling information corresponding to the multiple labeling box images, a second training set of the text recognition model is generated; and the text recognition model is trained using the second training set until the loss function of the text recognition model converges.

[0129] Each module in the above-mentioned bill information recognition device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0130] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 7 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for identifying bill information is implemented.

[0131] Those skilled in the art will understand that Figure 7 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0132] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the above-mentioned bill information recognition method when executing the computer program.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned bill information recognition method is implemented.

[0134] In one embodiment, a computer program product is provided, including a computer program, which implements the above-mentioned bill information recognition method when executed by a processor.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0137] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0138] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for identifying bill information, characterized in that: The method comprises: Acquire an image of the bill to be detected; The image of the bill to be detected is input into a text detection model, and prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model are obtained; the text detection model is used to identify text features of the text field, determine the text type to which the corresponding text field belongs according to the text features, and determine the prediction box of the text field according to the text type of the text field; Identify text information in multiple prediction boxes through a text recognition model; According to the screening rules in the regularized judgment instruction, the bill information of the bill to be detected is screened out from the text information in the multiple prediction boxes.

2. The method according to claim 1, characterized in that The ticket to be tested is a bus ticket; the text type includes the original font and the printed font on the ticket to be tested; the ticket information includes at least one of the following: ticket code, ticket number, license plate number, boarding date, boarding and alighting time, mileage, and bus fare.

3. The method according to claim 2, characterized in that The step of filtering the bill information of the bill to be detected from the text information in the plurality of prediction boxes according to the filtering rule in the regularized judgment instruction includes: If the text type of the first text information is the original font on the bill to be detected, all characters in the first text information are numbers and the number of characters in the first text information is a set number of digits, then the first text information is determined to be the bill information of the bill to be detected, and the first text information is any text information in the text information in the multiple prediction boxes.

4. The method according to claim 2, characterized in that: The step of filtering the bill information of the bill to be detected from the text information in the plurality of prediction boxes according to the filtering rule in the regularized judgment instruction includes: If a keyword corresponding to the bill information is identified in the second text information, and the text format of the third text information is the text format corresponding to the identified keyword, then the third text information is determined to be the bill information of the bill to be detected; Among them, the second text information and the third text information are any text information in the text information in the multiple prediction boxes, the text type of the second text information is the original font on the bill to be detected, the text type of the third text information is the printed font on the bill to be detected, and the horizontal displacement difference of the prediction boxes corresponding to the second text information and the third text information in the image of the bill to be detected is less than the target number of pixels.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: capturing images of a plurality of first sample bills; Annotating the original fonts and the printed fonts in the images of the plurality of first sample bills respectively to form a first training set for the text detection model; The pre-trained weights of the text detection model are loaded, and the text detection model is trained using the first training set until the loss function of the text detection model converges.

6. The method according to any one of claims 1 to 4, characterized in that: The text recognition model includes a scene text recognition model; the method further includes: Acquire multiple images of second sample bills, wherein the images of the second sample bills include a marking box and marking information of a text field in the marking box; According to the marking frame, the images of the plurality of second sample bills are cropped to obtain a plurality of marking frame images; Generate a second training set for the text recognition model according to the multiple annotation frame images and the annotation information respectively corresponding to the multiple annotation frame images; The text recognition model is trained using the second training set until the loss function of the text recognition model converges.

7. A device for identifying bill information, characterized in that: The device comprises: An acquisition module, used for acquiring an image of a bill to be detected; A positioning module, used for inputting the image of the bill to be detected into a text detection model, and obtaining prediction boxes of multiple text fields of different text types in the image of the bill to be detected output by the text detection model; the text detection model is used for identifying text features of the text field, determining the text type to which the corresponding text field belongs according to the text features, and determining the prediction box of the text field according to the text type of the text field; A recognition module, used to recognize text information in multiple prediction boxes through a text recognition model; The screening module is used to screen out the bill information of the bill to be detected from the text information in the multiple prediction boxes according to the screening rules in the regularized judgment instruction.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images