Bill recognition method, device, electronic device, and medium based on target detection

Abstract

The present invention belongs to the field of bill recognition, and provides a bill recognition method, device, electronic device, and medium based on target detection. The method includes: inputting a bill image into a pre-trained target detection model; determining the bill area of ​​the target bill and the identification area of ​​the reference identification from the bill image; determining the offset angle of the target bill in the bill image according to the bill center coordinates of the bill area and the identification center coordinates of the identification area; determining target adjustment information according to the offset angle and the reference angle, adjusting the bill image according to the target adjustment information, and obtaining the target image; performing OCR recognition processing on the target image to obtain the bill information. According to the technical solution of this embodiment, the target bill and the reference identification area can be determined by the target detection model, and the target bill can be adjusted to the positive direction according to the obtained target adjustment information, ensuring that the OCR recognition processing can correctly obtain the bill information, and effectively improving the success rate and accuracy of bill recognition.

Description

Technical Field

[0001] The present invention belongs to the technical field of bill recognition, and in particular relates to a bill recognition method, device, electronic equipment, and medium based on target detection. Background Art

[0002] At present, with the development of electronic office, more and more paper bills need to be entered into the electronic system. The manual input method can no longer meet today's needs. The use of optical character recognition (OCR) technology to extract bill information from bill images is the development direction of the automated system. The common ways to enter bill images are scanning or taking photos. Since the bill information of each bill has a specific layout, the recognition position of the bill information is fixed. In order to ensure the accuracy of bill recognition, if the bill direction is not correct when taking photos or scanning, the area corresponding to the recognition position does not have the corresponding bill information, which affects the accuracy of bill recognition. Summary of the invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present invention provide a bill recognition method, device, electronic device, and medium based on target detection, which can adjust the direction of a bill image in the bill image to ensure the accuracy of bill recognition.

[0005] In a first aspect, an embodiment of the present invention provides a bill recognition method based on target detection, comprising:

[0006] Obtain a bill image to be identified, and input the bill image into a pre-trained object detection model;

[0007] Determining a bill region of a target bill and a marking region of a reference marking of the target bill from the bill image by using the target detection model;

[0008] Determine the bill center coordinates of the bill area and the identification center coordinates of the identification area, and determine the offset angle of the target bill in the bill image according to the bill center coordinates and the identification center coordinates;

[0009] Determining target adjustment information according to the offset angle and a preset reference angle, and adjusting the bill image according to the target adjustment information to obtain a target image;

[0010] Perform OCR recognition processing on the target image to obtain the bill information of the target bill.

[0011] In some embodiments, determining the bill region of the target bill and the identification region of the reference identification of the target bill from the bill image by using the target detection model includes:

[0012] Determining a plurality of bill candidate regions and a plurality of identification candidate regions from the bill image by using the object detection model;

[0013] When the intersection-and-union ratio of the bill candidate area and the identification candidate area is 1, the corresponding bill candidate area is determined as the bill area of ​​the target bill, and the corresponding identification candidate area is determined as the identification area of ​​the reference identification.

[0014] In some embodiments, before determining a plurality of bill candidate regions and a plurality of identification candidate regions from the bill image by the object detection model, the method further includes:

[0015] Determining the bill type of the target bill from the bill image;

[0016] The reference identifier of the target document is determined according to the document type.

[0017] In some embodiments, the offset angle of the target bill in the bill image is determined according to the bill center coordinates and the marker center coordinates, and is obtained by the following formula:

[0018] ;

[0019] in, is the offset angle, are the center coordinates of the marker, is the center coordinate of the bill.

[0020] In some embodiments, determining the target adjustment information according to the offset angle and a preset reference angle includes:

[0021] Determine a target angle difference between the offset angle and the reference angle, and determine a target rotation angle and a target rotation direction according to the target angle difference;

[0022] The target rotation angle and the target rotation direction are determined as the target adjustment information.

[0023] In some embodiments, the number of the reference angles is at least two, and determining the angle difference between the offset angle and the reference angle, and determining the target rotation angle and the target rotation direction according to the angle difference, comprises:

[0024] Determine a plurality of candidate angle differences according to the offset angle and a plurality of the reference angles;

[0025] Determine the candidate angle difference with the smallest absolute value as the target angle difference;

[0026] When the target angle difference is a positive number, the target rotation direction is determined to be clockwise, and when the target angle difference is a negative number, the target rotation direction is determined to be counterclockwise.

[0027] In some embodiments, the target detection model is trained by the following method:

[0028] Acquire a plurality of sample bill images, wherein the sample bill images are pre-marked with marking information, wherein the marking information includes a marked bill area and a marked identification area;

[0029] A plurality of sample bill images are input into the target detection model, and the target detection model is trained until convergence.

[0030] In a second aspect, an embodiment of the present invention provides a bill recognition device based on target detection, comprising:

[0031] An image acquisition unit, used to acquire a bill image to be identified, and input the bill image into a pre-trained target detection model;

[0032] A detection unit, configured to determine a bill region of a target bill and a marking region of a reference marking of the target bill from the bill image by using the target detection model;

[0033] A positioning unit, used to determine the bill center coordinates of the bill area and the identification center coordinates of the identification area, and determine the offset angle of the target bill in the bill image according to the bill center coordinates and the identification center coordinates;

[0034] An image adjustment unit, configured to determine target adjustment information according to the offset angle and a preset reference angle, and adjust the bill image according to the target adjustment information to obtain a target image;

[0035] The recognition unit is used to perform OCR recognition processing on the target image to obtain the bill information of the target bill.

[0036] In a third aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the target detection-based bill recognition method as described in the first aspect is implemented.

[0037] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program is used to execute the target detection-based bill recognition method as described in the first aspect.

[0038] The embodiment of the present invention includes: obtaining a bill image to be identified, inputting the bill image into a pre-trained target detection model; determining the bill area of ​​the target bill and the identification area of ​​the reference identification of the target bill from the bill image through the target detection model; determining the bill center coordinates of the bill area and the identification center coordinates of the identification area, and determining the offset angle of the target bill in the bill image according to the bill center coordinates and the identification center coordinates; determining target adjustment information according to the offset angle and a preset reference angle, adjusting the bill image according to the target adjustment information, and obtaining the target image; performing OCR recognition processing on the target image to obtain the bill information of the target bill. According to the technical solution of this embodiment, the target bill and the reference identification area can be determined by the target detection model, and the target bill can be adjusted to the positive direction according to the obtained target adjustment information, ensuring that the OCR recognition processing can correctly obtain the bill information, effectively improving the success rate and accuracy of bill recognition.

[0039] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0041] Figure 1 is a flow chart of a bill recognition method based on target detection provided by an embodiment of the present invention;

[0042] Figure 2 is a schematic diagram of a bill image provided by another embodiment of the present invention;

[0043] Figure 3 is a flow chart of determining a bill area and a marking area provided by another embodiment of the present invention;

[0044] Figure 4 is a flowchart of determining a reference identifier provided by another embodiment of the present invention;

[0045] Figure 5 is a flow chart of determining target adjustment information provided by another embodiment of the present invention;

[0046] Figure 6 is a schematic diagram of determining target adjustment information according to multiple reference angles provided by another embodiment of the present invention;

[0047] Figure 7 is a flowchart of a training target detection model provided by another embodiment of the present invention;

[0048] Figure 8 is a structural diagram of a bill recognition device based on target detection provided by another embodiment of the present invention;

[0049] Fig. 9 is a device diagram of an electronic device provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0052] The present invention provides a method, device, electronic device, and medium for bill recognition based on target detection, the method comprising: obtaining a bill image to be recognized, inputting the bill image into a pre-trained target detection model; determining the bill area of ​​the target bill and the identification area of ​​the reference identification of the target bill from the bill image through the target detection model; determining the bill center coordinates of the bill area and the identification center coordinates of the identification area, and determining the offset angle of the target bill in the bill image according to the bill center coordinates and the identification center coordinates; determining target adjustment information according to the offset angle and a preset reference angle, adjusting the bill image according to the target adjustment information, and obtaining the target image; performing OCR recognition processing on the target image to obtain the bill information of the target bill. According to the technical solution of this embodiment, the target bill and the reference identification area can be determined by the target detection model, and the target bill can be adjusted to the positive direction according to the obtained target adjustment information, ensuring that the OCR recognition processing can correctly obtain the bill information, effectively improving the success rate and accuracy of bill recognition.

[0053] The embodiments of the present application can compile, acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application device that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0054] The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction devices, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0055] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0056] Computer vision technology (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0057] It should be noted that the data of the embodiments of the present invention can be stored in a server. The server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), as well as big data and artificial intelligence platforms.

[0058] OCR is the process by which an electronic device (such as a scanner or digital camera) examines characters printed on paper, determines their shape by detecting dark and light patterns, and then uses character recognition methods to translate the shape into computer text.

[0059] like Figure 1 As shown, Figure 1 The flowchart of a bill recognition method based on target detection provided by an embodiment of the present invention includes but is not limited to the following steps:

[0060] Step S110, obtaining a bill image to be identified, and inputting the bill image into a pre-trained object detection model;

[0061] Step S120, determining a bill region of a target bill and a marking region of a reference marking of the target bill from the bill image by using a target detection model;

[0062] Step S130, determining the bill center coordinates of the bill area and the mark center coordinates of the mark area, and determining the offset angle of the target bill in the bill image according to the bill center coordinates and the mark center coordinates;

[0063] Step S140, determining target adjustment information according to the offset angle and a preset reference angle, and adjusting the bill image according to the target adjustment information to obtain a target image;

[0064] Step S150, performing OCR recognition processing on the target image to obtain the bill information of the target bill.

[0065] It should be noted that since the bill image can be recorded by taking a photo or scanning, taking a photo as an example, it is difficult to ensure that the shooting frame is completely aligned with the bill. Even if a fixed high-definition scanner is used for shooting, the operator may cause the bill to shift when placing it on the shooting area. For examples of the target bill shifting in the bill image, please refer to Figure 2 As shown, in Figure 2In the bill image 200 shown, the target bill 211 is offset from the border of the bill image 200 at a certain angle. At this time, if the preset style of the target bill is used for OCR recognition, recognition errors are likely to occur due to area mismatch or text tilt, affecting the accuracy of bill recognition. In order to solve this problem, the offset of the target bill needs to be corrected to ensure that regular OCR recognition can be performed on the bill image.

[0066] It should be noted that, since the target bill has a certain offset in the bill image, not all of the bill image is the target bill. In this embodiment, the bill area of ​​the target bill is detected by the target detection model to determine, for example, Figure 2 In the bill image 200 shown, the bill region 212 of the target bill 211 can be determined by the target detection model, and subsequent processing of the bill region 212 can effectively eliminate the interference of the background image on bill recognition.

[0067] It should be noted that the process of performing target detection on an image can be implemented through a deep learning algorithm, for example, through a two-stage target detection algorithm such as a regional algorithm based on convolutional neural network features (R-CNN) and a fast regional algorithm based on convolutional neural network features (Faster R-CNN), or can be implemented through a one-stage target detection algorithm such as a You Only Look Once (YOLO) and a single shot multi-box detector (Single Shot MultiBoxDetector, SSD). The specific algorithm can be selected according to actual needs, and this embodiment does not impose too many restrictions on this.

[0068] It should be noted that the reference identifier can be determined according to different bill types. For example, most bills have a seal, and the position of the seal is relatively fixed. Figure 2 As shown, taking the reference mark 231 in the upper middle part of the target bill 211 as an example, during the training process of the target detection model, the seal can be pre-set as the reference mark, and the corresponding area can be marked in the sample image, so that the target detection model can obtain the reference mark. Figure 2 After the receipt image 200 is shown, a secondary object detection can be performed from the receipt area 212 to determine the identification area 232 of the seal 231.

[0069] It should be noted that, after determining the mark area and the bill area, the present embodiment determines the mark center coordinates from the mark area and determines the bill center coordinates from the bill area. Since the position of the reference mark is usually fixed, the angle between the line segment between the mark center coordinates and the bill center coordinates and the vertical line or parallel line of the bill image is positively correlated with the offset angle of the target bill. For example, Figure 2 As shown, the center reference line 221 between the center coordinates of the mark and the center coordinates of the bill and the horizontal reference line 222 parallel to the upper and lower sides of the bill image 200 form an angle α. When the position of the target bill is in the positive direction, α is 90 degrees. Therefore, after determining the offset angle, the angle and direction required to correct the target bill can be determined with 90 degrees as the reference angle, that is, the target adjustment information. According to the above technical solution, no matter how the target bill is placed in the bill image, the target adjustment information can be determined by the offset angle, so that the target bill is adjusted to the positive direction, so that the bill information can be recognized from the bill image through conventional OCR recognition.

[0070] It should be noted that the method for determining the offset angle can be selected according to actual needs, such as Figure 2 As shown, the offset angle is determined according to the horizontal reference line 222 and the center reference line 221, the reference angle is 90 degrees, for example, the calculated offset angle is 95 degrees, then the offset angle is 5 degrees, the specific calculation method can be determined according to the selection of the reference angle and the reference line, and no further limitation is made here. The target adjustment information may include a rotation angle and a rotation direction, such as 5 degrees counterclockwise rotation, 25 degrees clockwise rotation, etc., and the specific form of the target adjustment information is not further limited here.

[0071] It should be noted that after obtaining the target image according to the target adjustment information, the target bill in the target image has been adjusted to the positive direction. Then, the bill information can be obtained by performing bill recognition on the target bill according to conventional OCR recognition technology. The specific recognition process is not an improvement made in this embodiment and will not be elaborated here.

[0072] In addition, in one embodiment, reference Figure 3 , Figure 1 Step S120 of the illustrated embodiment also includes but is not limited to the following steps:

[0073] Step S310, determining a plurality of bill candidate regions and a plurality of identification candidate regions from the bill image by using an object detection model;

[0074] Step S320, when the intersection-and-union ratio of the bill candidate region and the identification candidate region is 1, the corresponding bill candidate region is determined as the bill region of the target bill, and the corresponding identification candidate region is determined as the identification region of the reference identification.

[0075] It should be noted that in order to improve the efficiency of bill recognition, multiple bills may be recorded in the bill image at the same time. For example, the image is recorded in a manner of arranging multiple bills in sequence, and each bill is recognized separately by the recognition system to obtain multiple bill information. In order to correct the angle of the bill, the technical solution of this embodiment can be executed independently for each bill, that is, the offset angle of each bill is determined. Since the determination of the offset angle depends on the fixed relative position of the reference mark and the target bill, it is necessary to first distinguish the images of different bills from the bill image.

[0076] It should be noted that, for the target detection model, multiple bill candidate areas and multiple identification candidate areas can be quickly detected from the bill image. On this basis, it is only necessary to determine the identification candidate area associated with each bill candidate area to distinguish different target bills.

[0077] It should be noted that Intersection-over-Union (IoU) is a concept often used in target detection, which is the overlap ratio between the generated candidate box and the original marked box, that is, the ratio of their intersection to their union. When the candidate box is located in the original marked box, the ratio of the IoU is 1. In this embodiment, the calculation of the IoU can be completed by the following formula: ,in, is the candidate area for the bill, To identify the candidate area, the reference identifier of this embodiment is located in the target bill, so the identification candidate area is the candidate frame, and the bill candidate area is the original marking frame. When the intersection and union ratio of an identification candidate area and a bill candidate area is 1, it can be determined that the identification candidate area is located in the voting candidate area, and the two belong to the same target bill. Of course, the positional relationship between the bill candidate area and the identification candidate area can also be determined in other forms. For example, the pixel coordinates of the identification candidate area are all located in the bill candidate area. Those skilled in the art are motivated to select a specific method according to the actual situation, and it is sufficient to determine that the bill candidate area and the identification candidate area belong to the same target bill.

[0078] It should be noted that when multiple target bills are determined from the bill image according to the intersection-union ratio, the following steps can be performed for each target bill in turn: Figure 1 The operation of the illustrated embodiment is to identify the bill information from each target bill, and the present embodiment will not elaborate on the repeatedly executed steps.

[0079] In addition, in one embodiment, reference Figure 4 , in execution Figure 3 Before step S310 in the illustrated embodiment, the following steps are also included but not limited to:

[0080] Step S410, determining the bill type of the target bill from the bill image;

[0081] Step S420: determining the reference identifier of the target bill according to the bill type.

[0082] It should be noted that since different bills have different reference marks, for example, common invoices not only include seals but also have QR codes set at fixed positions, technical personnel in this field are motivated to have different reference marks according to actual needs, and can pre-determine the position of the reference mark so that the target detection model can learn the shape of the reference mark from this position during the training process.

[0083] It should be noted that in order for the target detection model to detect the identification area of ​​the reference identification from the bill image, it is necessary to mark the identification area and the reference identification in the training sample. Therefore, it is necessary to determine the reference identification corresponding to different bill types before training. For example, two types of bill annotations and reference identification annotations can be pre-defined, and the bill type of each sample can be determined, so that the target detection model can learn the reference identification corresponding to different bill types during the training process. The specific annotation process is not limited here.

[0084] It should be noted that in order to determine the reference identifier according to the bill type of the target bill, the bill type can be input at the same time as the bill image, so that the target detection model can directly determine the bill type based on the input information. Of course, the target detection model can also perform simple image recognition on the bill image and determine the bill type based on the bill's typesetting or keywords. The specific method can be selected according to actual needs, and no further limitations will be made here.

[0085] In addition, in one embodiment, reference Figure 4 , Figure 1 Step S130 of the illustrated embodiment is obtained by the following formula:

[0086] ;

[0087] in, is the offset angle, To identify the center coordinates, The coordinates of the bill center.

[0088] It should be noted that after determining the center coordinates of the marker and the center coordinates of the bill, a line segment can be obtained based on the two points. Based on this, a right triangle can be made parallel or perpendicular to the bill image, so as to calculate the offset angle according to the trigonometric functions shown above. The specific calculation process is a technology well known to technicians in this field and will not be elaborated here.

[0089] In addition, in one embodiment, reference Figure 5 , Figure 1 Step S140 of the illustrated embodiment also includes but is not limited to the following steps:

[0090] Step S510, determining a target angle difference between the offset angle and the reference angle, and determining a target rotation angle and a target rotation direction according to the target angle difference;

[0091] Step S520: determining the target rotation angle and the target rotation direction as target adjustment information.

[0092] It should be noted that since the offset angle represents the inclination angle of the target bill in the bill image, it is not an angle directly used for adjustment. Since bill recognition requires the target bill to be adjusted to a reference position, for example, the border of the bill area is parallel to the bill image, when an offset angle is available, it is necessary to make adjustments based on the reference angle corresponding to the reference position. For example, in the above embodiment, 90 is used as the reference angle. When the offset angle is calculated to be 95 degrees, the target angle difference is 5 degrees, and the rotation direction is clockwise. Based on this, the bill image is rotated 5 degrees clockwise to obtain a bill image in the positive direction.

[0093] It should be noted that the target rotation direction can be clockwise or counterclockwise, and the specific direction can be selected according to actual needs. For example, in order to reduce the process of determining the direction, the clockwise direction is pre-set as the target rotation direction. For example, in order to improve the rotation efficiency, the direction with a smaller rotation angle is used as the target rotation direction. This embodiment does not impose any restrictions on this.

[0094] In addition, in one embodiment, the number of reference angles is at least two. Figure 6 , Figure 5 Step S520 of the illustrated embodiment also includes but is not limited to the following steps:

[0095] Step S610, determining a plurality of candidate angle differences according to the offset angle and a plurality of reference angles;

[0096] Step S620, determining the candidate angle difference with the smallest absolute value as the target angle difference;

[0097] Step S630: When the target angle difference is a positive number, the target rotation direction is determined to be clockwise; when the target angle difference is a negative number, the target rotation direction is determined to be counterclockwise.

[0098] It should be noted that, since the bill image can be rotated, to adjust the target bill to the positive direction, it is actually necessary to adjust the bill image to be parallel to the border of the bill image in any corresponding manner. For example, the upper border of the bill image is parallel to the upper border, lower border, left border or right border of the bill image. After achieving any kind of alignment, the direction of the target bill can be corrected by adjusting the bill image. For example, the upper border of the bill image is parallel to and close to the lower border of the bill image. At this time, the target bill is upside down in the bill image. At this time, rotating the bill image 180 degrees can achieve the target bill in the positive direction.

[0099] It should be noted that, based on the above, multiple reference angles can be pre-set and multiple candidate angle differences can be determined. In order to reduce the impact of image rotation on image clarity, this embodiment selects the candidate angle difference with the smallest absolute value as the target angle difference, so as to rotate with the smallest target angle difference. At the same time, the rotation direction is determined according to the positive and negative value of the target angle difference. For example, if the target angle difference is 5 degrees, it represents a clockwise rotation of 5 degrees. For example, if the target angle difference is -5 degrees, it represents a counterclockwise rotation of 5 degrees.

[0100] In order to better illustrate the technical solution of this embodiment, a specific example is provided below. In this example, the offset angle is 95 degrees, and the reference angle set is [-90, 0, 90, 180]. The candidate angle difference is calculated according to the following formula: ,in, is the offset angle, The matrix composed of base angles is [-90, 0, 90, 180], It is a unit matrix with one row and four columns [1,1,1,1]. The resulting set of candidate angle differences is [185,95,5,-85]. The candidate angle difference with the smallest absolute value is 5. Therefore, the target angle difference is 5 degrees. Therefore, a clockwise rotation of 5 degrees can achieve the correction of the target bill.

[0101] In addition, in one embodiment, reference Figure 7 ,The training steps of the target detection model include but are not limited to:

[0102] Step S710, obtaining a plurality of sample bill images, wherein the sample bill images are pre-annotated with annotation information, and the annotation information includes an annotated bill area and an annotated identification area;

[0103] Step S720: input a plurality of sample bill images into the target detection model, and train the target detection model until convergence.

[0104] It should be noted that the labeling of the sample bill image can be carried out in the form of the coco data set. The reference mark and the target bill area are marked in the sample bill image to obtain the marked bill area and the marked mark area, and the center point, width and height of each area are marked respectively, so that the target detection model can be trained according to the labeled information, so that the trained target detection model can regress the bill area and the reference mark area and the center point coordinates from the input image. The specific training process is a technology well known to technicians in this field and will not be elaborated here.

[0105] It should be noted that in order to verify whether the target detection model has been trained to convergence, an unannotated test image can be input into the target detection model. When the output value of the target detection model is the same as the actual value of the bill area and the logo area or the difference meets the preset threshold, it can be determined that the target detection model has been trained to convergence.

[0106] In addition, refer to Figure 8 The embodiment of the present invention provides a bill recognition device based on target detection. The bill recognition device based on target detection 800 includes:

[0107] The image acquisition unit 810 is used to acquire the bill image to be identified and input the bill image into a pre-trained object detection model;

[0108] A detection unit 820, configured to determine a bill region of a target bill and a marking region of a reference marking of the target bill from the bill image by using a target detection model;

[0109] A positioning unit 830, used to determine the bill center coordinates of the bill area and the mark center coordinates of the mark area, and determine the offset angle of the target bill in the bill image according to the bill center coordinates and the mark center coordinates;

[0110] An image adjustment unit 840 is used to determine target adjustment information according to the offset angle and a preset reference angle, and adjust the bill image according to the target adjustment information to obtain a target image;

[0111] The recognition unit 850 is used to perform OCR recognition processing on the target image to obtain the bill information of the target bill.

[0112] In addition, refer to Fig. 9 An embodiment of the present invention further provides an electronic device, the electronic device 900 comprising: a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920.

[0113] The processor 920 and the memory 910 may be connected via a bus or in other ways.

[0114] The non-transient software program and instructions required to implement the target detection-based bill recognition method of the above embodiment are stored in the memory 910. When executed by the processor 920, the target detection-based bill recognition method of the above embodiment is executed, for example, the above described Figure 1 Steps S110 to S150 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S420 of the method, Figure 5 Steps S510 to S520 of the method, Figure 6 Steps S610 to S630 of the method, Figure 7 Method steps S710 to S720.

[0115] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the above processor can execute the object detection-based bill recognition method in the above embodiment, for example, execute the above described Figure 1 Steps S110 to S150 of the method, Figure 3 Steps S310 to S320 of the method, Figure 4 Steps S410 to S420 of the method, Figure 5 Steps S510 to S520 of the method, Figure 6 Steps S610 to S630 of the method, Figure 7Method steps S710 to S720 in the method. It will be appreciated by those skilled in the art that all or some of the steps and devices in the method disclosed above may be implemented as software, firmware, hardware, and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable storage medium, which may include a computer storage medium (or a non-transitory storage medium) and a communication storage medium (or a temporary storage medium). As is well known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable storage media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other storage medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication storage media generally contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery storage media.

[0117] The present embodiment can be used in many general or special computer device environments or configurations. For example: personal computers, server computers, handheld electronic devices or portable electronic devices, tablet electronic devices, multiprocessor devices, microprocessor-based devices, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above devices or electronic devices, etc. The present application can be described in the general context of a computer program executed by a computer, such as a program module. Generally, a program module includes routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing electronic devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage electronic devices.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of the code contains one or more programs for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0119] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0120] It should be noted that, although several modules or units of the electronic device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0121] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing electronic device (which can be a personal computer, a server, a touch terminal, or a network electronic device, etc.) to execute the method according to the implementation method of the present application.

[0122] The electronic device of this embodiment may include: a radio frequency (RF) circuit, a memory, an input unit, a display unit, a sensor, an audio circuit, a wireless fidelity (WiFi) module, a processor, and a power supply. The RF circuit can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is processed by the processor; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to the Global System of Mobile communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, Short Messaging Service (SMS), etc. The memory can be used to store software programs and modules, and the processor executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the storage data area may store data created according to the use of the electronic device (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The input unit may be used to receive input digital or character information, and to generate key signal inputs related to the settings and function control of the electronic device. Specifically, the input unit may include a touch panel and other input devices. The touch panel, also known as a touch screen, can collect touch operations on or near it (such as operations using any suitable object or accessory such as a finger, a stylus, etc. on or near the touch panel), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel may include two parts: a touch detection device and a touch controller.Among them, the touch detection device detects the touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor, and can receive the command sent by the processor and execute it. In addition, the touch panel can be implemented in various categories such as resistive, capacitive, infrared and surface acoustic waves. In addition to the touch panel, the input unit can also include other input devices. Specifically, other input devices may include but are not limited to one or more of a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, a joystick, etc. The display unit can be used to display the input information or the information provided and various menus of the electronic device. The display unit may include a display panel, and optionally, the display panel can be configured in the form of a liquid crystal display (LiquidCrystal Display, LCD), an organic light-emitting diode (Organic Light-Emitting Diode, OLED), etc. Further, the touch panel can cover the display panel, and when the touch panel detects a touch operation on or near it, it is transmitted to the processor to determine the category of the touch event, and then the processor provides a corresponding visual output on the display panel according to the category of the touch event. The touch panel and the display panel are two independent components to realize the input and output functions of the electronic device, but in some embodiments, the touch panel and the display panel can be integrated to realize the input and output functions of the electronic device. The electronic device may also include at least one sensor, such as a light sensor, a motion sensor and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel according to the brightness of the ambient light, and the proximity sensor may turn off the display panel and / or backlight when the electronic device is moved to the ear. As a kind of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of electronic devices (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can be configured in electronic devices, they will not be repeated here. The audio circuit, speaker, and microphone can provide an audio interface. The audio circuit can transmit the electrical signal converted from the received audio data to the speaker, which converts it into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit and converted into audio data. The audio data is then output to the processor for processing and then sent to another electronic device through the RF circuit, or the audio data is output to a memory for further processing.

[0123] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0124] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

[0125] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A bill recognition method based on target detection, characterized in that: include: Obtain a bill image to be identified, and input the bill image into a pre-trained object detection model; Determining a bill region of a target bill and a marking region of a reference marking of the target bill from the bill image by using the target detection model; Determine the bill center coordinates of the bill area and the identification center coordinates of the identification area, and determine the offset angle of the target bill in the bill image according to the bill center coordinates and the identification center coordinates; Determining target adjustment information according to the offset angle and a preset reference angle, and adjusting the bill image according to the target adjustment information to obtain a target image; Performing OCR recognition processing on the target image to obtain the bill information of the target bill; The step of determining the bill region of the target bill and the identification region of the reference identification of the target bill from the bill image by using the target detection model includes: Determining a plurality of bill candidate regions and a plurality of identification candidate regions from the bill image by using the object detection model; When the intersection-over-union ratio of the bill candidate region and the identification candidate region is 1, the corresponding bill candidate region is determined as the bill region of the target bill, and the corresponding identification candidate region is determined as the identification region of the reference identification; The offset angle of the target bill in the bill image is determined according to the bill center coordinates and the identification center coordinates, and is obtained by the following formula: ； in, is the offset angle, are the center coordinates of the marker, are the center coordinates of the bill; The determining target adjustment information according to the offset angle and a preset reference angle includes: Determine a target angle difference between the offset angle and the reference angle, and determine a target rotation angle and a target rotation direction according to the target angle difference; Determining the target rotation angle and the target rotation direction as the target adjustment information; The number of the reference angles is at least two, and the determining of the angle difference between the offset angle and the reference angle, and determining the target rotation angle and the target rotation direction according to the angle difference, comprises: Determine a plurality of candidate angle differences according to the offset angle and a plurality of the reference angles; Determine the candidate angle difference with the smallest absolute value as the target angle difference; When the target angle difference is a positive number, the target rotation direction is determined to be clockwise, and when the target angle difference is a negative number, the target rotation direction is determined to be counterclockwise.

2. A method for bill recognition based on target detection according to claim 1, characterized in that: Before determining a plurality of bill candidate regions and a plurality of identification candidate regions from the bill image by the object detection model, the method further includes: Determining the bill type of the target bill from the bill image; The reference identifier of the target document is determined according to the document type.

3. The method for bill recognition based on target detection according to claim 1, characterized in that: The target detection model is trained by the following method: Acquire a plurality of sample bill images, wherein the sample bill images are pre-marked with marking information, wherein the marking information includes a marked bill area and a marked identification area; A plurality of sample bill images are input into the target detection model, and the target detection model is trained until convergence.

4. A bill recognition device based on target detection, characterized in that: include: An image acquisition unit, used to acquire a bill image to be identified, and input the bill image into a pre-trained target detection model; A detection unit, configured to determine a bill region of a target bill and a marking region of a reference marking of the target bill from the bill image by using the target detection model; A positioning unit, used to determine the bill center coordinates of the bill area and the identification center coordinates of the identification area, and determine the offset angle of the target bill in the bill image according to the bill center coordinates and the identification center coordinates; An image adjustment unit, configured to determine target adjustment information according to the offset angle and a preset reference angle, and adjust the bill image according to the target adjustment information to obtain a target image; A recognition unit, used for performing OCR recognition processing on the target image to obtain the bill information of the target bill; Wherein, determining the bill area of ​​the target bill and the identification area of ​​the reference identification of the target bill from the bill image by using the target detection model includes: Determining a plurality of bill candidate regions and a plurality of identification candidate regions from the bill image by using the object detection model; When the intersection-over-union ratio of the bill candidate region and the identification candidate region is 1, the corresponding bill candidate region is determined as the bill region of the target bill, and the corresponding identification candidate region is determined as the identification region of the reference identification; The offset angle of the target bill in the bill image is determined according to the bill center coordinates and the identification center coordinates, and is obtained by the following formula: ； in, is the offset angle, are the center coordinates of the marker, are the center coordinates of the bill; The determining target adjustment information according to the offset angle and a preset reference angle includes: Determine a target angle difference between the offset angle and the reference angle, and determine a target rotation angle and a target rotation direction according to the target angle difference; Determining the target rotation angle and the target rotation direction as the target adjustment information; The number of the reference angles is at least two, and the determining of the angle difference between the offset angle and the reference angle, and determining the target rotation angle and the target rotation direction according to the angle difference, comprises: Determine a plurality of candidate angle differences according to the offset angle and a plurality of the reference angles; Determine the candidate angle difference with the smallest absolute value as the target angle difference; When the target angle difference is a positive number, the target rotation direction is determined to be clockwise, and when the target angle difference is a negative number, the target rotation direction is determined to be counterclockwise.

5. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the target detection-based bill recognition method as described in any one of claims 1 to 3 when executing the computer program.

6. A computer-readable storage medium storing a computer program, characterized in that: The computer program is used to execute the target detection-based bill recognition method as described in any one of claims 1 to 3.

Images