A mobile driver's license recognition method and system based on deep learning

Abstract

The present invention discloses a mobile terminal driver's license recognition method and system based on deep learning. The method comprises: constructing a driver's license segmentation detection model, an angle detection model and a driver's license text detection model based on a Pytorch deep learning model; converting the trained driver's license segmentation detection model, the angle detection model and the driver's license text detection model into ONNX format models, and saving the corresponding ONNX format models in a mobile terminal; recognizing a driver's license image by the mobile terminal, and inputting the image into the driver's license segmentation angle detection model and the angle detection model, recognizing the driver's license text area by the driver's license segmentation detection model and the angle detection model, removing the background, and then flipping the driver's license text area to a standard angle by the driver's license segmentation detection model and the angle detection model; inputting the image information processed by the driver's license segmentation detection model and the angle detection model into the driver's license text detection model to recognize the driver's license text information.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a mobile terminal driver's license recognition method and system based on deep learning Background Art

[0002] Driver's license information is particularly important in financial sectors such as auto insurance and bank loans. The method of entering driver's license information has evolved from manual entry to digital image algorithms using optical character recognition (OCR). The recent rise of deep learning technology has significantly boosted the field of computer vision, with significant contributions in areas such as object detection, image segmentation, face detection and recognition, and optical character recognition (OCR). However, traditional driver's license recognition methods based on digital image processing algorithms require clear document images and a clean background, which limits their robustness and versatility, impacting user experience and data entry efficiency. Deep learning image algorithms have completely solved this problem. Deep learning offers strong robustness and versatility, unrestricted by complex backgrounds and image quality.

[0003] With the rise of mobile internet, more and more insurance and banking services are being conducted through mobile devices. However, current solutions mostly upload ID images captured by mobile devices to a server via the internet, where an OCR recognition service is deployed to complete the recognition task. This solution has the following drawbacks: it increases the cost and pressure of server deployment; it is significantly affected by the network, especially since mobile phone images are relatively large and network transmission is time-consuming; it does not adequately protect customer privacy; and its recognition speed is slow, including the time it takes to transmit the image over the network.

[0004] Currently, OCR technology primarily utilizes a process consisting of text detection and recognition. Text detection uses a text detection algorithm to identify all text areas in an image, then uses a recognition model to identify the text within those areas. Finally, using prior knowledge such as semantics and relative positional relationships, key information fields are extracted from complex text lists. This process requires extensive sample testing and code debugging, resulting in extremely low efficiency. Summary of the Invention

[0005] One of the inventive purposes of the present invention is to provide a mobile driver's license recognition method based on deep learning. The method and system convert the trained deep learning model into a lightweight model through format conversion and deploy it on the mobile terminal, so that the mobile terminal can realize automatic detection, so that the driver's license detection will not be affected by the network environment, thereby improving the efficiency and stability of driver's license recognition, while also protecting the driver's privacy and reducing the risk of leakage due to communication.

[0006] Another object of the present invention is to provide a mobile driver's license recognition method based on deep learning. The method and system set up multiple types of detection models, and construct key-value pairs through simple text areas and text categories, and identify text information by searching for key values ​​through key names. Therefore, the entire process is simpler and does not require a large amount of subsequent processing work.

[0007] Another invention object of the present invention is to provide a mobile driver's license recognition method based on deep learning. The method and system set a segmentation detection model and an angle classification model, and save the segmentation detection model and the angle classification model in a lightweight manner on the mobile terminal. The segmentation detection model and the angle classification model can crop the driver's license into a driver's license area image without background, and judge the angle of the driver's license area image, so that the driver's license area image without background can be flipped into an image with the front side facing up, thereby improving the accuracy of driver's license text recognition.

[0008] In order to achieve at least one of the above-mentioned objects, the present invention further provides a method for mobile driver's license recognition based on deep learning, the method comprising:

[0009] Build a driver's license segmentation detection model, angle detection model, and driver's license text detection model based on the Pytorch deep learning model;

[0010] Convert the trained driver's license segmentation detection model, angle detection model, and driver's license text detection model into ONNX format models, and save the corresponding ONNX format models on the mobile terminal;

[0011] Recognize the driver's license image through a mobile terminal, and input the image into the driver's license segmentation angle detection model and the angle detection model, and use the driver's license segmentation detection model and the angle detection model to recognize the driver's license text area, remove the background, and then flip it to a standard angle;

[0012] The image information processed by the driver's license segmentation detection model and the angle detection model is input into the driver's license text detection model to recognize the driver's license text information.

[0013] According to one of the preferred embodiments of the present invention, the driver's license segmentation detection model is trained using a deep learning fully convolutional network DBNet, and outputs a binary prediction probability map based on the driver's license area and the background area. The gradient descent method is used to calculate the minimum model of the binary prediction probability map and the label cross entropy loss function and save it.

[0014] According to another preferred embodiment of the present invention, the recognition method includes: using a trained segmentation detection model to recognize the input driver's license image information, the trained segmentation detection model will be input into the driver's license image to predict and obtain a driver's license area map, and cropping according to the border of the predicted area map to obtain a cropped driver's license area map.

[0015] According to another preferred embodiment of the present invention, after obtaining the cropped driver's license area image, the cropped driver's license area image is further input into an angle detection model, and the angle detection model detects the angle information of the cropped image relative to the camera illumination surface, and flips the cropped driver's license area image into an image of standard angle according to the angle information for subsequent text detection.

[0016] According to another preferred embodiment of the present invention, after completing the angle adjustment of the driver's license area map, the driver's license area map with the adjusted angle is further input into a text detection model, wherein the text detection model includes a multi-type text detection model and a text recognition model, and the multi-type text detection model is obtained by training the text detection algorithm DBNet through the Pytorch model. The multi-type text detection model uses the four vertices of the square structure of the field text to be obtained as the category label of the multi-type text detection model, and the multi-type text detection model predicts and outputs a predicted binary map of the text area and the field type according to the type of annotation.

[0017] According to another preferred embodiment of the present invention, the method for identifying text information on a driver's license includes: inputting the cropped text area into a text recognition model, the text recognition model includes: scaling the identified text area as a slice, and normalizing the scaled slice, using a trained CRNN+CTC model to identify the normalized text area, outputting the confidence of the corresponding Chinese character in the CRNN+CTC model, and outputting the Chinese character with the highest confidence as the recognition result.

[0018] According to another preferred embodiment of the present invention, the identification method includes: performing structured storage based on the text recognition results of the identified field type and field text area, wherein the corresponding field type of the identified field is used as the key name of the key-value pair, and the text recognition result of the same field text area is used as the key value to generate a key-value pair for storage.

[0019] According to another preferred embodiment of the present invention, the field types include: name, gender, ethnicity, address and citizen identification number, driving license type, and validity period, and the above field types are obtained according to the labels of the multi-type text detection.

[0020] In order to achieve at least one of the above-mentioned purposes of the invention, the present invention further provides a mobile terminal driver's license recognition system based on deep learning, which executes the above-mentioned mobile terminal driver's license recognition method based on deep learning.

[0021] The present invention further provides a computer-readable storage medium storing a computer program, which can be executed by a processor to implement the mobile driver's license recognition method based on deep learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 Shown is a flow chart of a mobile driver's license recognition method based on deep learning in the present invention.

[0023] Figure 2 What is shown is the overall flow chart of the driver's license recognition method in the present invention. DETAILED DESCRIPTION

[0024] The following description is intended to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are for illustrative purposes only, and those skilled in the art will readily appreciate other obvious variations. The basic principles of the present invention defined in the following description may be applied to other embodiments, variations, improvements, equivalents, and other technical solutions that do not depart from the spirit and scope of the present invention.

[0025] It is to be understood that the term "one" should be understood as "at least one" or "one or more", that is, in one embodiment, the number of an element may be one, while in another embodiment, the number of the elements may be multiple, and the term "one" should not be understood as a limitation on the quantity.

[0026] Please combine Figure 1-Figure 2 The present invention discloses a mobile terminal driver's license recognition method and system based on deep learning, wherein the method is built based on different deep learning models, and multiple trained models are converted into lightweight models and input into the mobile terminal. The driver's license image is obtained by the mobile terminal and then recognized. Therefore, the present invention can realize rapid recognition of the driver's license under no network conditions, thereby avoiding the terminal from being affected by the communication environment and reducing the leakage problem caused by data communication.

[0027] It is worth mentioning that the present invention is based on the Pytorch framework training including the driver's license segmentation detection model, the angle detection model and the driver's license text detection model, and converts the trained driver's license segmentation detection model, the angle detection model and the driver's license text detection model into an ONNX format model, and saves the ONNX format model in a mobile terminal. The mobile terminal can quantize the ONNX format model, convert the parameters in the ONNX format model from float type to int8 type in the mobile terminal, and deploy OpenCV on the mobile terminal to load the ONNX model.

[0028] The method for generating a driver's license segmentation detection model described in the present invention includes: obtaining driver's license image information using a camera, including but not limited to a mobile terminal, constructing a driver's license image sample, and inputting the driver's license image sample into a deep learning fully convolutional network (DBNet). The deep learning fully convolutional network (DBNet) is trained based on a Pytorch framework whose backbone network is MobileNet V3. After training, a 640*640 pixel probability map is output. The output image after training is a binary probability map. If the current pixel is predicted to be in the driver's license image area, the current pixel value is 1; if the current pixel value is 0, it indicates that the current pixel is not in the driver's license image area. Therefore, based on the output predicted binary probability map, the predicted driver's license image area can be cropped to remove the background image area to form a clean driver's license image area. In the present invention, the deep learning fully convolutional network (DBNet) is used to minimize the predicted binary probability map and the label cross entropy loss function using a gradient descent algorithm. The model is obtained and saved to obtain the original driver's license segmentation detection model. The driver's license segmentation detection model can effectively segment the valid area of ​​the driver's license and the invalid area of ​​the background.

[0029] It is worth mentioning that the basic construction methods of the above-mentioned driver's license segmentation detection model, angle detection model and driver's license text detection model of the present invention mainly include: normalizing training image samples; constructing a deep learning model calculation graph; initializing training parameters; calculating hidden layer and output layer vectors; updating weights and biases; calculating the total unit error; judging whether the error threshold or the maximum number of iterations has been reached, and if so, solidifying and saving the model structure and parameters; if not, continuing to calculate the hidden layer and output layer vectors; updating weights and biases. The basic construction methods of the above-mentioned driver's license segmentation detection model, angle detection model and driver's license text detection model all use a gradient descent algorithm to minimize the loss function of the model training and obtain the optimal model. It should be noted that the basic training method of the above-mentioned model is a prior art, and the weights and training parameters of different models are different, which will not be described in detail in the present invention.

[0030] After obtaining a clean driver's license area image outline through the driver's license segmentation detection model, the driver's license area image is cropped and input into the angle detection model to determine the angular difference between the current driver's license area and the camera acquisition plane. Since the driver's license area is a normal rectangular image with a fixed length and width, the angle difference caused by the difference in image acquisition angle can be calculated by calculating the aspect ratio. For example, if the aspect ratio is 0 or infinite, it can be concluded that the current driver's license area image is perpendicular to the camera acquisition plane and needs to be further flipped. Of course, in some embodiments, if the aspect ratio is 0 or infinite, the driver's license segmentation detection model cannot detect the corresponding driver's license area, and a detection error message will be output. If the driver's license area image input by the angle detection model is parallel to the camera acquisition plane, the calculated driver's license area aspect ratio is a preset fixed value, which is a standard 0-degree driver's license area image. In some embodiments, when the angle detection model obtains a non-standard 0-degree driving area image, it is necessary to rotate the driver's license area image along a preset rotation axis to obtain an image with a preset driver's license area aspect ratio of a fixed value, and use the rotated image as the input image for the text detection model. If the detected driving area image is 180 degrees, it is necessary to flip the image to a 0-degree image along a preset rotation axis. It should be noted that the rotation axis can be set to multiple, with different rotation axes extending in different directions, and the rotation axis will be set according to the camera's detection plane. In one preferred embodiment of the present invention, the above-mentioned angle detection model can obtain driver's licenses at different angles for detection, wherein four fixed points on the driver's license can be collected and the length relationship of the line connecting the four points can be calculated based on trigonometric functions to obtain the corresponding driver's license angle. In the present invention, it is preferred to calculate the trigonometric function of the four corner points of the square outline of the driver's license area. It is further used to correct the angle of the driver's license area image. It should be noted that the above-mentioned angle calculation method is prior art and will not be described in detail in the present invention.

[0031] The image obtained after the angle of the driver's license area is corrected is input into the driver's license text detection model. It should be noted that the driver's license text detection model includes a multi-classification text detection model and a text recognition model, wherein the multi-classification text detection model will be based on the deep learning full convolutional network DBNet, after identifying the four vertices of the required field text area and marking them, and using the marked label as the label for model training, the type of the annotation is the type of the field on the driver's license, and the annotation may include but is not limited to field types such as name, gender, nationality, address and citizen identification number, driving type, validity period, etc. The present invention uses Pytorch as the framework for the model training, and adopts the gradient descent algorithm to obtain the optimal model, the deep learning full convolutional network DBNet outputs the predicted field type including a binary map of the text area, and outputs the field type with the highest probability as the current detected field type. The above-mentioned multi-classification text detection model is converted into ONNX format and stored in the mobile terminal

[0032] After completing the detection output of the currently detected field type, a text recognition model is further used to identify the text area corresponding to the field type. In the present invention, the text area after the field type is identified is scaled as a text slice, scaled to a 32-height pixel image, and the scaled image is normalized. The text recognition model is converted into ONNX format and stored in the mobile terminal. It is worth mentioning that the present invention uses a CRNN+CTC model to construct the text recognition model, and the CRNN+CTC model is trained based on Pytorch. The Pytorch training method of the CRNN+CTC model includes: recognizing a large amount of printed text for recognizing 6869 commonly used words, punctuation marks and English characters contained in the dictionary library. The slices of the text area on the driver's license recognized by the camera are divided into single words. After being divided into single words, the height of the single word image is reduced to 32 pixels. The input single word image with a height of 32 pixels is written into the text, and each word is used as a label for training. The softmax activation function is used to calculate the confidence of each word label and 6869 common words, punctuation marks and English characters. The common words, punctuation marks and English characters with the highest confidence are calculated as the output words of the final text recognition model.

[0033] Finally, based on the text areas in different fields and the text recognition results, the corresponding field names and recognition results are used to extract structured information from the driver's license, including name, gender, ethnicity, address, citizen ID number, driving license type, and validity period. This information is then combined in a key-value format. For example, if the multi-classification text detection model detects the name field type, and the text recognition model detects that the text area in the name field type contains the character "Zhang San", {name, Zhang San} will be stored as the key-value, facilitating subsequent difference queries.

[0034] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part, and / or installed from a removable medium. When the computer program is executed by the central processing unit (CPU), the above-mentioned functions defined in the method of the present application are executed. It should be noted that the computer-readable medium mentioned above in the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of an electrical, magnetic, optical, electromagnetic, infrared segment, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more wire segments, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, an electromagnetic signal, an optical signal, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transfer a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, electrical wire, optical fiber cable, RF, etc., or any suitable combination thereof.

[0035] The flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0036] Those skilled in the art should understand that the embodiments of the present invention described above and shown in the accompanying drawings are only examples and do not limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and explained in the embodiments. Without departing from the principles, the implementation methods of the present invention may be subject to any deformation or modification.

Claims

1. A mobile driver's license recognition method based on deep learning, characterized in that: The method comprises: Build a driver's license segmentation detection model, a driver's license angle detection model, and a driver's license text detection model based on the Pytorch deep learning model; Convert the trained driver's license segmentation detection model, driver's license angle detection model, and driver's license text detection model into ONNX format models, and save the corresponding ONNX format models on the mobile terminal; Recognize the driver's license image through a mobile terminal, and input the image into the driver's license segmentation detection model and the driver's license angle detection model, and use the driver's license segmentation detection model and the driver's license angle detection model to recognize the driver's license text area, remove the background, and then flip it to a standard angle; Inputting the image information processed by the driver's license segmentation detection model and the driver's license angle detection model into the driver's license text detection model to recognize the driver's license text information; The driver's license text detection model includes a multi-type text detection model and a text recognition model. The multi-type text detection model is obtained by training the text detection algorithm DBNet through the Pytorch model. The multi-type text detection model forms a square structure with four vertices marked as category labels for the multi-type text detection model after the field text to be acquired is formed. The multi-type text detection model predicts and outputs a predicted binary image of the text area and the field type according to the marked type; The cropped text area is input into the text recognition model, which includes: scaling the recognized text area as a slice, normalizing the scaled slice, recognizing the normalized text area using a trained CRNN+CTC model, outputting the confidence of the corresponding Chinese character in the CRNN+CTC model, and outputting the Chinese character with the highest confidence as the recognition result.

2. A mobile driver's license recognition method based on deep learning according to claim 1, characterized in that: The driver's license segmentation detection model is trained using a deep learning fully convolutional network (DBNet) and outputs a binary prediction probability map based on the driver's license area and background area. The gradient descent method is used to calculate the minimum model of the binary prediction probability map and the label cross entropy loss function and save it.

3. A mobile driver's license recognition method based on deep learning according to claim 2, characterized in that: The driver's license recognition method includes: using a trained driver's license segmentation detection model to recognize input driver's license image information, the trained segmentation detection model predicts the input driver's license image to obtain a driver's license area map, and cropping the predicted driver's license area map according to the border of the predicted driver's license area map to obtain a cropped driver's license area map.

4. A mobile driver's license recognition method based on deep learning according to claim 3, characterized in that: After obtaining the cropped driver's license area image, the cropped driver's license area image is further input into the driver's license angle detection model. The driver's license angle detection model detects the angle information of the cropped image relative to the camera illumination surface, and rotates the cropped driver's license area image into an image of standard angle according to the angle information for subsequent text detection.

5. The method for mobile driver's license recognition based on deep learning according to claim 4 is characterized in that: After the angle adjustment of the driver's license area map is completed, the driver's license area map with the adjusted angle is further input into the driver's license text detection model.

6. A mobile driver's license recognition method based on deep learning according to claim 5, characterized in that: The method for identifying text information in the driver's license area includes: structured storage based on the identified field type and the text recognition results of the field text area, wherein the corresponding field type of the identified field is used as the key name of the key-value pair, and the text recognition result of the same field text area is used as the key value to generate a key-value pair for storage.

7. A mobile driver's license recognition method based on deep learning according to claim 5, characterized in that: The field types include: name, gender, ethnicity, address and citizen identification number, driving license type, and validity period. The above field types are obtained according to the labels of the multi-type text detection.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement a mobile driver's license recognition method based on deep learning as described in any one of claims 1 to 7.

Images