An identity card recognition method and device based on deep learning

Abstract

The application discloses an identity card recognition method based on deep learning, and belongs to the technical field of computer vision, and solves the technical problem of low recognition rate of the existing identity card recognition method, and the method comprises the following steps: S1, acquiring an identity card image, and performing inclination correction processing on the identity card image to obtain a pretreated image; S2, constructing a region detection model based on yolov3, and detecting each character region in the pretreated image through the region detection model; S3, constructing a feature extractor based on a convolutional neural network, which is used for extracting image features of each character region, and extracting a feature vector sequence from the image features; S4, constructing a recurrent neural network to process the feature vector sequence, and obtaining a final character recognition result. The application further discloses an identity card recognition device based on deep learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, more particularly, it relates to an identity card recognition method and device based on deep learning. BACKGROUND

[0002] Identity card recognition refers to using computer technology to read out the text or characters in the identity card image and converting them into a format that can be understood by humans. The computer can automatically recognize the information on the identity card and input it into the system, which not only saves the cost of manual input into the system, but also greatly improves the efficiency of various business operations.

[0003] At present, identity card recognition uses statistical mode to realize character recognition, and the main steps are: preprocessing the original image; performing binarization operation on the preprocessed image; performing layout analysis on the binarized image, and then performing line segmentation and word segmentation on the character regions therein; finally, realizing single word recognition and performing some post-processing operations.

[0004] The biggest disadvantage of the above method is that the processing flow is too long, and the recognition system is divided into too many links. Any imperfect module will produce errors, and the accumulation of errors will cause the recognition rate to drop significantly. In addition, manual intervention is required in multiple links, and the method parameters are set according to the scene, which is difficult to achieve end-to-end training, increases the use cost, and the recognition method has poor flexibility. For complex images taken in natural scenes, it is basically impossible to process, there is no room for modification and improvement, and the usability is low. SUMMARY

[0005] The technical problem to be solved by the present application is to solve the above-mentioned deficiencies of the prior art. The first object of the present application is to provide an identity card recognition method based on deep learning which can improve recognition accuracy.

[0006] The second object of the present application is to provide an identity card recognition device based on deep learning which can improve recognition accuracy.

[0007] In order to achieve the above-mentioned first object, the present application provides an identity card recognition method based on deep learning, comprising:

[0008] Step S1. Obtain an identity card image, and perform tilt correction processing on the identity card image to obtain a preprocessed image;

[0009] Step S2. Construct a region detection model based on yolov3, and detect each character region in the preprocessed image through the region detection model;

[0010] Step S3. Construct a feature extractor based on a convolutional neural network to extract image features of each character region, and extract a feature vector sequence from the image features.

[0011] Step S4. Construct a recurrent neural network to process the sequence of feature vectors to obtain a final character recognition result.

[0012] As a further improvement, step S1 comprises:

[0013] Step S11. Obtain a gray-scale image of the ID card image;

[0014] Step S12. Perform Gaussian blur processing on the gray-scale image obtained in step S11;

[0015] Step S13. Perform Canny edge detection on the gray-scale image obtained in step S12 to obtain a Canny edge detection image;

[0016] Step S14. Perform Hough transform on the Canny edge detection image obtained in step S13 to obtain an array of all straight lines in polar coordinates form in the edge detection image, and then calculate the inclination of the image according to the obtained polar coordinate array;

[0017] Step S15. Adjust the ID card image according to the inclination calculated in step S14 to obtain a preprocessed image after inclination correction.

[0018] Further, in step S2, constructing a region detection model comprises:

[0019] Step S21. Design a CBL unit, which is composed of a convolution layer, a BN layer and a Leaky Relu layer connected in sequence, and the Leaky Relu layer uses Leaky Relu as the activation function;

[0020] Step S22. Design a residual unit, which contains two CBL units inside, and the two CBL units are connected in sequence. The input of the residual unit will be processed by the two CBL units, and the results obtained by the processing will be added to the input of the residual unit to obtain the output of the residual unit;

[0021] Step S23. Design a residual module, which contains two CBL units and three residual units. The two CBL units are connected in sequence, and then connected with the three residual units in sequence;

[0022] Step S24. Design the main network of the model by connecting a plurality of residual modules to form the main network of the model;

[0023] Step S25. Design a multi-scale detection unit for detecting character regions at multiple scales.

[0024] Further, in the step S24, a backbone network of the model is formed by connecting three residual modules in series, and the convolution kernel parameter configurations of the three residual modules are as follows:

[0025] The convolution kernel parameters used by the two CBL units of the first residual module are both 3x3x16; the convolution kernel parameter configurations of the three residual units are all the same, and the convolution kernel parameter configurations of the two CBL units in the residual unit are 3x3x8 and 1x1x16 respectively;

[0026] The convolution kernel parameters used by the two CBL units of the second residual module are both 3x3x32; the convolution kernel parameter configurations of the three residual units are all the same, and the convolution kernel parameter configurations of the two CBL units in the residual unit are 3x3x16 and 1x1x32 respectively;

[0027] The convolution kernel parameters used by the two CBL units of the third residual module are both 3x3x64; the convolution kernel parameter configurations of the three residual units are all the same, and the convolution kernel parameter configurations of the two CBL units in the residual unit are 3x3x32 and 1x1x64 respectively.

[0028] Further, in the step S25, the process of designing the multi-scale detection unit is as follows:

[0029] The large target scale unit contains three CBL units and one full convolution layer, and the convolution kernel parameter configurations of the three CBL units are 1x1x64, 3x3x128 and 1x1x64 respectively. The parameter configuration of the full convolution layer is 1x1x21. The unit takes the output of the third residual module of the backbone network as input, outputs 8x8x[3x(2+5)] results through the full convolution layer, represents setting three bounding boxes at each point on the 8x8 feature map, and then outputs the scores of two classes of each bounding box, the center point coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box and the bounding box confidence;

[0030] The medium target scale unit contains three CBL units and one full convolution layer, and the convolution kernel parameter configurations of the three CBL units are 1x1x32, 3x3x64 and 1x1x32 respectively. The parameter configuration of the full convolution layer is 1x1x21. The unit takes the output of the second residual module of the backbone network as input, outputs 16x16x[3x(2+5)] results through the full convolution layer, represents setting three bounding boxes at each point on the 16x16 feature map, and then outputs the scores of two classes of each bounding box, the center point coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box and the bounding box confidence;

[0031] The small target scale unit is designed, the small target scale unit comprises three CBL units and one full convolution layer, the convolution kernel parameter configurations of the three CBL units are 1*1*16, 3*3*32 and 1*1*16 respectively, and the parameter configuration of the full convolution layer is 1*1*21; the unit takes the output of the first residual module of the backbone network as input, outputs 32*32*[3*(2+5)] results through the full convolution layer, represents that three bounding boxes are set at each point on the 32*32 feature map, and then outputs the scores of two classes of each bounding box, the center coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box and the bounding box confidence.

[0032] Further, the step S26 is further included, after the region detection model is constructed, the region detection model needs to be trained using the set identity card image data set, and the specific process is as follows:

[0033] A large number of virtual identity card images are collected;

[0034] The identity card image data set is obtained by performing a data enhancement operation on the virtual identity card image, and the enhancement operation comprises at least one of image flipping, image smoothing and adding Gaussian noise;

[0035] An optimization objective function for training is designed, and a simplified model of the optimization objective function is as follows:

[0036] L = L (x,y) + L size + L conf + L classes

[0037] Wherein, L (x,y) is a bounding box center point error, L size is a bounding box size error, L conf is a bounding box confidence error, and L classes is a class error.

[0038] Adam is used as an optimizer, the initial learning rate is set to 1e -4 , and the number of training iterations is set to 50,000 times.

[0039] The training process is as follows:

[0040] Step S261. 128 images are taken from the identity card image data set each time to input the model, and the loss value of the optimization objective function is calculated according to the output of the region detection model and the true value;

[0041] Step S262. The weight parameters of the region detection model are updated using the Adam gradient descent algorithm, and the number of iterations is increased by 1.

[0042] Step S263. Repeat step S261, step S262, until the number of iterations reaches 50000 times, the model training is finished.

[0043] Further, in step S3, the process of constructing the feature extractor is as follows:

[0044] Step S31. Obtain the gray image of the character region image, and adjust the size of the character region image to 16x160. A 16x160x1 input layer is constructed to receive the character region gray image;

[0045] Step S32. Construct a convolutional pooling unit, which includes 1 convolutional layer, 1 BN layer, 1 Relu layer and 1 max pooling layer connected in sequence;

[0046] Step S33. Construct the feature extractor by connecting 1 input layer and 4 convolutional pooling units in sequence; the convolutional pooling parameter configuration of each convolutional pooling unit is as follows: the first convolutional pooling unit uses 32 3x3 convolutional kernels, and the maximum pooling window is set to 2x2; the second convolutional pooling unit uses 64 3x3 convolutional kernels, and the maximum pooling window is set to 2x2; the third convolutional pooling unit uses 128 3x3 convolutional kernels, and the maximum pooling window is set to 2x1; the fourth convolutional pooling unit uses 256 3x3 convolutional kernels, and the maximum pooling window is set to 2x1.

[0047] Further, the height and width of the feature map of the character region gray image after passing through the first convolutional pooling unit become 1 / 2 of the original image; the height and width of the feature map after passing through the second convolutional pooling unit become 1 / 4 of the original image; the height of the feature map after passing through the third convolutional pooling unit becomes 1 / 8 of the original image, and the width becomes 1 / 4 of the original image; the height of the feature map after passing through the fourth convolutional pooling unit becomes 1 / 16 of the original image, and the width becomes 1 / 4 of the original image; finally, 256 1x40 feature maps are extracted through the feature extractor;

[0048] The feature vector sequence is extracted from the 256 1x40 feature maps. Specifically, 256 pixel points at each position from left to right on the feature map are taken as a feature vector to obtain 40 256-dimensional feature vector sequences.

[0049] Further, in step S4, the process of constructing the recurrent neural network is as follows:

[0050] Step S41. Construct a 40-time-step LSTM network to process the feature vector sequence containing 40 feature vectors, with each time step corresponding to a feature vector. The LSTM network includes 2 intermediate hidden layers, each containing 256 neurons, and all performing full connection operations with the previous layer to further extract features of the feature sequence;

[0051] Step S42. The output layer of the LSTM network is designed, and the output layer mainly functions to output the probability of each character predicted by the LSTM at each time step. The output layer of the LSTM network contains 1037 neurons, and the LSTM outputs 1037 probability values at each time step, representing the probability of classification as a certain character. The 1037 neurons correspond to 10 types of numbers, 26 types of English characters, 1000 types of Chinese characters and 1 empty character representing the position corresponding to the time step on the ID image, and the empty character is represented by "-".

[0052] Step S43. For the 1037 output probabilities of each time step obtained in step S42, the character category with the maximum probability is taken as the character category output at the time step, and 40 characters are output for 40 time steps. A de-duplication operation is performed on the output characters. Specifically, when the LSTM outputs a sequence, a "-" is inserted between repeated characters in the text label. Continuous identical characters without "-" separation are merged, and identical characters with "-" separation are not merged. Finally, if there is a blank character, it is deleted to obtain the final character recognition result.

[0053] To achieve the above-mentioned purpose two, the application provides an ID card recognition device based on deep learning, comprising a memory and a processor; the memory is used for storing a computer program; the processor is used for executing the computer program to realize the ID card recognition method based on deep learning as described above.

[0054] Advantages

[0055] Compared with the prior art, the application has the following advantages:

[0056] The application can automatically extract character features on the ID image, solve the problem of manual design and feature extraction in traditional character recognition technology, increase the robustness and generalization of the model by improving the network structure, realize accurate recognition of ID characters, reduce the use cost, and improve the efficiency of character recognition. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 The flowchart of the method of the application;

[0058] Figure 2 The schematic diagram of the CBL unit in the application;

[0059] Figure 3 The schematic diagram of the residual unit in the application;

[0060] Figure 4 The schematic diagram of the residual module in the application;

[0061] Figure 5 A schematic diagram of a convolutional pooling unit in the present application;

[0062] Figure 6 A flowchart of an identity card recognition method in the present application;

[0063] Figure 7 A schematic diagram of a system for implementing identity card recognition using the method of the present application. DETAILED DESCRIPTION

[0064] The present application will be further described below with reference to specific embodiments in the accompanying drawings.

[0065] Reference Figures 1-7 An identity card recognition method based on deep learning, comprising:

[0066] Step S1. Obtain an identity card image and perform tilt correction processing on the identity card image to obtain a preprocessed image;

[0067] Step S2. Construct a region detection model based on yolov3, and detect each character region in the preprocessed image through the region detection model;

[0068] Step S3. Construct a feature extractor based on a convolutional neural network to extract image features of each character region, and extract a feature vector sequence from the image features;

[0069] Step S4. Construct a recurrent neural network to process the feature vector sequence to obtain a final character recognition result.

[0070] The identity card image captured by the camera may be tilted, rotated, inverted, or the like. In order to reduce the influence of such situations on the subsequent identity card recognition result, the input identity card image is first subjected to tilt correction processing. Specifically, step S1 comprises:

[0071] Step S11. Obtain a gray image of the identity card image;

[0072] Step S12. Perform Gaussian blur processing on the gray image obtained in step S11;

[0073] Step S13. Perform Canny edge detection on the gray image obtained in step S12 to obtain a Canny edge detection image;

[0074] Step S14. Perform Hough transform on the Canny edge detection image obtained in step S13 to obtain an array of all straight lines in the edge detection image in polar coordinate form, and then calculate the tilt of the image according to the obtained polar coordinate array;

[0075] Step S15. Adjust the identity card image according to the inclination calculated in step S14 to obtain a preprocessed image after inclination correction.

[0076] Before recognizing the characters in the identity card image, the region containing the characters in the image needs to be detected, and then the character recognition operation is performed on each region. Compared with directly recognizing the characters in the whole image, this method can greatly improve the accuracy of character recognition. Therefore, a character region detection model is first constructed. The region detection model based on yolov3 is used in the present application. Since the yolov3 model is relatively large, it is too complex for the identity card character detection task. Therefore, the structure thereof is optimized and adjusted, and then a set identity card image data set is used for training. In step S2, the region detection model is constructed, including:

[0077] Step S21. A CBL unit is designed, which is composed of a convolution layer, a BN layer and a Leaky Relu layer connected in sequence. The Leaky Relu layer refers to using Leaky Relu as an activation function, as shown in FIG. 1. The unit is the smallest basic component of the whole model. Figure 2

[0078] Step S22. A residual unit is designed, which contains two CBL units connected in sequence. The input of the residual unit is processed by the two CBL units, and the result is added to the input of the residual unit to obtain the output of the residual unit, as shown in FIG. 2. Figure 3

[0079] Step S23. A residual module is designed, which contains two CBL units and three residual units connected in sequence, as shown in FIG. 3. Figure 4

[0080] Step S24. The backbone network of the model is designed by connecting a plurality of residual modules to form the backbone network of the model. In this embodiment, the backbone network of the model is formed by connecting three residual modules. The convolution kernel parameter configurations of the three residual modules are as follows:

[0081] The convolution kernel parameters of the two CBL units of the first residual module are both 3x3x16, that is, 16 3x3 convolution kernels. The convolution kernel parameter configurations of the three residual units are the same, and the convolution kernel parameter configurations of the two CBL units in the residual unit are 3x3x8 and 1x1x16, respectively.

[0082] ​​​The convolution kernel parameters used by the two CBL units of the second residual module are both 3x3x32, i.e., 32 3x3 convolution kernels; the convolution kernel parameter configurations of the three residual units are all the same, and the convolution kernel parameter configurations of the two CBL units in the residual unit are 3x3x16 and 1x1x32, respectively;

[0083] The convolution kernel parameters used by the two CBL units of the third residual module are both 3x3x64, i.e., 64 3x3 convolution kernels; the convolution kernel parameter configurations of the three residual units are all the same, and the convolution kernel parameter configurations of the two CBL units in the residual unit are 3x3x32 and 1x1x64, respectively;

[0084] Step S25. Since the character regions in the ID image are of different sizes, such as the size difference between the "ID number" character region and the "name" character region, it is necessary to detect the character regions at multiple scales, and a multi-scale detection unit is designed for detecting the character regions at multiple scales; the process of designing the multi-scale detection unit is as follows:

[0085] A large target scale unit is designed, which includes three CBL units and one full convolution layer. The convolution kernel parameter configurations of the three CBL units are 1x1x64, 3x3x128, and 1x1x64, respectively, and the parameter configuration of the full convolution layer is 1x1x21. The unit takes the output of the third residual module of the backbone network as input, outputs 8x8x[3x(2+5)] results through the full convolution layer, represents setting three bounding boxes at each point on the 8x8 feature map, and then outputs the scores of two classes, the center point coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box, and the bounding box confidence of each bounding box;

[0086] A medium target scale unit is designed, which includes three CBL units and one full convolution layer. The convolution kernel parameter configurations of the three CBL units are 1x1x32, 3x3x64, and 1x1x32, respectively, and the parameter configuration of the full convolution layer is 1x1x21. The unit takes the output of the second residual module of the backbone network as input, outputs 16x16x[3x(2+5)] results through the full convolution layer, represents setting three bounding boxes at each point on the 16x16 feature map, and then outputs the scores of two classes, the center point coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box, and the bounding box confidence of each bounding box;

[0087] The small target scale unit is designed, the small target scale unit includes three CBL units and one full convolution layer, the convolution kernel parameter configurations of the three CBL units are 1*1*16, 3*3*32 and 1*1*16 respectively, and the parameter configuration of the full convolution layer is 1*1*21; the unit takes the output of the first residual module of the backbone network as input, outputs 32*32*[3*(2+5)] results through the full convolution layer, represents that three bounding boxes are set at each point on the 32*32 feature map, and then outputs the scores of two classes of each bounding box, the center coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box and the bounding box confidence.

[0088] In step S2, step S26 is further included, after the region detection model is constructed, the region detection model needs to be trained using the set identity card image data set, and the specific process is as follows:

[0089] A large number of virtual identity card images are collected, since the identity card image involves security and privacy information, therefore, the real identity card image cannot be included in the data set, and since the characters on the identity card are all numbers or characters, therefore, the content on the identity card image can be modified using PS technology, for example, the identity card number is modified to a random number of the same number of digits, so as to generate as many identity card images as possible;

[0090] Since the size of the data set directly affects the quality of the model training, the virtual identity card image needs to be subjected to a data enhancement operation to obtain an identity card image data set, and the enhancement operation includes at least one of image flipping, image smoothing and adding Gaussian noise;

[0091] An optimization objective function for training is designed, and a simplified model of the optimization objective function is as follows:

[0092] L = L (x,y) + L size + L conf + L classes

[0093] Wherein, L (x,y) is the center point error of the bounding box, L size is the size error of the bounding box, L conf is the confidence error of the bounding box, and L classes is the class error.

[0094] Adam is used as the optimizer, the initial learning rate is set to 1e -4 , and the number of training iterations is set to 50000 times.

[0095] The training process is as follows:

[0096] Step S261. 128 images are input into the model from the identity card image dataset each time, and the loss value of the optimization objective function is calculated according to the output of the region detection model and the true value;

[0097] Step S262. The weight parameters of the region detection model are updated using the Adam gradient descent algorithm, and the iteration number is incremented by 1;

[0098] Step S263. Steps S261 and S262 are repeated until the iteration number reaches 50,000, and the model training is completed.

[0099] In step S3, the process of constructing the feature extractor is as follows:

[0100] Step S31. The gray image of the character region image is obtained, and the size of the character region image is adjusted to 16x160. A 16x160x1 input layer is constructed to receive the character region gray image. The reason for adjusting the image size to 16x160 is that most character region images are small in height and long in width. Therefore, a reasonable processing window can be set in the subsequent convolution and pooling process to ensure that the information in the width direction of the image is not lost.

[0101] Step S32. A convolutional pooling unit is constructed, which includes a convolutional layer, a BN layer, a Relu layer, and a maximum pooling layer connected in series, as shown in Figure 5 .

[0102] Step S33. A feature extractor is constructed by connecting an input layer and four convolutional pooling units in series. The convolutional pooling parameters of each convolutional pooling unit are as follows: the first convolutional pooling unit uses 32 3x3 convolutional kernels, and the maximum pooling window is set to 2x2; the second convolutional pooling unit uses 64 3x3 convolutional kernels, and the maximum pooling window is set to 2x2; the third convolutional pooling unit uses 128 3x3 convolutional kernels, and the maximum pooling window is set to 2x1; and the fourth convolutional pooling unit uses 256 3x3 convolutional kernels, and the maximum pooling window is set to 2x1.

[0103] After the character region gray image passes through the first convolutional pooling unit, the height and width of the feature map become 1 / 2 of the original image; after passing through the second convolutional pooling unit, the height and width of the feature map become 1 / 4 of the original image; after passing through the third convolutional pooling unit, the height of the feature map becomes 1 / 8 of the original image, and the width becomes 1 / 4 of the original image; after passing through the fourth convolutional pooling unit, the height of the feature map becomes 1 / 16 of the original image, and the width becomes 1 / 4 of the original image; finally, 256 1x40 feature maps are extracted through the feature extractor;

[0104] The feature vector sequence is extracted from the 256 1*40 feature maps, specifically, 256 pixel points at each position from left to right on the feature map are taken as a feature vector to obtain 40 256-dimensional feature vector sequences, since the convolution and pooling operations are translation invariant, each feature vector corresponds to a rectangular region of the original image, and these rectangular regions have the same order as the corresponding positions from left to right on the feature map.

[0105] Since the recurrent neural network has great advantages in processing sequence data, a recurrent neural network is constructed to process the feature sequence obtained above, since the original recurrent neural network (RNN) is prone to gradient disappearance problem in the case of long time steps, the LSTM network is used to act as a recurrent neural network in the present application, and the LSTM can capture long-distance dependencies, so it can process long-time step sequence data. In step S4, the process of constructing the recurrent neural network is as follows:

[0106] Step S41. An LSTM network with 40 time steps is constructed to process the feature vector sequence containing 40 feature vectors, each time step corresponding to a feature vector, the LSTM network contains 2 intermediate hidden layers, each hidden layer contains 256 neurons, and all are fully connected with the previous layer for further extracting features of the feature sequence.

[0107] Step S42. The output layer of the LSTM network is designed, the main function of the output layer is to output the probability of each character predicted by the LSTM at each time step, the identity card image usually contains several types of characters such as numbers, Chinese characters and English characters, the number characters contain 10 types, the English characters contain 26 types, and the frequently used Chinese characters contain about 1000 types, considering that there is no character at the position corresponding to some time steps, a null character is defined, represented by “-”, therefore the output layer of the LSTM network contains 1037 neurons, the LSTM outputs 1037 probability values at each time step, indicating the probability of classification as a certain character, the 1037 neurons correspond to 10 types of numbers, 26 types of English characters, 1000 types of Chinese characters and 1 null character representing the position corresponding to the time step on the identity card image.

[0108] Step S43. For the 1037 output probabilities of each time step obtained in step S42, the character class with the largest probability is taken as the character class output at this time step, and 40 time steps output 40 characters. However, there are not so many characters on some character region images, so there may be repeated characters in the output result, and therefore a de-duplication operation needs to be performed on the output characters. The specific method is to insert a "-" between repeated characters in the text label when the LSTM output sequence is output, and consecutive identical characters without "-" are merged, and identical characters with "-" are not merged. Finally, if there is a blank character, it is deleted to obtain the final character recognition result.

[0109] The identity card recognition device based on deep learning comprises a memory and a processor.

[0110] Practical application

[0111] The following will take constructing a backend identity card recognition system as an example to illustrate an implementation method of the application. Figure 7 As shown in the figure, the system is mainly divided into an application layer, an SDK layer and a persistence layer.

[0112] The technology stack used by the application layer is "Springboot+Mybatis", and the main function is to expose service interfaces to the outside, receive identity card images transmitted by external requests, and complete some business logic processing, such as calling the identity card recognition capability of the SDK layer to obtain information on the identity card, and calling the persistence layer to store the information into the database.

[0113] The SDK layer is the core component of the system, which contains the identity card recognition model proposed by the application and the SDK toolkit encapsulating the model capability, and the main function is to process and recognize the identity card images transmitted by the application layer and return the recognition result to the application layer.

[0114] The persistence layer is responsible for persistently storing some intermediate results processed by the system, such as identity card information, processing time, processing log, etc.

[0115] When there is an external request transmitted, the main processing process of the system is as follows:

[0116] 1. The application layer receives the identity card image transmitted by the request, and at the same time performs some checking operations on the transmitted image, such as checking whether the image file is damaged and whether the image size is too small (width or height less than 50 pixels);

[0117] 2. The application layer writes log information to the persistent layer, recording the time of request arrival, the size of the incoming image file, and the path of image storage, etc.

[0118] 3. The application layer calls the SDK layer toolkit through JNA, and transmits the image to be detected;

[0119] 4. After receiving the image transmitted by the application layer, the SDK layer starts processing the image;

[0120] 5. The image is processed for grayscale, Gaussian blur, Canny edge detection, and Hough transform to obtain the inclination of the image, and the original image is corrected according to the inclination to obtain the corrected image;

[0121] 6. The image character area is detected, and the image is input to the detection model to obtain 8x8x3 bounding boxes for detecting large targets, 16x16x3 bounding boxes for detecting medium targets, and 32x32x3 bounding boxes for detecting small targets;

[0122] 7. The 4032 bounding boxes obtained in step 6 are screened. First, a category threshold threshold is set. If the classification score of the bounding box for the character area is lower than threshold, it means that the possibility of characters in this bounding box is small, and the candidate box is removed. The remaining boxes are subjected to secondary screening using a non-maximum suppression algorithm, the purpose of which is to screen out multiple highly overlapping bounding boxes that detect the same character area. Therefore, 5-6 bounding boxes after screening are selected as the detection results of the character detection model;

[0123] 8. Each character area detection result in step 7 is converted into a grayscale image, and the size of the image is adjusted to 16x160x1, and then input to the feature extractor for feature extraction to obtain 256 1x40 feature maps;

[0124] 9. Take 256 pixel points at each position from left to right on the feature map as a feature vector to obtain a sequence composed of 40 256-dimensional feature vectors;

[0125] 10. Input the feature vector sequence into the LSTM network with 40 time steps, and the LSTM outputs the prediction results of 1037 character classes at each time step;

[0126] 11. Take the character class with the highest probability as the output character class at each time step;

[0127] 12. Perform a de-duplication operation on the output results to obtain the final character detection results;

[0128] 13. The SDK layer returns the ID card character detection results to the application layer;

[0129] 14. The application layer writes the detection result into the database for storage, such as the information of "name", "date of birth", and "ID number" on the ID image;

[0130] 15. The application layer returns the detection result to the external client.

[0131] The above is only the preferred embodiment of the present application, it should be noted that for those skilled in the art, without departing from the structure of the present application, can also make several variations and improvements, these will not affect the effect of the present application and the practicality of the patent.

Claims

1. A deep learning-based identity card recognition method, characterized in that, The method comprises the following steps: Step S1. Obtain an ID card image, and perform tilt correction processing on the ID card image to obtain a preprocessed image; Step S2. Construct a region detection model based on yolov3, and detect each character region in the preprocessed image through the region detection model; Step S3. Construct a feature extractor based on a convolutional neural network to extract image features of each character region, and extract a feature vector sequence from the image features; Step S4. Construct a recurrent neural network to process the feature vector sequence to obtain a final character recognition result; In step S2, the region detection model is constructed as follows: Step S21. Design a CBL unit, which is composed of a convolutional layer, a BN layer and a Leaky Relu layer connected in sequence, and the Leaky Relu layer uses Leaky Relu as an activation function; Step S22. Design a residual unit, which contains two CBL units connected in sequence, and the input of the residual unit is processed through the two CBL units, and the result is added to the input of the residual unit to obtain the output of the residual unit; Step S23. Design a residual module, which contains two CBL units and three residual units connected in sequence; Step S24. Design the main network of the model by connecting a plurality of residual modules to form the main network of the model; Step S25. Design a multi-scale detection unit for detecting character regions at multiple scales; In step S24, the main network of the model is formed by connecting three residual modules, and the convolution kernel parameters of the three residual modules are configured as follows: The convolution kernel parameters of the two CBL units in the first residual module are both 3x3x16; the convolution kernel parameters of the three residual units are the same, and the convolution kernel parameters of the two CBL units in the residual unit are 3x3x8 and 1x1x16, respectively; The convolution kernel parameters of the two CBL units in the second residual module are both 3x3x32; the convolution kernel parameters of the three residual units are the same, and the convolution kernel parameters of the two CBL units in the residual unit are 3x3x16 and 1x1x32, respectively; The convolution kernel parameters of the two CBL units in the third residual module are both 3x3x64; the convolution kernel parameters of the three residual units are the same, and the convolution kernel parameters of the two CBL units in the residual unit are 3x3x32 and 1x1x64, respectively; In step S25, the multi-scale detection unit is designed as follows: The large target scale unit is designed, the large target scale unit comprises three CBL units and one full convolution layer, the convolution kernel parameter configurations of the three CBL units are 1*1*64, 3*3*128 and 1*1*64 respectively, and the full convolution layer parameter configuration is 1*1*21; the unit takes the output of the third residual module of the backbone network as input, outputs 8*8*[3*(2+5)] results through the full convolution layer, represents that three bounding boxes are set at each point on the 8*8 feature map, and then outputs the scores of two categories of each bounding box, the center point coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box and the bounding box confidence; The medium target scale unit is designed, the medium target scale unit comprises three CBL units and one full convolution layer, the convolution kernel parameter configurations of the three CBL units are 1*1*32, 3*3*64 and 1*1*32 respectively, and the full convolution layer parameter configuration is 1*1*21; the unit takes the output of the second residual module of the backbone network as input, outputs 16*16*[3*(2+5)] results through the full convolution layer, represents that three bounding boxes are set at each point on the 16*16 feature map, and then outputs the scores of two categories of each bounding box, the center point coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box and the bounding box confidence; The small target scale unit is designed, the small target scale unit comprises three CBL units and one full convolution layer, the convolution kernel parameter configurations of the three CBL units are 1*1*16, 3*3*32 and 1*1*16 respectively, and the full convolution layer parameter configuration is 1*1*21; the unit takes the output of the first residual module of the backbone network as input, outputs 32*32*[3*(2+5)] results through the full convolution layer, represents that three bounding boxes are set at each point on the 32*32 feature map, and then outputs the scores of two categories of each bounding box, the center point coordinates (x, y) of the bounding box, the width w of the bounding box, the height h of the bounding box and the bounding box confidence. 2.The identity card recognition method based on deep learning according to claim 1, characterized in that, Step S1 comprises: Step S11. Obtain a gray image of the identity card image; Step S12. Perform Gaussian blur processing on the gray image obtained in step S11; Step S13. Perform Canny edge detection on the gray image obtained in step S12 to obtain a Canny edge detection image; Step S14. Perform Hough transform on the Canny edge detection image obtained in step S13 to obtain an array composed of polar coordinates of all straight lines in the edge detection image, and then calculate the inclination of the image according to the obtained polar coordinate array; Step S15. Adjust the identity card image according to the inclination calculated in step S14 to obtain a preprocessed image after inclination correction. 3.The identity card recognition method based on deep learning according to claim 1, characterized in that, Further comprising step S26, after the region detection model is constructed, the region detection model needs to be trained using a set of identity card image data, and the specific process is as follows: Collect a large number of virtual identity card images; The virtual ID card image is subjected to a data enhancement operation to obtain an ID card image dataset, and the enhancement operation includes at least one of image flipping, image smoothing, and adding Gaussian noise. An optimization objective function of the training is designed, and a simplified model of the optimization objective function is as follows: L = L (x,y) + L size + L conf + L classes where L (x,y) is the bounding box center point error, L size is the bounding box size error, L conf is the bounding box confidence error, and L classes is the class error. Adam as the optimizer, the initial learning rate is set to 1e -4 , the number of training iterations is set to 50000 times; The training process is as follows: Step S261. 128 images are taken from the ID card image dataset each time to input the model, and the loss value of the optimization objective function is calculated according to the output of the region detection model and the true value; Step S262. The weight parameters of the region detection model are updated using the Adam gradient descent algorithm, and the iteration number is increased by 1; Step S263. Steps S261 and S262 are repeated until the iteration number reaches 50,000, and the model training is completed. 4.The identity card recognition method based on deep learning of claim 1, characterized in that, In step S3, the process of constructing the feature extractor is as follows: Step S31. The gray image of the character region image is obtained, and the size of the character region image is adjusted to 16x160. A 16x160x1 input layer is constructed to receive the character region gray image; Step S32. A convolutional pooling unit is constructed, which includes a convolutional layer, a BN layer, a Relu layer, and a maximum pooling layer connected in sequence; Step S33. A feature extractor is constructed by connecting an input layer and four convolutional pooling units in sequence. The convolutional pooling parameters of each convolutional pooling unit are as follows: the first convolutional pooling unit uses 32 3x3 convolutional kernels, and the maximum pooling window is set to 2x2; the second convolutional pooling unit uses 64 3x3 convolutional kernels, and the maximum pooling window is set to 2x2; the third convolutional pooling unit uses 128 3x3 convolutional kernels, and the maximum pooling window is set to 2x1; and the fourth convolutional pooling unit uses 256 3x3 convolutional kernels, and the maximum pooling window is set to 2x1. 5.The identity card recognition method based on deep learning according to claim 4, characterized in that, After the character region gray image passes through the first convolutional pooling unit, the height and width of the feature map become 1 / 2 of the original image; after passing through the second convolutional pooling unit, the height and width of the feature map become 1 / 4 of the original image; after passing through the third convolutional pooling unit, the height of the feature map becomes 1 / 8 of the original image, and the width becomes 1 / 4 of the original image; and after passing through the fourth convolutional pooling unit, the height of the feature map becomes 1 / 16 of the original image, and the width becomes 1 / 4 of the original image. Finally, 256 1x40 feature maps are extracted by the feature extractor; A feature vector sequence is extracted from the 256 1x40 feature maps. Specifically, 256 pixel points at each position from left to right on the feature map are taken as a feature vector to obtain 40 256-dimensional feature vector sequences. 6.The identity card recognition method based on deep learning according to claim 5, characterized in that, In step S4, the process of constructing the recurrent neural network is as follows: Step S41. An LSTM network with 40 time steps is constructed to process the feature vector sequence containing 40 feature vectors. Each time step corresponds to a feature vector, and the LSTM network includes two intermediate hidden layers, each containing 256 neurons, and all of which are fully connected to the previous layer to further extract the features of the feature sequence; Step S42. Design the output layer of the LSTM network, which is mainly used to output the probability of each character predicted by the LSTM at each time step. The output layer of the LSTM network contains 1037 neurons, and the LSTM outputs 1037 probability values at each time step, representing the probability of classification as a certain character. The 1037 neurons correspond to 10 types of numbers, 26 types of English characters, 1000 types of Chinese characters, and 1 empty character representing the position corresponding to the time step. The empty character is represented by "-". Step S43. For the 1037 output probabilities of each time step obtained in step S42, take the character class with the maximum probability as the character class output at that time step. 40 time steps will output 40 characters. Perform a de-duplication operation on the output characters. Specifically, when the LSTM outputs a sequence, insert a "-" between repeated characters in the text label. Continuous identical characters without "-" will be merged, and identical characters with "-" will not be merged. Finally, if there is a blank character, it is deleted to obtain the final character recognition result.

7. A deep learning-based identity card recognition device, characterized by, The system comprises a memory and a processor. The memory is used to store a computer program. The processor is used to execute the computer program to implement the identity card recognition method based on deep learning according to any one of claims 1-6.

Images