Passport MRZ character recognition method

Abstract

The invention relates to the technical field of character recognition, in particular to a passport MRZ character recognition method, which comprises the following steps: acquiring an image; detecting the position of the MRZ area; extracting an MRZ region image; detecting and classifying each character in the MRZ area image, and outputting the coordinate of each character and the corresponding character category; and sorting the characters, and outputting an identification result. Wherein the step of detecting the position of the MRZ area comprises the following steps of: carrying out binarization processing on an acquired image; detecting a straight line in the image; the MRZ region is located according to the straight line. The MRZ characters of the passport are recognized through the network model, compared with a traditional OCR algorithm and a target detection algorithm, character-by-character scanning of the image is reduced, the calculation time is greatly shortened, and the recognition speed is increased.

Description

Technical Field

[0001] The invention relates to the technical field of character recognition, and in particular to a passport MRZ character recognition method. Background Art

[0002] In the process of character recognition in the machine readable zone (MRZ) of a passport, traditional recognition methods are mainly based on optical character recognition (OCR) technology. OCR technology scans images character by character and converts them into editable text, and is widely used in the recognition of various types of documents and certificates. However, in an environment with limited hardware computing power, the computational complexity of the OCR algorithm is high and the processing speed is slow, especially in scenarios with high real-time requirements. In addition, the OCR algorithm has high requirements for image clarity and character integrity. When the characters are damaged, distorted, or the background is complex, the OCR recognition rate tends to drop significantly.

[0003] In order to increase recognition speed and adapt to resource-constrained devices, some solutions try to improve recognition efficiency by compressing the OCR model, reducing the resolution of character scanning, etc., but these methods often sacrifice recognition accuracy to a certain extent. In recent years, deep learning technology has been introduced into the field of character recognition, and image classification or detection models combined with convolutional neural networks (CNN) have made significant progress in improving recognition accuracy. However, existing deep learning models are mostly used to enhance OCR systems, and have failed to effectively solve the inefficiency caused by the inherent character-by-character scanning problem of OCR algorithms.

[0004] In addition, the wide application of target detection technology in the field of object detection has demonstrated its high efficiency and can quickly locate and classify different targets in images. However, there is currently little work that applies target detection technology to the recognition of passport MRZ characters, and most character recognition tasks still rely on OCR methods.

[0005] Therefore, there is an urgent need for a passport MRZ character recognition method that can improve the speed while ensuring recognition accuracy and can run efficiently on resource-constrained hardware devices. Summary of the invention

[0006] The present invention provides a passport MRZ character recognition method, which is used to solve the defect that most character recognition tasks in the prior art still rely on the OCR method, and realize passport MRZ character recognition that runs efficiently on resource-constrained hardware devices.

[0007] The present invention provides a passport MRZ character recognition method, comprising:

[0008] Collect images;

[0009] Detect the location of the MRZ area;

[0010] Extract MRZ area image;

[0011] Detect and classify each character in the MRZ area image, and output the coordinates of each character and the corresponding character category;

[0012] Sort the characters and output the recognition results.

[0013] According to a passport MRZ character recognition method provided by the present invention, the detecting the position of the MRZ area includes:

[0014] Binarize the collected images;

[0015] Detect straight lines in an image;

[0016] The MRZ area is located according to the straight line.

[0017] According to a passport MRZ character recognition method provided by the present invention, before binarizing the image, the image is preprocessed, and the preprocessing includes:

[0018] The collected RGB image is converted into a grayscale image using the weighted average method. The weighted average method is performed using the following formula (1):

[0019] Gray = 0.299×R+0.587×G+0.114×B (1)

[0020] Among them, Gray is the value of the gray channel of each pixel in the grayscale image, and R, G, and B are the values ​​of the red, green, and blue channels of each pixel in the RGB image, respectively.

[0021] According to a passport MRZ character recognition method provided by the present invention, the binarization process includes: filtering the image, then calculating the gradient of the image, then performing edge detection using non-maximum suppression, threshold processing, and edge connection, and outputting a binarized image;

[0022] The filtering process is performed using the following formula (2):

[0023]

[0024] In formula (2), H(x, y) is the Gaussian weight value of the pixel at the coordinate (x, y), δ is the standard deviation of the Gaussian distribution, and δ is 1.0;

[0025] The image gradient is calculated using the following formula (3) and formula (4):

[0026]

[0027] In formula (3), G(x,y) is the gradient amplitude of the pixel at the coordinate (x,y), G x is the gradient amplitude of the pixel in the horizontal direction, G y is the gradient amplitude of the pixel in the horizontal direction;

[0028] In formula (4), θ(x,y) is the gradient direction of the pixel at the coordinate (x,y), G x is the gradient amplitude of the pixel in the horizontal direction, G y It is the gradient amplitude of the pixel in the horizontal direction.

[0029] According to a passport MRZ character recognition method provided by the present invention, the straight line in the detection image includes:

[0030] Map edge points to parameter space;

[0031] Use an accumulator to count the votes of each point in parameter space;

[0032] Find the point with the most votes and determine the straight line;

[0033] Repeat the above steps until all straight lines are found;

[0034] Among them, the edge points are black points on the binary image, and the edge points are mapped using the following formula (5):

[0035] r = x*cos(θ) + y*sin(θ) (5)

[0036] In formula (5), (x, y) is the coordinate of the edge point in the image, r is the distance from the edge point to the line, and θ is the angle between the normal of the line and the positive direction of the x-axis.

[0037] According to a passport MRZ character recognition method provided by the present invention, the extracting of the MRZ area image comprises:

[0038] The MRZ area image containing the characters is cut out according to the straight line.

[0039] According to a passport MRZ character recognition method provided by the present invention, after extracting the MRZ area image, the method further includes adjusting the MRZ area image to a resolution of 160*800 and normalizing the pixel value of the MRZ area image to a range of [-1,1];

[0040] The normalization is performed using the following formula (6):

[0041]

[0042] Where X is the pixel value of the image, X min is the minimum pixel value in the image, Xmax is the maximum pixel value in the image, X norm is the normalized pixel value of the image.

[0043] According to a passport MRZ character recognition method provided by the present invention, each character in the MRZ area image is detected and classified, and the coordinates of each character and the corresponding character category are output, including:

[0044] The MRZ region image with adjusted resolution is input into the network model as the original feature map. After several convolution operations using formula (7), the obtained feature map is divided into two parts: X 1 and X 2 ,in:

[0045] Where C is the number of channels of the feature map;

[0046] X 1 Perform feature extraction and obtain feature map Y 1 ;

[0047] The feature map Y 1 With X 2 Connect and get the feature map Y 2 , for Y 2 Perform feature extraction to obtain the final feature map;

[0048] The above two feature extractions also use the following formula (7) for convolution operation:

[0049] Y(i,j) =∑ m ∑ n X(i+m,j+n)·K(m,n) (7)

[0050] In formula (7), Y(i, j) is the value of the output feature map at position (i, j), X is the input feature map, K is the convolution kernel with a size of k×k, k=3; m and n are the indices of the convolution kernel.

[0051] A passport MRZ character recognition method provided by the present invention also includes predicting the final feature map at different scales, outputting multiple predicted bounding boxes for each character, the confidence corresponding to each predicted bounding box and the probability of each character belonging to each category, and selecting a target bounding box from the multiple predicted bounding boxes; wherein the bounding box is obtained by the bounding box coordinates.

[0052] According to a passport MRZ character recognition method provided by the present invention, a method for selecting a target boundary box includes:

[0053] Sort the confidence scores of multiple predicted bounding boxes, select the predicted bounding box with the highest score as the candidate box and the one with the highest category probability as the character category;

[0054] Calculate the degree of overlap between the candidate box and other boxes of the same category. If the degree of overlap between the candidate box and other boxes exceeds a preset threshold, delete the other boxes.

[0055] Repeat the operation until all predicted bounding boxes are processed, and obtain the target bounding box and category of each character.

[0056] According to a passport MRZ character recognition method provided by the present invention, the characters are sorted and the recognition result is output, including:

[0057] Sort each character according to the coordinates of the target bounding box and output the corresponding character category.

[0058] The passport MRZ character recognition method provided by the present invention recognizes the MRZ characters of the passport through a network model. Compared with the traditional OCR algorithm, the target detection algorithm reduces the character-by-character scanning of the image, greatly reduces the calculation time, and improves the recognition speed; at the same time, since the computational complexity of the target detection algorithm is relatively low, it is suitable for resource-constrained devices and can be effectively run on embedded systems or low-computing power devices. The entire recognition method uses a deep learning target detection algorithm to effectively avoid the recognition errors that are prone to occur in traditional OCR when processing damaged or distorted characters while retaining high accuracy. This method is not only suitable for passport MRZ character recognition, but can also be extended to other character recognition tasks, such as the recognition of identity cards, driver's licenses and other documents. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 It is a flow chart of the passport MRZ character recognition method provided by the present invention;

[0061] Figure 2 yes Figure 1 A schematic diagram of a process for detecting the position of the MRZ area;

[0062] Figure 3 yes Figure 2 Schematic diagram of the process of detecting straight lines in an image. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0064] Combine the following Figure 1-Figure 3 A passport MRZ character recognition method of the present invention is described, comprising the following steps:

[0065] S100: Capture an image; the specific method is to use a camera to capture a video stream on the object containing the passport, and then intercept the current frame and send it to the area detection module to detect the position of the MRZ area.

[0066] S200: Detecting the position of the MRZ area; specifically including:

[0067] S201: pre-processing the collected image;

[0068] S202: Binarization processing is performed on the image;

[0069] S203: Detect straight lines in the image;

[0070] S204: Locate the MRZ area according to the straight line.

[0071] In step S201, the preprocessing includes: converting the collected RGB image into a grayscale image using a weighted average method, and the weighted average method is performed using the following formula (1):

[0072] Gray = 0.299×R+0.587×G+0.114×B (1)

[0073] Among them, Gray is the value of the gray channel of each pixel in the grayscale image, and R, G, and B are the values ​​of the red, green, and blue channels of each pixel in the RGB image, respectively.

[0074] Convert the color image to a grayscale image, remove the color information, and retain only the brightness information of the image. For line detection, the grayscale image is sufficient to describe the edges in the image and reduces the amount of calculation.

[0075] In step S202, the binarization process includes: filtering the image, calculating the gradient of the image, performing edge detection using non-maximum suppression, threshold processing, and edge connection, and outputting a binarized image;

[0076] The filtering process is performed using the following formula (2):

[0077]

[0078] In formula (2), H(x, y) is the Gaussian weight value of the pixel at the coordinate (x, y), δ is the standard deviation of the Gaussian distribution, and δ is 1.0;

[0079] The image gradient is calculated using the following formula (3) and formula (4):

[0080]

[0081] In formula (3), G(x,y) is the gradient amplitude of the pixel at the coordinate (x,y), G x is the gradient amplitude of the pixel in the horizontal direction, G y is the gradient amplitude of the pixel in the horizontal direction;

[0082] In formula (4), θ(x,y) is the gradient direction of the pixel at the coordinate (x,y), G x is the gradient amplitude of the pixel in the horizontal direction, G y It is the gradient amplitude of the pixel in the horizontal direction.

[0083] In this step, Canny edge detection is used to output a binary image. By calculating the gradient of the image and using non-maximum suppression to extract the edge, the significant edges in the image can be effectively detected while suppressing noise.

[0084] In step S203, the detecting of straight lines in the image specifically includes:

[0085] Map edge points to parameter space;

[0086] Use an accumulator to count the votes of each point in parameter space;

[0087] Find the point with the most votes and determine the straight line;

[0088] Repeat the above steps until all straight lines are found;

[0089] Among them, the edge points are black points on the binary image, and the edge points are mapped using the following formula (5):

[0090] r = x*cos(θ) + y*sin(θ) (5)

[0091] In formula (5), (x, y) is the coordinate of the edge point in the image, r is the distance from the edge point to the line, and θ is the angle between the normal of the line and the positive direction of the x-axis.

[0092] In this step, Hough Line Transform is used to detect straight lines in the image. In Hough Transform, the corresponding r value is calculated for each edge point, and votes are performed in the parameter space of r and θ. Finally, the (r, θ) corresponding to the point with the most votes in the parameter space represents a straight line in the image.

[0093] Since the arrangement characteristics of the MRZ character area are fixed horizontal straight lines, the MRZ area can be located by detecting these straight lines. The MRZ area on the passport usually includes two rows of characters, each with 44 characters, which are neatly arranged and fixed in the passport.

[0094] S300: Extracting the MRZ area image; specifically: cutting out the MRZ area image containing the characters according to the straight lines. Once the horizontal straight lines are detected, the system will cut out the MRZ area containing the characters according to the positions of these straight lines. This step ensures that the subsequent character detection is only performed in the specified area, thereby improving efficiency and reducing interference.

[0095] After extracting the MRZ area image, the method further includes adjusting the MRZ area image to a resolution of 160*800 and normalizing the pixel value of the MRZ area image to a range of [-1,1];

[0096] The normalization is performed using the following formula (6):

[0097]

[0098] Where X is the pixel value of the image, X min is the minimum pixel value in the image, X max is the maximum pixel value in the image, X norm is the normalized pixel value of the image.

[0099] Adjusting the image resolution to a uniform size and normalizing the image pixel values ​​to a specific range is to adapt to the subsequent network model, reduce the amount of calculation and improve recognition efficiency.

[0100] S400: Detect and classify each character in the MRZ area image, and output the coordinates of each character and the corresponding character category; specifically including:

[0101] S401: The MRZ region image with adjusted resolution is input into the network model as the original feature map. After performing several convolution operations using formula (7), the obtained feature map is divided into two parts: X 1 and X 2 ,in:

[0102] Where C is the number of channels of the feature map;

[0103] X 1 Perform feature extraction and obtain feature map Y 1 ;

[0104] The feature map Y 1 With X 2 Connect and get the feature map Y 2 , for Y 2 Perform feature extraction to obtain the final feature map;

[0105] The above two feature extractions also use the following formula (7) for convolution operation:

[0106] Y(i,j) =∑ m ∑ n X(i+m,j+n)·K(m,n) (7)

[0107] In formula (7), Y(i, j) is the value of the output feature map at position (i, j), X is the input feature map, K is the convolution kernel with a size of k×k, k=3; m and n are the indices of the convolution kernel.

[0108] In this step, the network model uses a lightweight and efficient convolutional neural network structure, such as CSPNet (Cross Stage Partial Networks), to improve the effect of feature extraction while reducing the amount of calculation. At the same time, the network model also uses a feature pyramid network (FPN) to enhance the transmission and fusion of features of different scales, thereby improving the detection performance of small objects.

[0109] S402: Predict the final feature map at different scales, output multiple predicted bounding boxes for each character, the confidence level corresponding to each predicted bounding box, and the probability that each character belongs to each category, and select a target bounding box from the multiple predicted bounding boxes.

[0110] The bounding box is obtained from the bounding box coordinates; that is, during prediction, each character will be identified as an independent target and output several groups of 6 numbers, representing its coordinates (x 1 ,y 1, w,h), the corresponding character category (class) and confidence (confidence);

[0111] Among them, at the coordinate (x 1 ,y 1, w,h), (x 1 ,y 1) represents the coordinate of the upper left corner of the target, w represents the width of the target, and h represents the height of the target; class represents a class in the character set; confidence represents the reliability of the result, and we take a result greater than 0.5.

[0112] There are multiple predicted bounding boxes in the above output, so the best bounding box needs to be selected as the target bounding box.

[0113] Among them, the method of selecting the target bounding box includes:

[0114] Sort the confidence scores of multiple predicted bounding boxes, select the predicted bounding box with the highest score as the candidate box and the one with the highest category probability as the character category;

[0115] Calculate the degree of overlap between the candidate box and other boxes of the same category. If the degree of overlap between the candidate box and other boxes exceeds a preset threshold, delete the other boxes.

[0116] Repeat the operation until all predicted bounding boxes are processed, and obtain the target bounding box and category of each character.

[0117] This method is mainly used to select an optimal box from multiple overlapping candidate bounding boxes, eliminate redundant bounding boxes, and retain the most likely target box. Its function is to reduce repeated detection results and ensure that each target is detected only once; in the above method, "selecting the predicted bounding box with the highest score as the candidate box and the one with the highest category probability as the character category" can narrow the scope and thus reduce the amount of calculation; "calculating the degree of overlap between the candidate box and other boxes of the same category" emphasizes other boxes of the same category, further reducing the amount of calculation.

[0118] In this step, the IoU (Intersection over Union) formula is used to calculate the overlap between the candidate box and other boxes (or between the candidate boxes):

[0119]

[0120] Among them: B i and B j There are two candidate boxes; Area of ​​Intersection (B i ,B j ) is the area of ​​the intersection of the two boxes; Area of ​​Union (B i ,B j ) is the area of ​​the union of the two boxes.

[0121] S500: Sort the characters and output the recognition results; specifically: sort each character according to the coordinates of the target bounding box and output the corresponding character category; more specifically, based on the coordinates output by the network model, the system arranges the characters in horizontal and vertical order. Since the MRZ zone characters strictly follow the arrangement format of two rows of 44 characters on the passport, the sorting algorithm can output orderly results according to the character position very accurately.

[0122] The number of categories in the network model is set to 40, covering all possible characters in the passport MRZ. The character set of the passport MRZ includes 26 English letters (AZ), 10 numbers (0-9), and some special symbols, so the total number of character categories is 40.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A passport MRZ character recognition method, characterized in that: include: Collect images; Detect the location of the MRZ area; Extract MRZ area image; Detect and classify each character in the MRZ area image, and output the coordinates of each character and the corresponding character category; Sort the characters and output the recognition results.

2. The passport MRZ character recognition method according to claim 1, characterized in that: The detecting the position of the MRZ area includes: Binarize the collected images; Detect straight lines in an image; The MRZ area is located according to the straight line.

3. The passport MRZ character recognition method according to claim 2, characterized in that: Before performing binarization processing on the image, the image is preprocessed, and the preprocessing includes: The collected RGB image is converted into a grayscale image using the weighted average method. The weighted average method is performed using the following formula (1): Gray = 0.299×R+0.587×G+0.114×B (1) Among them, Gray is the value of the gray channel of each pixel in the grayscale image, and R, G, and B are the values ​​of the red, green, and blue channels of each pixel in the RGB image, respectively.

4. The passport MRZ character recognition method according to claim 3, characterized in that: The binarization process includes: filtering the image, calculating the gradient of the image, performing edge detection using non-maximum suppression, threshold processing, and edge connection, and outputting a binarized image; The filtering process is performed using the following formula (2): In formula (2), H(x, y) is the Gaussian weight value of the pixel at the coordinate (x, y), δ is the standard deviation of the Gaussian distribution, and δ is 1.0; The image gradient is calculated using the following formula (3) and formula (4): In formula (3), G(x,y) is the gradient amplitude of the pixel at the coordinate (x,y), G x is the gradient amplitude of the pixel in the horizontal direction, G y is the gradient amplitude of the pixel in the horizontal direction; In formula (4), θ(x,y) is the gradient direction of the pixel at the coordinate (x,y), G x is the gradient amplitude of the pixel in the horizontal direction, G y It is the gradient amplitude of the pixel in the horizontal direction.

5. The passport MRZ character recognition method according to claim 4, characterized in that: The detecting a straight line in an image comprises: Map edge points to parameter space; Use an accumulator to count the votes of each point in parameter space; Find the point with the most votes and determine the straight line; Repeat the above steps until all straight lines are found; Among them, the edge points are black points on the binary image, and the edge points are mapped using the following formula (5): r = x*cos(θ) + y*sin(θ) (5) In formula (5), (x, y) is the coordinate of the edge point in the image, r is the distance from the edge point to the line, and θ is the angle between the normal of the line and the positive direction of the x-axis.

6. The passport MRZ character recognition method according to claim 1, characterized in that: After extracting the MRZ area image, the method further includes adjusting the MRZ area image to a resolution of 160*800 and normalizing the pixel value of the MRZ area image to a range of [-1,1]; The normalization is performed using the following formula (6): Where X is the pixel value of the image, X min is the minimum pixel value in the image, X max is the maximum pixel value in the image, X norm is the normalized pixel value of the image.

7. The passport MRZ character recognition method according to claim 1, characterized in that: The detecting and classifying of each character in the MRZ area image and outputting the coordinates of each character and the corresponding character category include: The MRZ region image with adjusted resolution is input into the network model as the original feature map. After several convolution operations using formula (7), the obtained feature map is divided into two parts: X1 and X2, where: Where C is the number of channels of the feature map; Extract features from X1 and obtain feature map Y1; Connect feature map Y1 and X2 to obtain feature map Y2, perform feature extraction on Y2 to obtain the final feature map; The above two feature extractions also use the following formula (7) for convolution operation: Y(i,j) =∑ m ∑ n X(i+m,j+n)·K(m,n) (7) In formula (7), Y(i, j) is the value of the output feature map at position (i, j), X is the input feature map, K is the convolution kernel with a size of k×k, k=3; m and n are the indices of the convolution kernel.

8. The passport MRZ character recognition method according to claim 7, characterized in that: It also includes predicting the final feature map at different scales, outputting multiple predicted bounding boxes for each character, the confidence corresponding to each predicted bounding box and the probability that each character belongs to each category, and selecting a target bounding box from the multiple predicted bounding boxes; wherein the bounding box is obtained by the bounding box coordinates.

9. The passport MRZ character recognition method according to claim 8, characterized in that: Methods for selecting target bounding boxes include: Sort the confidence scores of multiple predicted bounding boxes, select the predicted bounding box with the highest score as the candidate box and the one with the highest category probability as the character category; Calculate the degree of overlap between the candidate box and other boxes of the same category. If the degree of overlap between the candidate box and other boxes exceeds a preset threshold, delete the other boxes. Repeat the operation until all predicted bounding boxes are processed, and obtain the target bounding box and category of each character.

10. The passport MRZ character recognition method according to claim 9, characterized in that: The step of sorting the characters and outputting the recognition results includes: Sort each character according to the coordinates of the target bounding box and output the corresponding character category.

Images