Passport image alignment method and device

Abstract

The present invention discloses a pass image alignment method and device, which relates to the field of information recognition technology, including: circumscribing a rectangular frame for feature information in a calibration image; generating a first matrix of all 0s; accumulating a normalized image to a second matrix, and not accumulating positions with pixel values ​​of -1 in the second matrix; calculating key points; adding key points to a set S; judging whether the number Q of key points in the set S is equal to n1, if not, jumping to the key point calculation step; if so, using the set S as a template key point set; training a key point prediction model and calculating the key point set of the image to be aligned; matching the key point set of the image to be aligned with the template key point set; using a perspective transformation matrix to perform a perspective transformation on the image to be aligned to obtain an aligned image. The present invention is based on data statistics, has many selectable key points, and selects key points with stable structures from candidate key points to solve the transformation matrix, which can effectively reduce errors.

Description

Technical Field

[0001] The present invention relates to the field of information recognition technology, and in particular to a pass image alignment method and device. Background Art

[0002] Pass identification has important applications in many occasions. Currently, chip reading and optical image acquisition are performed through specific reading devices. After optical image acquisition, OCR (Optical Character Recognition) and information structuring are required to obtain relevant document information. Therefore, high accuracy of OCR recognition is the key to the application.

[0003] At present, the more mature solution in the research field is to detect the text lines in the image through the text line detection model, and then send it to the recognition core for recognition, and extract key information through rule-based methods or deep learning-based methods. Key information extraction refers to the extraction of key information such as name and validity period in the document image. However, the direct application of existing technologies in actual engineering environments will encounter some problems, such as slow speed, errors in the detection and recognition of some images with stains, wear, and blurred imaging, which will lead to failure in key information extraction. To solve this problem, we can use the prior knowledge of the fixed document format. After image alignment, we can extract key information based on the format structure and the text line detection results. Therefore, the image alignment method can be embedded in the existing recognition technology process to improve recognition accuracy. At present, there is a method that detects the four corner points of the target, rotates according to the detected four points, and normalizes and aligns the image size. However, this method has two problems. First, when a positioning point is missing, the algorithm cannot achieve image alignment; second, the document is usually rounded, the corner features are not obvious, and there is a problem of inaccurate positioning. Summary of the invention

[0004] The present invention provides a pass image alignment method and device to overcome at least one technical problem existing in the prior art.

[0005] On the one hand, an embodiment of the present invention provides a pass image alignment method, comprising:

[0006] Acquiring multiple sample images;

[0007] Preprocessing the sample image to obtain a calibration image;

[0008] A rectangular frame is drawn around the feature information in the calibration image to mark the position of the feature information;

[0009] Generate a first matrix of all zeros, where the size of the first matrix is ​​the same as the size of the calibration image;

[0010] Setting all pixel values ​​in the first matrix corresponding to the rectangular frame to a taboo state of -1 to obtain a second matrix;

[0011] Normalizing the grayscale pixel values ​​of the calibration image to obtain a normalized image;

[0012] Accumulate the normalized image into the second matrix, and do not accumulate the positions with pixel values ​​of -1 in the second matrix, to obtain a third matrix;

[0013] Obtain k points with the largest pixel values ​​and k points with the smallest pixel values ​​in the third matrix, and generate a set Max_xy and a set Min_xy respectively;

[0014] Take the first point from the set Max_xy and the set Min_xy respectively and add them to the set S, and set the number of key points in the set S to Q=2;

[0015] After deleting the extracted points from the set Max_xy and the set Min_xy, merging the set Max_xy and the set Min_xy to obtain the set MaxMin_xy;

[0016] Calculate key points based on the distances from points in the set MaxMin_xy to each point in the set S;

[0017] Add the key point to the set S, set Q=Q+1, and delete the key point from the set MaxMin_xy;

[0018] Determine whether the number Q of key points in the set S is equal to n1. If not, jump to the key point calculation step; if so, use the set S as the template key point set S2;

[0019] Using the template key point set S2, training a key point prediction model;

[0020] The key point set of the image to be aligned is calculated using the key point prediction model as S1;

[0021] Matching the key point set S1 of the image to be aligned with the key point set S2 of the template to obtain matching point pairs;

[0022] Calculating a perspective transformation matrix using the matching point pairs;

[0023] The perspective transformation matrix is ​​used to perform perspective transformation on the image to be aligned to obtain an aligned image.

[0024] Optionally, the sample image is preprocessed, specifically:

[0025] The sample image is cropped, and the size of the sample image is normalized and rotated to obtain a calibration image with a size of width w and height h; the difference between the boundary of the calibration image and the boundary of the target in the calibration image is less than or equal to 1 pixel.

[0026] Optionally, the characteristic information includes text line information related to the document owner.

[0027] Optionally, the grayscale pixel values ​​of the calibration image are normalized, specifically:

[0028] The grayscale pixel value of the calibration image is divided by 255 and normalized to a floating point number between 0 and 1.

[0029] Optionally, the set Max_xy includes the coordinates of k points with the largest pixel values ​​in the third matrix, and the points in the set Max_xy are arranged in descending order of pixel values;

[0030] The set Min_xy includes the coordinates of k points with the smallest pixel values ​​in the third matrix, and the points in the set Min_xy are arranged in order from small to large pixel values.

[0031] Optionally, the key point is calculated according to the distance from the point in the set MaxMin_xy to each point in the set S, specifically:

[0032] Traverse and calculate the sum of the distances from the points in the set MaxMin_xy to the points in the set S, and obtain the point with the largest sum of distances , ,in, represents the Euclidean distance from the i-th point in the set MaxMin_xy to the j-th point in the set S, Indicates the minimum distance between key points;

[0033] The point with the largest sum of the distances is taken as the key point.

[0034] Optionally, the key point set is used to construct a key point prediction model, specifically:

[0035] Using the template key point set S2 to train the deep convolutional neural network to obtain a loss function;

[0036] The loss function is used to iteratively calculate the deep convolutional neural network to obtain a key point prediction model.

[0037] Optionally, the loss function is ,in, Represents the predicted coordinate point, represents the coordinate point in the template key point set S2, represents the weight of the i-th key point.

[0038] Optionally, a RANSAC (Random Sampling Consensus) algorithm is used to match the key point set S1 of the image to be aligned and the template key point set S2.

[0039] On the other hand, the present invention also provides a passport image alignment device, comprising:

[0040] An acquisition module, used for acquiring multiple sample images;

[0041] A preprocessing module, used for preprocessing the sample image to obtain a calibration image;

[0042] A marking module, used for enclosing a rectangular frame around the feature information in the calibration image to mark the position of the feature information;

[0043] A generating module, used for generating a first matrix of all zeros, wherein the size of the first matrix is ​​the same as the size of the calibration image;

[0044] A setting module, used for setting all pixel values ​​in the first matrix corresponding to the rectangular frame to a taboo state of -1, to obtain a second matrix;

[0045] A normalization module, used for normalizing the grayscale pixel values ​​of the calibration image to obtain a normalized image;

[0046] An accumulation module, used for accumulating the normalized image into the second matrix, and not accumulating the positions with pixel values ​​of -1 in the second matrix, to obtain a third matrix;

[0047] An acquisition module, used to acquire k points with the largest pixel values ​​and k points with the smallest pixel values ​​in the third matrix, and generate a set Max_xy and a set Min_xy respectively;

[0048] An extraction module, used for taking out the first point from the set Max_xy and the set Min_xy respectively and adding them to the set S, and setting the number of key points in the set S to Q=2;

[0049] A merging module, used for deleting the points taken out from the set Max_xy and the set Min_xy, and then merging the set Max_xy with the set Min_xy to obtain the set MaxMin_xy;

[0050] A first calculation module, used for calculating key points according to the distances from the points in the set MaxMin_xy to the points in the set S;

[0051] An adding module, used for adding the key point to the set S, setting Q=Q+1, and deleting the key point from the set MaxMin_xy;

[0052] A judging module, used to judge whether the number Q of key points in the set S is equal to n1, if not, jump to the key point calculation step; if yes, take the set S as the template key point set S2;

[0053] A training module, used for training a key point prediction model using the template key point set S2;

[0054] A second calculation module, used for calculating the key point set S1 of the image to be aligned by using the key point prediction model;

[0055] A matching module, used for matching the key point set S1 of the image to be aligned with the key point set S2 of the template to obtain matching point pairs;

[0056] A third calculation module, used to calculate a perspective transformation matrix using the matching point pairs;

[0057] The transformation module is used to perform perspective transformation on the image to be aligned using the perspective transformation matrix to obtain an aligned image.

[0058] The innovative features of the embodiments of the present invention include:

[0059] 1. In this embodiment, based on data statistics, representative key points of the image related to the layout are adaptively found, and there are many selectable key points with higher robustness, which is one of the innovative points of the embodiment of the present invention.

[0060] 2. In this embodiment, structurally stable key points are selected from candidate key points to solve the transformation matrix, which can effectively reduce errors and is one of the innovative points of the embodiment of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0062] Figure 1 A flow chart of a passport image alignment method provided by an embodiment of the present invention;

[0063] Figure 2 A schematic structural diagram of a pass image alignment device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0066] The embodiment of the present invention discloses a pass image alignment method and device, which are described in detail below.

[0067] Figure 1 For a flowchart of the pass image alignment method provided by an embodiment of the present invention, please refer to Figure 1 , the pass image alignment method provided by the embodiment of the present invention includes:

[0068] Step 1: Collect multiple sample images;

[0069] Step 2: Preprocess the sample image to obtain the calibration image;

[0070] Step 3: Draw a rectangular frame around the feature information in the calibration image to mark the location of the feature information;

[0071] Step 4: Generate a first matrix of all zeros, the size of the first matrix is ​​the same as the size of the calibration image;

[0072] Step 5: Set all pixel values ​​in the positions corresponding to the rectangular frame in the first matrix to the taboo state -1 to obtain the second matrix;

[0073] Step 6: Normalize the grayscale pixel values ​​of the calibration image to obtain a normalized image;

[0074] Step 7: Add the normalized image to the second matrix, and do not add the pixel value of -1 in the second matrix to obtain the third matrix;

[0075] Step 8: Obtain the k points with the largest pixel values ​​and the k points with the smallest pixel values ​​in the third matrix, and generate the set Max_xy and the set Min_xy respectively;

[0076] Step 9: Take the first point from the set Max_xy and the set Min_xy respectively and add them to the set S, and set the number of key points in the set S to Q=2;

[0077] Step 10: After deleting the points taken out from the set Max_xy and the set Min_xy, merge the set Max_xy and the set Min_xy to obtain the set MaxMin_xy;

[0078] Step 11: Calculate the key points based on the distances from the points in the set MaxMin_xy to the points in the set S;

[0079] Step 12: Add the key points to the set S, set Q = Q + 1, and delete the key points from the set MaxMin_xy;

[0080] Step 13: Determine whether the number Q of key points in set S is equal to n1. If not, jump to the key point calculation step; if so, use set S as the template key point set S2;

[0081] Step 14: Using the template key point set S2, train a key point prediction model;

[0082] Step 15: Calculate the key point set of the image to be aligned using the key point prediction model as S1;

[0083] Step 16: Match the key point set S1 of the image to be aligned with the key point set S2 of the template to obtain matching point pairs;

[0084] Step 17: Use the matching point pairs to calculate the perspective transformation matrix;

[0085] Step 18: Use the perspective transformation matrix to perform perspective transformation on the image to be aligned to obtain the aligned image.

[0086] For details, please refer to Figure 1 The passport image alignment method provided in the embodiment of the present invention first collects multiple sample images through step 1. The sample here refers to a passport document of a certain format, which is obtained by scanning or collecting with a passport machine without perspective distortion. In order to meet the requirements of subsequent model training, the image needs to be preprocessed. In this embodiment, through step 2, the sample image collected in step 1 is preprocessed, such as cropping, image size normalization, and rotation, to obtain a calibration image.

[0087] Normally, the text information in the certificate can be roughly divided into two parts. One part is the key information, that is, the information that is printed in advance and has a fixed character category and a fixed position; the other part is the value information related to the owner of the certificate, such as the text line information related to the owner of the certificate, that is, the feature information. The character category of this type of information is uncertain, and the length of the text line is different. Since the image appearance characteristics of the corresponding part of the value information are variable, the reference point of this part is not fixed and is not easy to use for calculating candidate key points. Therefore, the present invention uses step 3 to mark the position of this part in the calibration image by enclosing a rectangular frame around the value information, so that the pixels corresponding to this part can be ignored in the subsequent calculation of candidate key points.

[0088] In step 4, a first matrix with the same size as the calibration image is generated. The first matrix is ​​used as the initial matrix. Therefore, the first matrix is ​​set to an accumulation matrix of all 0s. At the same time, since the pixels corresponding to the value information are not easy to be used to calculate the candidate key points, these pixels need to be ignored during the image accumulation process. Therefore, the present invention sets all pixel values ​​in the corresponding positions of the rectangular frame in the first matrix to -1 through step 5, indicating a taboo state. By setting the pixel value to -1, these pixels are marked. In the subsequent accumulation process, these pixels do not participate, which can reduce the amount of calculation and improve the calculation efficiency without affecting the selection of candidate key points.

[0089] Since image accumulation is to be performed, in order to ensure that the accumulation does not overflow, the present invention performs normalization processing on the grayscale pixel values ​​of the calibration image through step 6, such as dividing the grayscale pixel values ​​of the calibration image by 255, normalizing to a floating point number between 0 and 1, and obtaining a normalized image. Then, through step 7, all normalized images are accumulated to the second matrix. It should be noted that since the pixels corresponding to the value information are not easy to be used to calculate the candidate key points, the positions with pixel values ​​of -1 in the second matrix are not accumulated during the accumulation process. In this way, the amount of calculation can be reduced and the calculation efficiency can be improved without affecting the selection of candidate key points.

[0090] After accumulating the third matrix, extract the candidate key points from the third matrix. In step 8, extract the k points with the largest pixel values ​​and the k points with the smallest pixel values ​​in the third matrix, and generate the set Max_xy by taking the coordinates corresponding to the k points with the largest pixel values, and generate the set Min_xy by taking the coordinates corresponding to the k points with the smallest pixel values. Max_xy and Min_xy are the candidate key points of the current format document image. In order to extract as many candidate key points as possible, the value of k can be relatively large. Since 4 points are required to calculate the matrix, in this embodiment, the value of k is set to 16. It should be noted that k taking 16 is only an implementation in this embodiment and is not intended to be a limitation of the present invention. In other embodiments, the value of k can also be other.

[0091] Different pixel values ​​have different reliability as key points. When selecting key points, it is necessary to search for more critical points first. Therefore, in order to facilitate the search according to reliability in the subsequent key point calculation, the two sets generated in step 8 are arranged in different orders. For example, the points in the set Max_xy are arranged in descending order of pixel value, and the points in the set Min_xy are arranged in ascending order of pixel value.

[0092] After extracting the candidate key points, further key point screening is required to obtain representative points as key points. In step 9, first take the first point in the set Max_xy and the first point in the set Min_xy as key points, add the key points to the set S, and set the number of key points in the set S to Q=2. In step 10, delete the two key points taken out in step 9 from their respective original sets, and then merge the remaining points in the set Max_xy and the set Min_xy to obtain the set MaxMin_xy.

[0093] In step 11, by traversing and calculating the sum of the distances from the points in the set MaxMin_xy to the points in the set S, the point with the largest sum of distances can be obtained. , ,in, represents the Euclidean distance from the i-th point in the set MaxMin_xy to the j-th point in the set S, Indicates the minimum distance between key points. The point with the largest sum of distances can be used as a key point.

[0094] It should be noted that when When , the distance between the coordinates of point i and the existing key point coordinates is too small, which easily causes the calculation of the mapping matrix to be unstable. Therefore, in the present invention, when calculating the key points, set , so the calculated key points are distributed farther away, and the error is smaller when calculating the mapping matrix. In addition, when the distance between the coordinates of point i and the existing key point coordinates is too small, that is, , delete the point from MaxMin_xy so that it no longer participates in the calculation.

[0095] After the key points are obtained, in step 12, the key points are added to the set S and deleted from the set MaxMin_xy, so that repeated calculations of the same points can be avoided in subsequent key point calculations. In addition, in order to facilitate the counting of the number of key points in the set S, the value of the number of key points Q is increased by 1 for each key point added to the set S.

[0096] Each time the Q value is updated, the number of key points Q in the set S must be judged through step 13. When the value of Q is equal to the target number of key points n1, it means that the number of key points has reached the target number, and the key point calculation is terminated, and the current set S is used as the template key point set S2. Otherwise, jump to step 11 to continue the key point calculation. Here, the value of the target number of key points n1 can be set according to the actual situation.

[0097] After obtaining the template key point set S2, in step 14, a deep convolutional neural network is used to predict the key points by regression or direct network output. Since the weights of the key points are different, the present invention obtains the loss function of the key point prediction model based on the predicted key points and the template key points S2: ,in, represents the coordinates of the predicted key points, represents the coordinates of the key points in the template key point set S2, In the present invention, the weight is calculated based on the Fibonacci sequence, and the calculation method is: assuming that the reciprocals of the first M values ​​of the sequence are 1, 1, 1 / 2, 1 / 3, 1 / 5, 1 / 8, 1 / 13, 1 / 21, ..., 1 / f(M), and the sum is F, then the weight of the i-th key point is for The present invention obtains redundant and stable key points through rule calculation, and provides a weighted key point loss function, which can be used for key point detection in the case of honor.

[0098] After obtaining the loss function, the loss function is used to perform iterative calculations to obtain the key point prediction model. Then in step 15, the image to be aligned is substituted into the key point prediction model to calculate the key point set S1 of the image to be aligned. In step 16, the key point set S1 of the image to be aligned and the template key point set S2 are matched to obtain matching point pairs. Since the RANSAC algorithm has strong robustness to noise data and outliers and can effectively estimate parameters, in this embodiment, when matching the key point set S1 of the image to be aligned and the template key point set S2, the RANSAC algorithm is used. The specific algorithm steps can refer to the prior art and will not be repeated here. Of course, other algorithms can also be used when performing key point matching, and the present invention does not specifically limit this.

[0099] After obtaining the matching point pairs, in step 17, the perspective transformation matrix is ​​calculated using the matching point pairs. In this embodiment, the model Calculate the perspective transformation matrix. Where H= represents the homography matrix, that is, the perspective transformation matrix, , , The matching point pairs in the key point set S1 and the template key point set S2. The degree of freedom of H is 8, which has scale equivalence. Therefore, the perspective transformation matrix can be calculated through 4 matching point pairs.

[0100] After obtaining the perspective change matrix, in step 18, the perspective transformation matrix is ​​used to perform perspective transformation on the image to be aligned, and the aligned image can be obtained as , where H is the perspective transformation matrix, I represents the image to be aligned, Represents an image that is aligned with the template passport image after transformation, and the relationship between the key points and key information in the image and the passport image boundary is exactly the same as the relationship between the key points and key information in the template image and the passport image boundary.

[0101] In practical applications, the position of each key in the template image and the vertical distance from the upper left corner of the key's circumscribed rectangular box to the two upper left boundaries of the document can be calibrated, and then the calibration information in the template can be mapped to the current image through an offset. The offset here is the vector formed by the difference between the upper left corner of the document corresponding to the current image and the upper left corner of the image. When extracting information, the key position information obtained based on the above mapping can be used to extract information. In addition, because the position and width and height of the key are fixed, the text line detection results can be segmented and corrected based on this information, which is conducive to further improving the recognition accuracy of the system.

[0102] The pass image alignment method provided by the present invention is based on data statistics and adaptively finds representative key points of the image related to the layout. There are many key points that can be selected and the robustness is higher. By selecting key points with stable structures from candidate key points to solve the transformation matrix, the error can be effectively reduced.

[0103] Optionally, the sample image is preprocessed, specifically: the sample image is cropped, and the sample image is normalized and rotated to obtain a calibration image with a size of width w and height h; the difference between the boundary of the calibration image and the boundary of the target in the calibration image is less than or equal to 1 pixel.

[0104] Specifically, in the present invention, the sample refers to a pass of a certain format, which is a perspective distortion-free sample obtained by scanning or collecting with a passport machine. In order to meet the requirements of subsequent model training, the image needs to be preprocessed. In this embodiment, a calibration image with a size of width w and height h is obtained by cropping, normalizing the image size, and rotating the sample image. In order to facilitate the calculation of the loss function when training the key point prediction model, when the sample image is cropped and normalized, the difference between the boundary of the calibration image and the boundary of the target in the calibration image is set to be less than or equal to 1 pixel.

[0105] Optionally, the set Max_xy includes the coordinates of the k points with the largest pixel values ​​in the third matrix, and the points in the set Max_xy are arranged in descending order of pixel values; the set Min_xy includes the coordinates of the k points with the smallest pixel values ​​in the third matrix, and the points in the set Min_xy are arranged in ascending order of pixel values.

[0106] Specifically, the k points with the largest pixel values ​​and the k points with the smallest pixel values ​​extracted from the third matrix are respectively generated into the set Max_xy and the set Min_xy as candidate key points of the current format document image. In order to facilitate the search for the position of the points when the key points are subsequently screened, in this embodiment, the coordinates corresponding to the candidate key points are respectively stored in the set Min_xy and the set Max_xy. In addition, due to the different sizes of pixel values, the reliability of the pixel values ​​as key points is different. When screening key points, it is necessary to search for more critical points first. Therefore, in order to facilitate the search according to the reliability when calculating the key points later, the two sets generated in the above steps are arranged in different orders, such as the points in the set Max_xy are arranged in the order of pixel values ​​from large to small, and the points in the set Min_xy are arranged in the order of pixel values ​​from small to large.

[0107] Based on the same inventive concept, the present invention also provides a pass image alignment device, Figure 2 For a schematic diagram of a structure of a pass image alignment device provided in an embodiment of the present invention, please refer to Figure 2, the pass image alignment device 100 provided by the embodiment of the present invention includes:

[0108] The acquisition module 101 is used to acquire multiple sample images;

[0109] A preprocessing module 102 is used to preprocess the sample image to obtain a calibration image;

[0110] A marking module 103, used to circumscribe a rectangular frame around the feature information in the calibration image to mark the location of the feature information;

[0111] A generating module 104, configured to generate a first matrix of all zeros, wherein the size of the first matrix is ​​the same as the size of the calibration image;

[0112] A setting module 105, used to set all pixel values ​​in the positions corresponding to the rectangular frame in the first matrix to a taboo state of -1, to obtain a second matrix;

[0113] A normalization module 106 is used to normalize the grayscale pixel values ​​of the calibration image to obtain a normalized image;

[0114] An accumulation module 107 is used to accumulate the normalized image into the second matrix, and positions with pixel values ​​of -1 in the second matrix are not accumulated, so as to obtain a third matrix;

[0115] An acquisition module 108 is used to acquire k points with the largest pixel values ​​and k points with the smallest pixel values ​​in the third matrix, and generate a set Max_xy and a set Min_xy respectively;

[0116] An extraction module 109 is used to extract the first point from the set Max_xy and the set Min_xy respectively and add them to the set S, and set the number of key points in the set S to Q=2;

[0117] A merging module 110 is used to merge the set Max_xy with the set Min_xy after deleting the extracted points from the set Max_xy and the set Min_xy to obtain the set MaxMin_xy;

[0118] A first calculation module 111, used to calculate key points according to the distances from the points in the set MaxMin_xy to the points in the set S;

[0119] An adding module 112 is used to add the key point to the set S, set Q=Q+1, and delete the key point from the set MaxMin_xy;

[0120] The judging module 113 is used to judge whether the number Q of key points in the set S is equal to n1. If not, jump to the key point calculation step; if so, take the set S as the template key point set S2;

[0121] A training module 114 is used to train a key point prediction model using the template key point set S2;

[0122] A second calculation module 115 is used to calculate the key point set of the image to be aligned using the key point prediction model as S1;

[0123] A matching module 116, used to match the key point set S1 of the image to be aligned with the key point set S2 of the template to obtain matching point pairs;

[0124] A third calculation module 117, used to calculate a perspective transformation matrix using matching point pairs;

[0125] The transformation module 118 is used to perform perspective transformation on the image to be aligned using a perspective transformation matrix to obtain an aligned image.

[0126] Specifically, please refer to Figure 2 The passport image alignment device 100 provided in the embodiment of the present invention first collects multiple sample images through the acquisition module 101. The sample here refers to a passport document of a certain format, which is obtained by scanning or collecting with a passport machine without perspective distortion. In order to meet the requirements of subsequent model training, the image needs to be preprocessed. In this embodiment, the collected sample images are preprocessed by the preprocessing module 102, such as cropping, image size normalization, and rotation, to obtain a calibration image.

[0127] Normally, the text information in the certificate can be roughly divided into two parts. One part is the key information, that is, the information that is printed in advance and has a fixed character category and a fixed position; the other part is the value information related to the owner of the certificate, such as the text line information related to the owner of the certificate, that is, the feature information. The character category of this type of information is uncertain, and the length of the text line is different. Since the image appearance characteristics of the corresponding part of the value information are variable, the reference point of this part is not fixed and is not easy to use for calculating candidate key points. Therefore, the present invention uses the marking module 103 to mark the position of this part in the calibration image by enclosing a rectangular frame around the value information, so that the pixels corresponding to this part can be ignored in the subsequent calculation of candidate key points.

[0128] Then, the generation module 104 is used to generate a first matrix with the same size as the calibration image. The first matrix is ​​used as the initial matrix. Therefore, the first matrix is ​​set to be an accumulation matrix of all 0s. At the same time, since the pixels corresponding to the value information are not easy to be used to calculate the candidate key points, these pixels need to be ignored during the image accumulation process. Therefore, the present invention sets all pixel values ​​in the first matrix corresponding to the rectangular frame to -1 through the setting module 105, indicating a taboo state. By setting the pixel value to -1, these pixels are marked. In the subsequent accumulation process, these pixels do not participate, which can reduce the amount of calculation and improve the calculation efficiency without affecting the selection of candidate key points.

[0129] Since image accumulation is to be performed, in order to ensure that the accumulation does not overflow, the present invention normalizes the grayscale pixel values ​​of the calibration image through the normalization module 106, such as dividing the grayscale pixel values ​​of the calibration image by 255, normalizing to a floating point number between 0 and 1, and obtaining a normalized image. Then, all normalized images are accumulated to the second matrix through the accumulation module 107. It should be noted that since the pixels corresponding to the value information are not easy to be used to calculate the candidate key points, the positions with pixel values ​​of -1 in the second matrix are not accumulated during the accumulation process. In this way, the amount of calculation can be reduced and the calculation efficiency can be improved without affecting the selection of candidate key points.

[0130] After accumulating the third matrix, extract the candidate key points from the third matrix, and use the acquisition module 108 to extract the k points with the largest pixel values ​​and the k points with the smallest pixel values ​​in the third matrix, and generate the set Max_xy with the coordinates corresponding to the k points with the largest pixel values, and generate the set Min_xy with the coordinates corresponding to the k points with the smallest pixel values. Max_xy and Min_xy are the candidate key points of the current format document image. In order to extract as many candidate key points as possible, the value of k can be relatively large. Since 4 points are required to calculate the matrix, in this embodiment, the value of k is set to 16. It should be noted that k taking 16 is only an implementation method in this embodiment, and is not intended to be a limitation of the present invention. In other embodiments, the value of k can also be other.

[0131] Different pixel values ​​have different reliability as key points. When selecting key points, it is necessary to search for more critical points first. Therefore, in order to facilitate the search according to reliability in the subsequent key point calculation, the two sets generated above are arranged in different orders. For example, the points in the set Max_xy are arranged in descending order of pixel value, and the points in the set Min_xy are arranged in ascending order of pixel value.

[0132] After extracting the candidate key points, further key point screening is required to obtain representative points as key points. Therefore, firstly, the first point in the set Max_xy and the first point in the set Min_xy are extracted as key points by the extraction module 109, and the key points are added to the set S, and the number of key points in the set S is set to Q=2. Then, the two extracted key points are deleted from their respective original sets by the merging module 110, and then the remaining points in the set Max_xy and the set Min_xy are merged to obtain the set MaxMin_xy.

[0133] The first calculation module 111 can obtain the point with the largest distance sum by traversing and calculating the sum of distances from points in the set MaxMin_xy to each point in the set S. , ,in, represents the Euclidean distance from the i-th point in the set MaxMin_xy to the j-th point in the set S, Indicates the minimum distance between key points. The point with the largest sum of distances can be used as a key point.

[0134] It should be noted that when When , the distance between the coordinates of point i and the existing key point coordinates is too small, which easily causes the calculation of the mapping matrix to be unstable. Therefore, in the present invention, when calculating the key points, set , so the calculated key points are distributed farther away, and the error is smaller when calculating the mapping matrix. In addition, when the distance between the coordinates of point i and the existing key point coordinates is too small, that is, , delete the point from MaxMin_xy so that it no longer participates in the calculation.

[0135] After the key point is obtained, the key point is added to the set S through the adding module 112, and the key point is deleted from the set MaxMin_xy, so that repeated calculation of the same point can be avoided when performing subsequent key point calculations. In addition, in order to facilitate the counting of the number of key points in the set S, each time a key point is added to the set S, the value of the number of key points Q is increased by 1.

[0136] Each time the Q value is updated, the number Q of key points in the set S is judged by the judgment module 113. When the value of Q is equal to the target number of key points n1, it means that the number of key points has reached the target number, and the key point calculation is terminated, and the current set S is used as the template key point set S2. Otherwise, the key point calculation needs to be continued. Here, the value of the target number of key points n1 can be specifically set according to the actual situation.

[0137] After obtaining the template key point set S2, the training module 114 uses the template key point set S2 to train the deep convolutional neural network and predict the key points by regression or direct network output. Since the weights of the key points are different, the present invention obtains the loss function of the key point prediction model based on the predicted key points and the template key points S2: ,in, represents the coordinates of the predicted key points, represents the coordinates of the key points in the template key point set S2, In the present invention, the weight is calculated based on the Fibonacci sequence, and the calculation method is: assuming that the reciprocals of the first M values ​​of the sequence are 1, 1, 1 / 2, 1 / 3, 1 / 5, 1 / 8, 1 / 13, 1 / 21, ..., 1 / f(M), and the sum is F, then the weight of the i-th key point is for The present invention obtains redundant and stable key points through rule calculation, and provides a weighted key point loss function, which can be used for key point detection in the case of honor.

[0138] After obtaining the loss function, the loss function is used to perform iterative calculations to obtain the key point prediction model. Then, through the second calculation module 115, the image to be aligned is substituted into the key point prediction model to calculate the key point set S1 of the image to be aligned. After obtaining the key point set S1 of the image to be aligned, the matching module 116 is used to match the key point set S1 of the image to be aligned and the template key point set S2 to obtain matching point pairs. Since the RANSAC algorithm has strong robustness to noise data and outliers and can effectively estimate parameters, in this embodiment, when matching the key point set S1 of the image to be aligned and the template key point set S2, the RANSAC algorithm is used. The specific algorithm steps can refer to the prior art and will not be repeated here. Of course, other algorithms can also be used when performing key point matching, and the present invention does not specifically limit this.

[0139] After obtaining the matching point pairs, the third calculation module 117 uses the matching point pairs to calculate the perspective transformation matrix. In this embodiment, the model Calculate the perspective transformation matrix. Where H= represents the homography matrix, that is, the perspective transformation matrix, , , The matching point pairs in the key point set S1 and the template key point set S2. The degree of freedom of H is 8, which has scale equivalence. Therefore, the perspective transformation matrix can be calculated through 4 matching point pairs.

[0140] After obtaining the perspective change matrix, the transformation module 118 uses the perspective transformation matrix to perform perspective transformation on the image to be aligned, and the aligned image can be obtained as , where H is the perspective transformation matrix, I represents the image to be aligned, Represents an image that is aligned with the template passport image after transformation, and the relationship between the key points and key information in the image and the passport image boundary is exactly the same as the relationship between the key points and key information in the template image and the passport image boundary.

[0141] In practical applications, the position of each key in the template image and the vertical distance from the upper left corner of the key's circumscribed rectangular box to the two upper left boundaries of the document can be calibrated, and then the calibration information in the template can be mapped to the current image through an offset. The offset here is the vector formed by the difference between the upper left corner of the document corresponding to the current image and the upper left corner of the image. When extracting information, the key position information obtained based on the above mapping can be used to extract information. In addition, because the position and width and height of the key are fixed, the text line detection results can be segmented and corrected based on this information, which is conducive to further improving the recognition accuracy of the system.

[0142] The pass image alignment device provided by the present invention is based on data statistics, adaptively finds representative key points of the image related to the layout, has more selectable key points, and is more robust. By selecting key points with stable structures from candidate key points to solve the transformation matrix, the error can be effectively reduced.

[0143] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0144] Those skilled in the art can understand that the modules in the device in the embodiment can be distributed in the device in the embodiment according to the description of the embodiment, or can be changed accordingly and located in one or more devices different from the embodiment. The modules in the above embodiment can be combined into one module, or can be further divided into multiple sub-modules.

[0145] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A passport image alignment method, characterized in that: include: Acquiring multiple sample images; Preprocessing the sample image to obtain a calibration image; A rectangular frame is drawn around the feature information in the calibration image to mark the position of the feature information; Generate a first matrix of all zeros, where the size of the first matrix is ​​the same as the size of the calibration image; Setting all pixel values ​​in the first matrix corresponding to the rectangular frame to a taboo state of -1 to obtain a second matrix; Normalizing the grayscale pixel values ​​of the calibration image to obtain a normalized image; Accumulate the normalized image into the second matrix, and do not accumulate the positions with pixel values ​​of -1 in the second matrix, to obtain a third matrix; Obtain k points with the largest pixel values ​​and k points with the smallest pixel values ​​in the third matrix, and generate a set Max_xy and a set Min_xy respectively; the points in the set Max_xy are arranged in descending order of pixel values, and the points in the set Min_xy are arranged in ascending order of pixel values; Take the first point from the set Max_xy and the set Min_xy respectively and add them to the set S, and set the number of key points in the set S to Q=2; After deleting the extracted points from the set Max_xy and the set Min_xy, merging the set Max_xy and the set Min_xy to obtain the set MaxMin_xy; Calculate key points based on the distances from points in the set MaxMin_xy to each point in the set S; Add the key point to the set S, set Q=Q+1, and delete the key point from the set MaxMin_xy; Determine whether the number Q of key points in the set S is equal to n1, if not, jump to the step of calculating key points; If so, set S is used as template key point set S2; Using the template key point set S2, training a key point prediction model; The key point set of the image to be aligned is calculated using the key point prediction model as S1; Matching the key point set S1 of the image to be aligned with the key point set S2 of the template to obtain matching point pairs; Calculating a perspective transformation matrix using the matching point pairs; The perspective transformation matrix is ​​used to perform perspective transformation on the image to be aligned to obtain an aligned image.

2. The pass image alignment method according to claim 1, characterized in that: The sample image is preprocessed, specifically: The sample image is cropped, and the size of the sample image is normalized and rotated to obtain a calibration image with a size of width w and height h; the difference between the boundary of the calibration image and the boundary of the target in the calibration image is less than or equal to 1 pixel.

3. The pass image alignment method according to claim 1, characterized in that: The characteristic information includes text lines of information related to the owner of the document.

4. The pass image alignment method according to claim 1, characterized in that: Normalize the grayscale pixel values ​​of the calibration image, specifically: The grayscale pixel value of the calibration image is divided by 255 and normalized to a floating point number between 0 and 1.

5. The pass image alignment method according to claim 1, characterized in that: According to the distances from the points in the set MaxMin_xy to the points in the set S, the key points are calculated, specifically: Traverse and calculate the sum of the distances from the points in the set MaxMin_xy to the points in the set S, and obtain the point with the largest sum of distances d ij >th, where d ij represents the Euclidean distance from the i-th point in the set MaxMin_xy to the j-th point in the set S, and th represents the minimum distance between key points; The point with the largest sum of the distances is taken as the key point.

6. The pass image alignment method according to claim 1, characterized in that: Using the key point set, a key point prediction model is constructed, specifically: Using the template key point set S2 to train the deep convolutional neural network to obtain a loss function; The loss function is used to iteratively calculate the deep convolutional neural network to obtain a key point prediction model.

7. The pass image alignment method according to claim 6, characterized in that: The loss function is Among them, x i ,y i Represents the predicted coordinate point, represents the coordinate point in the template key point set S2, w i represents the weight of the i-th key point.

8. The pass image alignment method according to claim 1, characterized in that: The RANSAC (Random Sampling Consensus) algorithm is used to match the key point set S1 of the image to be aligned and the template key point set S2.

9. A passport image alignment device, characterized in that: include: An acquisition module, used for acquiring multiple sample images; A preprocessing module, used for preprocessing the sample image to obtain a calibration image; A marking module, used for enclosing a rectangular frame around the feature information in the calibration image to mark the position of the feature information; A generating module, used for generating a first matrix of all zeros, wherein the size of the first matrix is ​​the same as the size of the calibration image; A setting module, used for setting all pixel values ​​in the first matrix corresponding to the rectangular frame to a taboo state of -1, to obtain a second matrix; A normalization module, used for normalizing the grayscale pixel values ​​of the calibration image to obtain a normalized image; An accumulation module, used for accumulating the normalized image into the second matrix, and not accumulating the positions with pixel values ​​of -1 in the second matrix, to obtain a third matrix; An acquisition module, used to acquire k points with the largest pixel values ​​and k points with the smallest pixel values ​​in the third matrix, and generate a set Max_xy and a set Min_xy respectively; the points in the set Max_xy are arranged in descending order of pixel values, and the points in the set Min_xy are arranged in ascending order of pixel values; An extraction module, used for taking out the first point from the set Max_xy and the set Min_xy respectively and adding them to the set S, and setting the number of key points in the set S to Q=2; A merging module, used for deleting the points taken out from the set Max_xy and the set Min_xy, and then merging the set Max_xy with the set Min_xy to obtain the set MaxMin_xy; A first calculation module, used for calculating key points according to the distances from the points in the set MaxMin_xy to the points in the set S; An adding module, used for adding the key point to the set S, setting Q=Q+1, and deleting the key point from the set MaxMin_xy; A judging module, used to judge whether the number Q of key points in the set S is equal to n1, if not, jump to the step of calculating key points; If so, set S is used as template key point set S2; A training module, used for training a key point prediction model using the template key point set S2; A second calculation module, used for calculating the key point set S1 of the image to be aligned by using the key point prediction model; A matching module, used for matching the key point set S1 of the image to be aligned with the key point set S2 of the template to obtain matching point pairs; A third calculation module, used to calculate a perspective transformation matrix using the matching point pairs; The transformation module is used to perform perspective transformation on the image to be aligned using the perspective transformation matrix to obtain an aligned image.

Images