A binocular living body detection left-right picture alignment method
By using face detection and ORB feature matching, the difference between image height and width is calculated, which solves the problems of cumbersome calibration and insufficient accuracy of existing binocular image alignment methods in multiple scenarios, and achieves efficient and accurate image alignment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INGENIC SEMICON CO LTD
- Filing Date
- 2022-04-26
- Publication Date
- 2026-07-03
AI Technical Summary
Existing binocular alignment methods require recalibration when applied in multiple scenarios. The calibration process is cumbersome and lacks versatility. Furthermore, existing feature matching methods do not achieve high accuracy in different scenarios.
The image is cropped using a face detection algorithm, and ORB feature matching is used for two filtering steps. The height and width differences between the left and right images are calculated to achieve image alignment in non-calibrated cases.
It improves the accuracy and efficiency of image alignment, simplifies the calibration process, and makes binocular image alignment more accurate and efficient in different scenarios.
Smart Images
Figure CN117011900B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method for aligning left and right images in binocular liveness detection. Background Technology
[0002] With the development of technology, existing technologies in facial liveness detection systems inevitably lead to differences in imaging parameters and assembly precision due to the use of different modules in binocular cameras and the inherent differences between the two cameras. Therefore, binocular alignment is necessary to ensure that both lenses use the same facial image for liveness detection and face comparison. Current binocular alignment methods calibrate each individual camera separately to obtain camera parameter information, then fuse the parameter information from both cameras for binocular alignment, obtaining rotation and translation matrices. These matrices are then mapped and transformed onto the left and right images to achieve binocular image alignment.
[0003] However, existing methods have some drawbacks, as follows:
[0004] Currently, the conventional method for binocular alignment involves calibrating the binocular camera to obtain its parameter information, calculating the relevant rotation and translation matrices, and mapping them onto the binocular images for alignment. However, when applied to multiple scenarios, the calibrated parameter information cannot be shared, requiring recalibration. This calibration process is cumbersome and lacks versatility.
[0005] In addition, the following are commonly used terms in the prior art:
[0006] ORB features: Composed of keypoints and descriptors.
[0007] Feature points: prominent points in an image, such as contour points, bright spots in darker areas, and dark spots in brighter areas.
[0008] Key point: A key point is a small patch of an image, referring to the location of a feature point on the image.
[0009] Descriptor: A descriptor is a mathematical structure that provides rotation-invariant features for different scenarios. It is usually a vector that describes the pixel information around the keypoint. Brute-force matching (BMMatcher): For each feature point in image 1, the distance of its descriptor is measured with all feature points in image 2. The descriptors are then sorted, and the closest one is taken as the matching point. Summary of the Invention
[0010] To address the problems in the prior art, the purpose of this application is to: based on a face detection and recognition algorithm, crop out the face bounding boxes in the left and right images, obtain effective key feature matching points through ORB feature matching, calculate the height difference and width difference between the left and right images based on the key points of the left image, and process the height and width of the right image using this difference, thereby achieving alignment of the left and right images of the binoculars under non-calibration conditions.
[0011] Specifically, this invention provides a method for aligning left and right images in binocular liveness detection. The method normalizes the left and right images obtained from a face detection model. During ORB feature point matching, the width and height differences are calculated by matching feature points. Points with poor matching performance are filtered twice. The poorly performing points are those where the positions of the two feature points differ significantly in the coordinate system. The first filtering occurs after brute-force matching, using a ratio test. A match is considered complete only if the distance between the first and second matches is less than a ratio threshold of 0.6. The second filtering occurs after feature point matching, when calculating the height difference of each matching point on the left and right sides. Matching points with large height differences are removed. These large height differences range from 0 to 15, ensuring that the final calculated height and width differences are not affected by matching points with large differences.
[0012] The method includes the following steps:
[0013] S1. Using a face detection model based on a face detection algorithm, crop the left and right images of the face bounding box: Determine the face bounding box in the left image using the face detection model, and expand the face bounding box in the left image from the center point outwards. The expansion is centered on the face bounding box, and the expansion size is 1.1 times the shortest side of the face bounding box. For the center point, increase the expansion size by half each time in the top, bottom, left, and right directions to increase the amount of facial feature information; At the same time, crop the expanded box in the left and right images to ensure that the effective cropping area is consistent.
[0014] S2.ORB Feature Point Matching:
[0015] When performing ORB feature point matching, the left and right cropped images need to be normalized to ensure that more feature matching points can be found in overly dark images.
[0016] S3. Calculate the difference in height and width between the left and right images:
[0017] The matched feature points are sorted, and feature points with large height differences are removed to make the calculated average width and height differences more accurate.
[0018] S4. Align the left and right images with both eyes.
[0019] The method further includes:
[0020] S1. Crops the left and right images based on the face detection algorithm:
[0021] The face data captured by the left camera (i.e., the face data captured by the left camera in the binocular camera setup) is used to detect the face bounding box position using a face detection model. This face detection model does not have a fixed version number; YOLOv5 can be used. The width, height, and center coordinates of the face bounding box are calculated. Assuming the vertex coordinates of the face bounding box are (50, 50), width 90, height 100, and center coordinates are (95, 100), the face bounding box is expanded horizontally, vertically, and horizontally at the center point. The expansion size is half the maximum side of the face bounding box. The expanded face bounding box is then used to crop the left and right images. The expanded bounding box has a vertex of (40, 45), width 110, and height 110. This area is then cropped. S2. ORB Feature Point Matching:
[0022] First, mean normalization is performed on the left and right cropped images to significantly improve and maintain the brightness of the left and right cropped images.
[0023] The ORB feature matching involves creating an ORB matcher, which is created as follows: Ptr <orb>orbDetector=create(int nfeatures=500,
[0024] (float scaleFactor = 1.2f, int nlevels = 8, int firstLevel = 0); Calculate the key points and descriptors of the left and right cropping images respectively; Perform brute-force matching of the descriptors using MFMacher to eliminate mismatched points; Retain matching feature points that meet the condition that the number of rows in the descriptor after matching must be greater than 2;
[0025] S3. Calculate the difference in height and width between the left and right images:
[0026] For the matched feature points, calculate the height difference between each matched feature point in the left and right cropping images based on the location of the key points. Filter out the matched points with large height differences. The height difference ranges from 0 to 15, and the width difference ranges from 0 to 8. Based on this, calculate the average width of all matched points and set the width difference between the left and right images.
[0027] S4. Align the left and right images of both eyes:
[0028] Based on the left cropped image, the calculated height and width differences are added to the left cropping box to obtain a new cropping box with no difference in the face frame. The right image is then cropped again using the new cropping box. The resulting right cropped image is then aligned with the left cropped image. The aligned left and right images are then applied to stereo vision for liveness detection.
[0029] The normalization process in step S2 includes:
[0030] S2.1, Input image; Calculate pixel mean; Compress mean to a scaling factor of 128, multiply the scaling factor by the input image to obtain...
[0031] S2.2, Original image pixel compression; Difference is calculated between the original image and 128 pixels to obtain the difference result; The mean of the difference is calculated; The mean is compressed to a scaling factor of 30, and this scaling factor is multiplied by the difference result to obtain S2.3, Difference compression; This difference compression is added to the original image pixel compression to obtain...
[0032] S2.4, the new pixel value; then it changes to 0~255, where pixels less than 0 become 0, and pixels greater than 255 become 255, completing the normalization.
[0033] In step S3, calculating the height and width difference between the left and right images further includes: S3.1, calculating the height difference for matching feature points, sorting and comparing them, and removing matching points with large differences.
[0034] S3.2, Calculate the width difference;
[0035] S3.3, Calculate the average of the width difference and the average of the height difference;
[0036] S3.4, Determine the difference between width and height.
[0037] Therefore, the advantage of this application is:
[0038] 1. By normalizing the left and right images obtained from the face detection model cropping, the influence of other transformations on image transformation is eliminated. At the same time, overly dark images are brightened, and more feature points are matched during ORB feature point matching, which improves the difference between the left and right images and makes the alignment of the left and right images more accurate.
[0039] 2. When calculating the width and height differences by matching feature points, points with poor matching performance were filtered twice. The first filtering was performed after brute-force matching, using a ratio test; a match was only considered complete when the distance between the first and second matches was less than a ratio threshold. The second filtering was performed after feature point matching, when calculating the height difference between each matching point, discarding points with large height differences to ensure that the final calculated height and width differences were not affected by points with significant height variations. These two filtering steps improved the accuracy of the difference calculation to some extent.
[0040] 3. Compared with traditional calibration methods, this method calculates the height and width differences of matching feature points, making the alignment of the left and right images of the binocular cameras more accurate, faster, simpler, more convenient and practical. It can be applied to all binocular cameras, so this method is better for aligning binocular images. Attached Figure Description
[0041] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0042] Figure 1 This is a flowchart of the method of the present invention.
[0043] Figure 2 This is a flowchart illustrating an embodiment of the method of the present invention.
[0044] Figure 3 This is a schematic diagram of the image normalization process in the method of the present invention.
[0045] Figure 4(1) is the original darker image in the normalization effect of the method of the present invention.
[0046] Figure 4(2) is a normalization graph showing the effect of normalization processing in the method of the present invention.
[0047] Figure 5 This is a diagram showing the ORB feature matching results in the method of this invention.
[0048] Figure 6 This is a schematic diagram of the ORB feature matching process in the method of this invention.
[0049] Figure 7 This is a flowchart of the method of the present invention for calculating the height and width difference between the left and right images. Detailed Implementation
[0050] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0051] The present invention relates to a method for aligning left and right images in binocular liveness detection, such as... Figure 1 As shown, the main implementation steps of this method are as follows:
[0052] S1. Using a face detection model based on a face detection algorithm, crop the left and right images of the face bounding box: Determine the face bounding box in the left image using the face detection model, and expand the face bounding box in the left image from the center point outwards. The expansion is centered on the face bounding box, and the expansion size is 1.1 times the shortest side of the face bounding box. For the center point, increase the expansion size by half each time in the top, bottom, left, and right directions to increase the amount of facial feature information; At the same time, crop the expanded box in the left and right images to ensure that the effective cropping area is consistent.
[0053] S2.ORB Feature Point Matching:
[0054] When performing ORB feature point matching, the left and right cropped images need to be normalized to ensure that more feature matching points can be found in overly dark images.
[0055] S3. Calculate the difference in height and width between the left and right images:
[0056] The matched feature points are sorted, and feature points with large height differences are removed to make the calculated width and height differences more accurate.
[0057] S4. Align the left and right images with both eyes.
[0058] Step S1 includes: determining the face bounding box in the left image using the YOLOv5 face detection model; expanding the face bounding box from its center point outwards to increase the facial feature information; cropping the expanded bounding box in both the left and right images to ensure consistent effective cropping areas; using the face data captured by the left camera (i.e., the face data captured by the left camera in the binocular camera system), detecting the position of the face bounding box using the YOLOv5 face detection model; calculating the width, height, and center coordinates of the face detection box; assuming the face detection box has vertex coordinates (50, 50), width 90, height 100, and center point coordinates (95, 100); expanding the face bounding box vertically, horizontally, and vertically at the center point, with the expansion size being half the maximum side of the face detection box; cropping the left and right images using the expanded face detection box; the expanded detection box has vertex coordinates (40, 45), width 110, and height 110; and cropping this area.
[0059] Step S2 includes: when performing ORB feature point matching, the mean normalization of the left and right cropped images is required to ensure that the dark images can find more feature matching points, as shown in Figure 4(1) and (2).
[0060] Step S3 includes: sorting the matched feature points and removing feature points with large height difference changes, so that the calculated average width difference and height difference are more accurate.
[0061] like Figure 2 As shown, the specific implementation steps of this method are described below:
[0062] S1. Crops the left and right images based on the face detection algorithm:
[0063] The face data of the left image captured by the left camera is used to detect the position of the face box through the face detection model. The width, height and center coordinates of the face detection box are calculated. The face box is expanded vertically and horizontally at the center point. The expansion size is half of the maximum side of the face detection box. The left and right images are cropped by expanding the face detection box.
[0064] S2.ORB Feature Point Matching:
[0065] First, mean normalization is performed on the left and right cropped images to significantly improve and maintain consistent brightness. The specific implementation flowchart is as follows: Figure 3 As shown, the normalization process includes:
[0066] S2.1, Input image; Calculate pixel mean; Compress mean to a scaling factor of 128, multiply the scaling factor by the input image to obtain...
[0067] S2.2, Original image pixel compression; Difference is calculated between the original image and 128 pixels to obtain the difference result; The mean of the difference is calculated; The mean is compressed to a scaling factor of 30, and this scaling factor is multiplied by the difference result to obtain S2.3, Difference compression; This difference compression is added to the original image pixel compression to obtain...
[0068] S2.4, the new pixel value; then it changes to 0~255, where pixels less than 0 become 0, and pixels greater than 255 become 255, completing the normalization.
[0069] Create an ORB matcher, and the matcher creation method is as follows: Ptr <orb>orbDetector=create(int nfeatures=500,
[0070] float scaleFactor=1.2f, int nlevels=8, int firstLevel=0);
[0071] Calculate the keypoints and descriptors for the left and right cropped images respectively. Perform MFMacher (brute-force matching) on the descriptors, remove mismatched points, and retain matching feature points that meet the condition that the number of rows in the descriptor after matching must be greater than 2. Figure 5 Unconnected matching points are considered mismatched points, while connected matching points are points that meet the conditions. The specific implementation process is as follows: Figure 6 As shown.
[0072] S3. Calculate the difference in height and width between the left and right images:
[0073] For the matched feature points, the height difference between the left and right cropping images is calculated based on the location of the key points. Matching points with large height differences are filtered out. Then, the average width of all matching points is calculated, and the width difference between the left and right images is set. The specific implementation flowchart is as follows: Figure 7 As shown:
[0074] S3.1 For matching feature points, calculate the height difference, sort and compare, and remove matching points with large differences;
[0075] S3.2, Calculate the width difference;
[0076] S3.3, Calculate the average of the width difference and the average of the height difference;
[0077] S3.4, Determine the difference between width and height.
[0078] S4. Align the left and right images of both eyes:
[0079] Based on the left cropping frame, the calculated height and width differences are added to the left cropping frame to obtain a new cropping frame with no difference in the face frame. The right image is then cropped again using this new cropping frame. The resulting right cropping image will be aligned with the left cropping image. The aligned left and right images are then applied to stereo vision for liveness detection.
[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / orb> < / orb>
Claims
1. A method for aligning left and right pictures of binocular liveness detection, characterized in that, The method normalizes the left and right images obtained by cropping the face detection model. During ORB feature point matching, it calculates the width and height differences by matching feature points. Points with poor matching results are filtered twice. Points with poor results are those where the positions of the two feature points differ greatly in the coordinate system. The first filtering is performed after brute-force matching, through a ratio test. A match is considered to be a match if the distance between the first and second matches is less than a ratio threshold of 0.
6. The second filtering is performed after feature point matching is completed. When calculating the height difference of each matching point on the left and right sides, matching points with large height differences are removed. The range of large height differences is 0-15, so that the final calculated height and width differences are not affected by matching points with large differences. The method includes the following steps: S1. Using the face detection algorithm and face detection model, obtain the left and right cropped face bounding boxes: The face bounding box in the left image is determined by a face detection model. The face bounding box in the left image is then expanded outward from its center point. The expansion is centered on the face bounding box and the expansion size is 1.1 times the shortest side of the face bounding box. The expansion size is increased by half each time for the center point, in order to increase the amount of facial feature information. At the same time, the expanded bounding box is cropped in the left and right images to ensure that the effective cropping area is consistent. S2. ORB Feature Point Matching: When performing ORB feature point matching, it is necessary to normalize the mean of the left and right cropped images to ensure that more feature matching points can be found in overly dark images; the normalization process in step S2 includes: S2.1, Input image; Calculate pixel mean; Compress the mean to a scaling factor of 128, multiply this scaling factor by the input image to obtain... S2.2, Original image pixel compression; Difference is performed between the original image and 128 pixels to obtain the difference result; Calculate the mean of the difference; Compress the mean to a scaling factor of 30, and multiply the scaling factor by the difference result to obtain... S2.3, Differential compression, which is added to the original image pixel compression to obtain... S2.4, the new pixel value; then it changes to 0~255, where pixels less than 0 become 0, and pixels greater than 255 become 255, thus completing the normalization; S3. Calculate the difference in height and width between the left and right images: The matched feature points are sorted, and feature points with large height differences are removed to make the calculated average width and height differences more accurate. S4. Align the left and right images with both eyes.
2. The method for binocular live detection and left-right picture alignment according to claim 1, characterized in that, The method further includes: S1. Crops the left and right images based on the face detection algorithm: The face data captured by the left camera (i.e., the face data captured by the left camera in the binocular camera system) is used to detect the position of the face bounding box through a face detection model. The width, height, and center coordinates of the face bounding box are calculated. Assuming the vertex coordinates of the face bounding box are (50, 50), the width is 90, the height is 100, and the center point coordinates are (95, 100), the face bounding box is expanded vertically, horizontally, and horizontally at the center point. The expansion size is half of the maximum side of the face bounding box. The left and right images are then cropped using the expanded face bounding box. The vertex of the expanded bounding box is (40, 45), the width is 110, and the height is 110. This area is then cropped. S2. ORB Feature Point Matching: First, mean normalization is performed on the left and right cropped images to significantly improve and maintain the brightness of the left and right cropped images; The ORB feature matching: create ORB matcher, which is created as follows: Ptr <orb> orbDetector=create(int nfeatures=500,< / orb> (float scaleFactor=1.2f, int nlevels=8, int firstLevel=0); Calculate the key points and descriptors of the left and right cropping images respectively; Perform brute-force matching of the descriptors using MFMacher to eliminate mismatched points; Retain matching feature points that meet the condition that the number of rows of the descriptor after matching must be greater than 2; S3. Calculate the difference in height and width between the left and right images: For the matched feature points, calculate the height difference between each matched feature point in the left and right cropping images based on the location of the key points. Filter out the matched points with large height differences. The height difference ranges from 0 to 15, and the width difference ranges from 0 to 8. Based on this, calculate the average width of all matched points and set the width difference between the left and right images. S4. Align the left and right images of both eyes: Based on the left cropped image, the calculated height and width differences are added to the left cropping box to obtain a new cropping box with no difference in the face frame. The right image is then cropped again using the new cropping box. The resulting right cropped image is then aligned with the left cropped image. The aligned left and right images are then applied to stereo vision for liveness detection.
3. The method of claim 2, wherein, In step S3, calculating the height and width difference between the left and right images further includes: S3.1 For matching feature points, calculate the height difference, sort and compare, and remove matching points with large differences; S3.2, Calculate the width difference; S3.3, Calculate the average of the width difference and the average of the height difference; S3.4, Determine the difference between width and height.