Iris and face weight adaptive multi-modal identity recognition method

Abstract

This invention discloses a multimodal recognition method with adaptive iris and face weights. The main steps include: 1) performing multi-level quality assessments on the acquired iris and face images to obtain quality assessment scores for each level of the iris and face images; 2) adjusting the weight ratio of the iris and face based on the quality assessment scores to achieve adaptive optimization of the weighted additive fusion weights of the iris and face modalities at the matching layer. This invention effectively improves the recognition accuracy and precision of traditional iris and face multimodal identity recognition systems, while solving the problem of user identification failure due to partial occlusion of a certain biometric feature of the face or iris, especially the problem of identification failure caused by wearing a mask.

Description

Technical Field

[0001] This invention relates to the field of identity recognition methods, specifically to a multimodal identity recognition method with adaptive iris and face weights. Background Technology

[0002] With mask-wearing becoming commonplace, there is a need for rapid and accurate user identification and management in various situations. On the one hand, traditional facial recognition systems cannot identify users due to facial features being obscured by masks. On the other hand, iris recognition is considered the most ideal biometric feature and has been widely used in various industries. Furthermore, single-feature biometric identification technologies have significant application limitations and cannot meet the current needs of routine epidemic prevention and control. Summary of the Invention

[0003] This invention provides a multimodal identity recognition method with adaptive iris and face weights to overcome the drawback of being unable to recognize a person due to partial occlusion or loss of a certain biometric feature of the face or iris.

[0004] To address the above problems, the technical solution provided by this invention is as follows:

[0005] A multimodal identity recognition method with adaptive iris and face weights includes the following steps:

[0006] Step 1: Obtain the user's video image sequence, collect N sets of iris and face images and number them.

[0007] Step 2: Perform a first-level quality assessment on the N sets of iris and face images collected, quickly filter out images with poor image quality and obtain the first-level quality assessment coefficient K1.

[0008] Step 3: Perform a second-level quality assessment on the N1 group of images that passed the first quality assessment, filter out images that do not have recognizable features, and obtain the second-level quality assessment coefficient K2.

[0009] Step 4: Perform a third-level quality assessment on the N2 groups of images that have passed the second-level quality assessment, accurately locate the recognition features in the images, and obtain the third-level quality assessment coefficient K3.

[0010] Step 5: Perform a fourth-level quality assessment on the N3 groups of images that have passed the third-level quality assessment, and obtain the fourth-level quality assessment coefficient K4.

[0011] Step 6: Calculate the iris image quality score K for each group based on the four-level image quality evaluation coefficients K1-K4. i and face image quality assessment score K f According to K i and K f Adjust the corresponding iris and face recognition weights.

[0012] Furthermore, in step 2, G is set. f If G1, then the face image is a valid image. i >G2 indicates that the iris image is a qualified image. Calculate the first-level quality assessment coefficient for the face image in the qualified image. First-level quality assessment coefficient of iris image The method for calculating the first-level quality assessment coefficient of the face image is as follows: First-level quality assessment coefficient of iris image

[0013] Discard unqualified images, reorganize into N1 groups of iris face images and number them. If there are an odd number of images, discard the one with the smallest gradient value.

[0014] Furthermore, the second-level quality assessment in step 3 includes the localization of the mouth in the face image and the coarse localization of the pupil in the iris image;

[0015] The mouth in the face image is located based on the color attribute characteristics of the mouth. The RGB color space method is used to search within a specified search range to achieve rapid mouth location. The search range is the rectangular area where the mouth is located, which is defined by the human-computer interaction interface.

[0016] The method of coarse pupil localization is to separate the pupil using a binarization method based on the grayscale attribute characteristics of the human eye, thereby quickly determining that the image has iris recognition features;

[0017] Four different combinations were obtained:

[0018] (1) Both the mouth and pupils were found

[0019] (2) The mouth was found, but the pupils were not found.

[0020] (3) The pupil was found, but the mouth was not found.

[0021] (4) No mouth or pupils were found.

[0022] If there are two or more combinations in the N1 group of images, then select the N2 group of iris and face images according to the priority of (1)>(2)>(3)>(4) and number them.

[0023] Furthermore, in step 3, the actual mouth area ratio P is calculated. If the ratio P is between the thresholds P1 and P2, the mouth is considered to be found; otherwise, the mouth is not found, and the system determines that the user is wearing a mask.

[0024] The threshold P1 is the ratio of the minimum actual mouth area S1 to the rectangular area S under normal conditions; the threshold P2 is the ratio of the maximum actual mouth area S2 to the rectangular area S under normal conditions.

[0025] Furthermore, in step 4, the third-level quality assessment coefficient of the face images in the N3 groups of images is calculated. And the third-level quality assessment coefficient of iris images The method for calculating the third-level quality assessment coefficient of the face image is as follows: The third-level quality assessment coefficient of the iris image The rotation of the iris is the angle A between the straight line formed by the two intersection points of the upper and lower eyelids and the horizontal direction. i The rotation of a face is the angle A between the line formed by the centers of the eyes in the face image and the horizontal direction. f , .

[0026] Furthermore, in step 5, the fourth-level image quality evaluation coefficient It is the percentage of the effective facial area. The effective iris region ratio is the ratio of the number of pixels in the skin color region of the face to the total number of pixels in the normalized face image. The effective face region evaluation is performed by using the RGB color space to segment the face for skin color and extract the effective face region.

[0027] The effective iris region assessment uses a binarization method to segment noise such as eyelids and eyelashes in the image of the normalized expanded iris region and extract the effective iris region. The effective iris region assessment is the ratio of the number of pixels in the iris noise region to the total number of pixels in the normalized expanded iris region.

[0028] Furthermore, the quality assessment score Ki of the iris image is calculated as follows: The quality assessment score K of the face image f The calculation method is as follows

[0029] Compared with the prior art, the beneficial effects of the present invention are:

[0030] (1) This invention provides a weighted fusion method for iris and face at the matching layer based on multi-level image quality assessment, which realizes the self-adaptive optimization of weights when iris and face are fused and recognized. The quality of iris and face images is evaluated through a multi-level quality assessment system with different measures, and the iris and face recognition weights are adjusted according to the quality assessment scores, thereby improving the recognition accuracy and precision when both iris and face are effectively recognized.

[0031] (2) When a certain biometric feature of a person’s face or iris is partially obscured, the method of the present invention can quickly determine whether the image has effective identification features and perform quality assessment on the image with effective identification features, and select the image with the highest assessment score for identification. Therefore, the user’s identity can still be accurately identified. It has a wide range of applications and solves the new social needs brought about by user identity recognition. Attached Figure Description

[0032] Figure 1 This is a flowchart of the present invention.

[0033] Figure 2 This is a schematic diagram of the second-level quality assessment.

[0034] Figure 3 This is a schematic diagram of the third-level quality assessment.

[0035] Figure 4 This is a schematic diagram of the fourth level of quality assessment. Detailed Implementation

[0036] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments and accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] Based on the respective requirements of iris and face recognition for images, this method must use four evaluation measures: whether the image clarity is up to standard, whether it has effective recognition features, the rotation degree of recognition features, and the proportion of effective recognition features. Therefore, the multi-level quality evaluation level is at least four levels, and other measures and levels can be added according to specific needs.

[0038] The order of multi-level quality assessment is as follows: for assessment algorithms that are sequential, they are sorted according to their sequential relationship; for assessment algorithms that are not sequential, they are classified according to their time complexity using the shortest job first method. This allows for the exit and re-selection of images for assessment when a certain level of quality assessment fails, thereby improving the efficiency of image assessment.

[0039] See the example. Figure 1 A multimodal identity recognition method with adaptive iris and face weights, specifically including the following steps:

[0040] Step 1: Obtain the user's video image sequence, collect N sets of iris and face images and number them.

[0041] 1.1 The user video image sequence specifically refers to the face video image sequence acquired by the distortion-free high-definition camera and the eye video image sequence acquired by the infrared iris camera.

[0042] Step 2: Perform a first-level quality assessment on the N sets of acquired iris and face images, quickly filtering out images with poor quality and deriving the first-level quality assessment coefficient K1. This step includes the following:

[0043] 2.1 The first image quality assessment specifically refers to the image sharpness assessment.

[0044] 2.2 The image sharpness quality assessment uses the Tenengrad function to calculate the image gradient value G in N groups of images.

[0045] 2.3 Sequentially determine the gradient value G of the face image in N groups of images. f Whether it is greater than the preset face image gradient threshold G1.

[0046] 2.4 Sequentially determine the gradient value G of the iris image in N sets of images. i Whether it is greater than the preset iris image gradient threshold G2.

[0047] 2.5 The preset face image gradient threshold G1 is the minimum gradient value that satisfies face recognition.

[0048] 2.6 The preset iris image gradient threshold G2 is the lowest gradient value that satisfies iris recognition.

[0049] 2.7 When G f If G1, then the face image is a valid image. i If G2 is selected, the iris image is considered a valid image; otherwise, it is considered an invalid image.

[0050] 2.8 Calculate the first-level quality assessment coefficient for face images in qualified images. First-level quality assessment coefficient of iris image

[0051] 2.9 The calculation method for the first-level quality assessment coefficient of face images is as follows: First-level quality assessment coefficient of iris image

[0052] 2.10 Remove unqualified images, reorganize into N1 groups of iris face images and number them. If there are an odd number of images, discard the one with the smallest gradient value.

[0053] 2.11 If all N sets of images fail, return to step 1 and re-acquire images from the video image sequence.

[0054] Step 3: Perform a second-level quality assessment on the N1 group of images that passed the first quality assessment, filter out images that do not have recognizable features, and obtain the second-level quality assessment coefficient K2.

[0055] 3.1 The second level of quality assessment specifically refers to the localization of the mouth in a face image and the coarse localization of the pupil in an iris image.

[0056] 3.2 See also Figure 2 The mouth in the face image is located based on the color attribute characteristics of the mouth, and the RGB color space method is used to search within a specified search range to achieve rapid mouth location.

[0057] 3.3 The search range is the rectangular area where the mouth is located, which is defined by the human-computer interaction interface.

[0058] 3.4 If the user is wearing a mask, the mouth in the face image cannot be located. Combine this with step 3.5 to determine whether the user is wearing a mask.

[0059] 3.5 Calculate the actual mouth area ratio P. If the ratio P is between the thresholds P1 and P2, the mouth is considered to be found; otherwise, the mouth is not found, and the system determines that the user is wearing a mask.

[0060] 3.6 The actual mouth proportion P mentioned is the actual mouth area S. f The ratio of S to the area of ​​the rectangular region

[0061] 3.7 The threshold P1 is the ratio of the minimum actual mouth area S1 to the rectangular area S under normal conditions.

[0062] The threshold P2 mentioned in 3.8 is the ratio of the maximum actual mouth area S2 to the rectangular area S under normal conditions.

[0063] See 3.9 Figure 2 The method for coarse pupil localization is to separate the pupil using a binarization method based on the grayscale attribute characteristics of the human eye, thereby quickly determining that the image has iris recognition features.

[0064] 3.10 From steps 3.2 and 3.9, four different combinations can be obtained: (1) both mouth and pupil are found; (2) mouth is found but pupil is not found; (3) pupil is found but mouth is not found; (4) neither mouth nor pupil is found. The four combinations correspond to different processing methods.

[0065] 3.11 If both the mouth and pupil are found in a set of images, then the images in this set are considered to possess recognizable features, and their second-level quality assessment coefficient K2 is specifically...

[0066] 3.12 If a mouth is found but pupils are not found in a set of images, then the set of images is considered to possess facial recognition features, and only the facial image quality assessment step is performed subsequently. The second-level quality assessment coefficient K2 is specifically...

[0067] 3.13 If a pupil is found but a mouth is not found in a set of images, then the set of images is considered to have iris recognition features, and only the iris image quality assessment step is performed subsequently. The second-level quality assessment coefficient K2 specifically corresponds to...

[0068] 3.14 If neither the pupil nor the mouth is found in a certain set of images, it is considered that the set of images does not have recognition features, and the process returns to step 1 to re-acquire images from the video image sequence.

[0069] 3.15 If the N1 group of images has two or more combinations as described in step 3.10, then select the N2 group of iris and face images and number them according to the priority selection of (1)>(2)>(3)>(4).

[0070] Step 4: Perform a third-level quality assessment on the N2 groups of images that have passed the second-level quality assessment, accurately locate the recognition features in the images, and obtain the third-level quality assessment coefficient K3.

[0071] 4.1 The third-level quality assessment specifically refers to the coarse and fine localization of the eyes in the face image and the assessment of the rotation of the face image, and the fine localization and the assessment of the rotation of the iris in the iris image.

[0072] 4.2 The coarse localization of the eyes in the face image is achieved by combining the position of the mouth in step 3.2 and based on the triangular relationship between the eyes and the mouth.

[0073] 4.3 The precise localization of the eyes in the face image is based on step 4.2, which uses image template matching to accurately detect the position of the eyes.

[0074] 4.4 See also Figure 3 The rotation degree of the face is the angle A between the straight line formed by the center positions of the eyes in the face image and the horizontal direction. f , .

[0075] 4.5 The precise positioning of the iris specifically includes the positioning of the inner and outer edges of the iris and the positioning of the upper and lower eyelids.

[0076] 4.5 The inner and outer edges of the iris in the iris image are located using the Hough transform method to determine the position and radius of the inner and outer center of the iris.

[0077] 4.6 The upper and lower eyelids in the iris image are located using a quadratic curve fitting method.

[0078] See 4.7 Figure 3 The rotation of the iris is the angle A between the straight line formed by the two intersection points of the upper and lower eyelids and the horizontal direction. i .

[0079] 4.8 Calculate the third-level quality assessment coefficient of face images in N3 groups of images. And the third-level quality assessment coefficient of iris images

[0080] 4.9 The method for calculating the third-level quality assessment coefficient of the face image is as follows: The third-level quality assessment coefficient of the iris image

[0081] Step 5: Perform a fourth-level quality assessment on the N3 groups of images that have passed the third-level quality assessment, and obtain the fourth-level quality assessment coefficient K4.

[0082] The fourth level of image quality assessment mentioned in 5.1 specifically refers to the effective face region assessment and the effective iris region assessment.

[0083] 5.2 The fourth-level image quality evaluation coefficient It is the percentage of the effective facial area. It represents the percentage of the effective iris area.

[0084] See 5.3 Figure 4 The effective facial region evaluation is performed by using the RGB color space to segment the face by skin color and extract the effective facial region.

[0085] See 5.4 Figure 4 The effective facial region percentage is the ratio of the number of pixels in the skin color region of the face to the total number of pixels in the normalized facial image.

[0086] See 5.5 Figure 4 The effective iris region evaluation is performed by using a binarization method to segment noise such as eyelids and eyelashes in the image of the normalized unfolded iris region and extract the effective iris region.

[0087] See 5.6 Figure 4 The effective iris region assessment is the ratio of the number of pixels in the iris noise region to the total number of pixels in the normalized expanded iris region.

[0088] Step 6: Calculate the iris image quality score K for each group based on the four-level image quality evaluation coefficients K1-K4. i and face image quality assessment score K f According to K i and K f Adjust the corresponding iris and face recognition weights.

[0089] 6.1 The calculation method for the iris image quality assessment score Ki is as follows:

[0090] 6.2 Face image quality assessment score K f The calculation method is as follows

Claims

1. A multi-modal identity recognition method with iris and face weight self-adaptation, comprising the following steps Step 1: obtaining a user video image sequence, collecting N groups of iris and face images and numbering; Step 2: performing first-level quality assessment on the collected N groups of iris and face images, quickly screening images with poor quality and obtaining a first-level quality assessment coefficient K1; In step 2, set G f G1 is the qualified image of the face image, G i G2 is the qualified image of the iris image The first-level quality evaluation coefficient K of the face image in the qualified image is calculated 1f and the first-level quality evaluation coefficient of the iris image The first-level quality evaluation coefficient K of the face image in the qualified image is calculated The first-level quality evaluation coefficient K of the face image in the qualified image is calculated Eliminating unqualified images, re-composing N1 groups of iris and face images and numbering, and removing the image with the smallest gradient value when there is an odd number of images; Step 3: performing second-level quality assessment on the N1 groups of images passing the first quality assessment, screening out images without recognition features and obtaining a second-level quality assessment coefficient K2; The second-level quality assessment in step 3 includes positioning of the mouth in the face image and rough positioning of the pupil in the iris image; The positioning of the mouth in the face image is achieved by searching in a specified search range according to the color attribute characteristics of the mouth using the RGB color space method; the search range is the rectangular region where the mouth is located, which is defined by the human-computer interaction interface; The rough positioning of the pupil is achieved by separating the pupil according to the gray scale attribute characteristics of the human eye using the binaryzation method, thereby quickly determining that the image has iris recognition features; Four different combinations are obtained: (1) both mouth and pupils found (2) Mouth found, no pupil found (3) Pupil found, mouth not found (4) neither the mouth nor the pupil is found When N1 groups of images appear two or more combinations, select according to the priority of (1)>(2)>(3)>(4), screen out N2 groups of iris and face images and number them; Step 4: performing third-level quality assessment on the N2 groups of images passing the second-level quality assessment, precisely positioning the recognition features in the images and obtaining a third-level quality assessment coefficient K3; In step 4, the third-level quality evaluation coefficient of the face image in the N3 group image is calculated and the third-level quality evaluation coefficient of the iris image The third-level quality evaluation coefficient of the face image is calculated by The third-level quality evaluation coefficient of the iris image Wherein the rotation degree of the iris is the included angle A between the straight line formed by the two intersection points of the upper and lower eyelids and the horizontal direction i ; the rotation degree of the face is the included angle A between the straight line formed by the center positions of the eyes in the face image and the horizontal direction f , Step 5: performing fourth-level quality assessment on the N3 groups of images passing the third-level quality assessment, obtaining a fourth-level quality assessment coefficient K4; In the step 5, the fourth-level image quality evaluation coefficient is a face effective area ratio, is an iris effective area ratio The face effective area assessment is to extract the face effective area by performing skin color segmentation on the face using the RGB color space, and the face effective area ratio is the ratio of the number of face skin color pixels to the total number of normalized face image pixels; The iris effective area assessment is to extract the iris effective area by performing segmentation on the eyelid and eyelash noise in the normalized and expanded iris area image using the binaryzation method, and the iris effective area assessment is the ratio of the number of iris noise area pixels to the total number of normalized and expanded iris area pixels; Step 6: According to the 4-level image quality evaluation coefficients K1-K4, the quality scores K of each group of iris images are calculated respectively i and the quality evaluation scores K of the face images are calculated respectively f According to K i and K f , the corresponding iris and face recognition weights are adjusted.

2. The iris and face weight adaptive multi-modal identity recognition method according to claim 1, characterized in that: In step 3, the actual mouth area ratio P is calculated, if the ratio P is between the threshold values P1 and P2, it is considered that the mouth is found, otherwise the mouth is not found, and the system determines that the user wears a mask; The threshold value P1 is the S ratio of the minimum actual mouth area S1 to the rectangular area in the normal state; the threshold value P2 is the S ratio of the maximum actual mouth area S2 to the rectangular area in the normal state.

3. The iris and face weight adaptive multi-modal identity recognition method according to claim 2, characterized in that: The quality evaluation score Ki of the iris image is calculated in the following manner The quality evaluation score K of the face image is calculated in the following manner f The quality evaluation score Ki of the iris image is calculated in the following manner

Images