Image processing device, image processing method, and program

Abstract

To provide an image processing device, image processing method, and program, which enable appropriate determination of the presence or absence of blinking to suppress unauthorized authentication breakthrough in facial authentication.SOLUTION: An image processing device is provided, comprising: an editing unit configured to cut out first and second determination images, including expected positions of at least either of pupils and irises based on organ points representing outlines of the eyes of an authentication target, from a first image and a second image including eyes of the authentication target captured in chronological order; an acquisition unit for acquiring pixel values of the first determination image and pixel values of the second determination image; and a determination unit configured to determine the presence or absence of blinking by the authentication target on the basis of the pixel values of the first determination image and the pixel values of the second determination image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a program.

Background Art

[0002] Conventionally, as one of the biometric authentication technologies, a face authentication technology is known in which a face image of a person to be authenticated is acquired and compared with a face image registered in advance to authenticate whether the person is the legitimate user. In order to prevent unauthorized breakthrough of face authentication, for example, a blink detection function for determining the presence or absence of blinks by detecting the movement of feature points around the eyes, as described in Non-Patent Document 1, may be added to the face authentication device.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the blink detection method disclosed in Non-Patent Document 1, the frame of the glasses may be misrecognized as an eye. Also, when reflected light is reflected in the lens of the glasses, etc., the shape of the eye cannot be correctly captured, making it difficult to set feature points, so there are cases where blink detection cannot be performed, and it may not contribute to suppressing unauthorized breakthrough in face authentication.

[0005] The present invention was made to solve the above problems, and aims to provide an image processing device, an image processing method, and a program that can suitably determine whether or not blinking occurs in order to suppress fraudulent authentication bypass in facial recognition. [Means for solving the problem]

[0006] To achieve the above objective, the image processing apparatus of the present invention is The editorial department extracts first and second discrimination images from first and second images, which include the eye to be authenticated and are captured in chronological order, based on organ points representing the external shape of the eye to be authenticated, including the assumed position of at least one of the pupil and iris. An acquisition unit that acquires the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, A discrimination unit that determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, Equipped with picture, The editorial department extracts a third discrimination image, including the assumed position based on the organ point, from a third image which is the image taken immediately before the first image and includes the eye to be authenticated. The acquisition unit acquires the pixel values ​​of the third discrimination image, The discrimination unit determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image, the pixel values ​​of the second discrimination image, and the pixel values ​​of the third discrimination image. The acquisition unit extracts the lowest pixel value from among the pixels arranged in a horizontal line approximately parallel to the baseline connecting the inner and outer corners of the eye to be authenticated, for all pixels of each of the first, second, and third discrimination images, and calculates the average value of the lowest pixel value for each of the first, second, and third discrimination images. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing device, an image processing method, and a program that suitably determine whether or not a person blinks in order to suppress fraudulent authentication bypass in facial recognition. [Brief explanation of the drawing]

[0008] [Figure 1] This flowchart shows an overview of the processing performed by the image processing apparatus according to an embodiment of the present invention. [Figure 2] This figure illustrates the results of organ point detection performed by an image processing device according to an embodiment of the present invention. [Figure 3](a) is a diagram showing an enlarged view of the vicinity of the right eye of the face image in FIG. 2, and (b) is a diagram for explaining the cutout rectangle determination process executed by the image processing apparatus according to an embodiment of the present invention using (a). [Figure 4] It is a diagram showing a discrimination image obtained by affine transformation based on the reference point in FIG. 3(b). [Figure 5] It is a block diagram of an image processing apparatus according to an embodiment of the present invention. [Figure 6] It is a flowchart of the blink detection incorporated face authentication process according to an embodiment of the present invention. [Figure 7] It is a flowchart of the discrimination image generation process according to an embodiment of the present invention. [Figure 8] It is a flowchart of the cutout rectangle calculation process according to an embodiment of the present invention. [Figure 9] It is a flowchart of the blink discrimination process according to an embodiment of the present invention. [Figure 10] It is a flowchart of the eye open / close state discrimination process according to an embodiment of the present invention. [Figure 11] It is a diagram illustrating the discrimination result of eye open / close by the image processing apparatus according to an embodiment of the present invention. [Embodiments for Carrying Out the Invention]

[0009] The image processing apparatus according to an embodiment of the present invention will be described in detail below with reference to the drawings.

[0010] [Overview of Image Processing Apparatus] The image processing apparatus according to an embodiment of the present invention performs blink detection on an authentication target in a management target area (security area) where only limited staff such as in commercial facilities are allowed to enter, in order to perform face authentication at the time of entry and exit and enhance the reliability of face authentication. The image processing apparatus determines whether the authentication target has blinked, thereby confirming that the authentication target, which is the subject of face authentication, is a living body, and suppressing unauthorized authentication breakthroughs in face authentication.

[0011] Figure 1 is a flowchart outlining the blink detection performed by the image processing device. First, face detection is performed on image P1, which is one frame of a video containing the eyes of face F of the subject to be authenticated T, as shown in Figure 2 (step S100), and organ point V is detected on the detected face F (step S101). Next, in step S102, the position of the eyes of the subject to be authenticated T is determined based on the detected organ point V that represents the outline of the eye. After determining the position of the eyes, in step S103, as shown in Figures 3(a) and (b), reference points K1 to K4 are set around the pupil and iris (hereinafter referred to as the colored part of the eyeball), and a rectangular image composed of reference points K1 to K4 is cropped (the setting method will be described later). After that, an affine transformation is performed to generate a square discrimination image D1 of L pixels × L pixels, as shown in Figure 4. Similarly, discrimination images D2 to Dn are generated for the next frames, images P2, ..., Pn, which are captured in chronological order following image P1. The process proceeds to step S104, where the smallest pixel value in a horizontal row of pixels in the discrimination image D1 is extracted. Then, the process moves one pixel above or below this value and similarly extracts the smallest pixel value in a horizontal row of pixels. This is repeated L times until all pixels in the discrimination image D1 are searched. The average of these minimum values ​​is calculated. The same process is performed for discrimination images D2-Dn, calculating the average of the minimum values ​​for each image. The process proceeds to step S105, where it is determined whether the ratio of the average value G of the minimum values ​​in discrimination image Dn to the historical average value H of the minimum values ​​of the seven discrimination images prior to discrimination image Dn is greater than the closed-eye threshold described later. If the eyes are closed, the (high) pixel values ​​of the eyelids are extracted instead of the (low) pixel values ​​of the colored part of the eyeball, and the average value G becomes larger. Based on this result, the image processing device (details described later) determines the open / closed state of the eyes of the authentication target T and whether or not blinking has occurred. If blinking is detected, the authentication target T is determined to be a living organism and facial recognition is performed. If it is determined that there is no blinking, the authentication target T is not a biological entity and it is determined that an unauthorized authentication bypass has been attempted, and facial authentication of the authentication target T is not performed. In this way, the image processing device 10 can suppress unauthorized authentication bypasses in facial authentication by using the pixel values ​​of the assumed positions of the colored parts of the eyeballs of the authentication target T's face to detect whether or not the eyes of the authentication target T are opening and closing and blinking.

[0012] [Configuration of Image Processing Apparatus] The configuration of the image processing apparatus 10 will be described below. As shown in FIG. 5, the image processing apparatus 10 includes a control unit 20 that performs blink detection and face authentication of the authentication target T, a storage unit 30 that stores an average value G, a face image of the authentication target T, etc., an imaging unit 40 that captures a face image of the authentication target T, etc., a communication unit 50 that communicates with an entrance / exit gate, etc., a display unit 60 that displays an authentication result, etc., and an input unit 70 that inputs information, etc.

[0013] The control unit 20 is composed of at least one processor such as a CPU (Central Processing Unit), and by executing programs and the like stored in the storage unit 30, it realizes the functions of each unit (image acquisition unit 21, detection unit 22, discrimination unit 23, editing unit 24, pixel value acquisition unit 25, face authentication unit 26) described later. Further, the control unit 20 has a clock (not shown) and can acquire the current date and time, count the elapsed time, etc.

[0014] The storage unit 30 is composed of at least one memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and part or all of the ROM is composed of an electrically rewritable memory (such as a flash memory). Functionally, the storage unit 30 has an image storage unit 31, a face image storage unit 32, an average value storage unit 33, a discrimination image storage unit 34, a face authentication image storage unit 35, and a setting storage unit 36. Programs executed by the CPU of the control unit 20 and data necessary in advance for executing the programs are stored in the ROM. Data created or changed during program execution is stored in the RAM.

[0015] The image storage unit 31 stores images P1, P2, P3, etc. captured by the imaging unit 40 in time series.

[0016] The face image storage unit 32 stores images P1, P2, P3, etc. with fiducial points attached.

[0017] The average value storage unit 33 stores the average value G, past average value H, etc., acquired by the pixel value acquisition unit 25.

[0018] The discrimination image storage unit 34 stores discrimination images D1, D2, D3, etc.

[0019] The facial recognition image storage unit 35 stores facial images of staff members and others who are permitted to enter the managed area, which are used by the facial recognition unit 26 for facial recognition. The system administrator can register new facial images and delete old ones when there are changes in the facial image, such as when a staff member needs glasses, when they no longer need glasses because they use contact lenses, or when they cut their hair short, or when a staff member permitted to enter the managed area retires or joins the company.

[0020] The setting storage unit 36 ​​stores various settings used by the image processing device 10. These include settings used in the blink detection-based face authentication process, such as the discrimination period Q which indicates the length of time for performing blink detection, the closed-eye threshold which determines whether the eyes of the authentication target T are closed or not, the movement threshold, outliers, and the frame rate, which will be described later. The system administrator can change these settings via the input unit 70 as needed.

[0021] The imaging unit 40 includes an imaging device 41 that images the object to be authenticated T, and a drive device 42 that changes the orientation of the imaging device 41.

[0022] The imaging device 41 is equipped with a CMOS (Complementary Metal Oxide Semiconductor) camera. The imaging device 41 captures the object to be authenticated T at a frame rate of f=30fps and generates images P1, P2, etc.

[0023] The drive unit 42 changes the orientation of the imaging device 41. This changes the imaging range and imaging angle of the imaging device 41, allowing for appropriate imaging of the face of the person to be authenticated T.

[0024] The communication unit 50 has a communication device 51, which is a module for communicating with external devices such as entrance / exit gates. When communicating with external devices, the communication device 51 is a wireless module including an antenna. For example, the communication device 51 is a wireless module for short-range wireless communication based on Bluetooth®. Through the communication unit 50, the image processing device 10 controls entrance / exit gates, etc., to manage the entry and exit of the authentication target T and the security of the managed area.

[0025] The display unit 60 includes a display device 61 made of a liquid crystal display panel (LCD).

[0026] The display device 61 is composed of thin-film transistors (TFTs), liquid crystals, or organic EL displays, etc. The display device 61 displays certification results, etc.

[0027] The input unit 70 has an input device 71 for the authenticated user T to input various information to the image processing device 10. The input unit 70 also has a keyboard, mouse, etc. for administrators, authenticated users T, etc. to input information.

[0028] Next, the functional configuration of the control unit 20 of the image processing device 10 will be described. The control unit 20 implements the functions of the image acquisition unit 21, detection unit 22, discrimination unit 23, editing unit 24, pixel value acquisition unit 25, and face recognition unit 26, and performs face recognition biometric detection entry / exit processing, etc., which will be described later.

[0029] The image acquisition unit 21 causes the imaging unit 40 to capture an image within a range pre-set in the image processing device 10 or under user-defined conditions, and acquires a video including images P1, P2, ..., Pn, etc. The image acquisition unit 21 transmits the acquired images to the detection unit 22. The image acquisition unit 21 also stores the acquired images P1, P2, ..., Pn, etc. in the image storage unit 31.

[0030] The detection unit 22 performs face detection and organ point detection in images P1, P2, ..., Pn. The detection unit 22 performs face detection and organ point detection using a known InsightFace or the like. The detection unit 22 stores the face image with the detected organ points V in the face image storage unit 32.

[0031] The discrimination unit 23 determines whether the eyes of the subject T to be authenticated in each image are open or closed. The discrimination unit 23 also determines whether the subject T to be authenticated in each image is blinking or not. Furthermore, the discrimination unit 23 determines whether the amount of movement from the organ point coordinates of the inner and outer corners of the eye in image Pn-1 to the organ point coordinates of the inner and outer corners of the eye in image Pn is within a movement threshold.

[0032] The editing unit 24 extracts the discrimination images from images P1 to Pn. It also converts the discrimination images D1 to Dn to grayscale and stores them in the discrimination image storage unit 34. The editing unit 24 also calculates the positions and lengths of the perpendicular lines J1 and J2 relative to the baseline W connecting the inner corner E1 and outer corner E2 of the eye. The editing unit 24 places the reference points K1, K2, K3, and K4.

[0033] The pixel value acquisition unit (acquisition unit) 25 acquires the pixel values ​​of the discrimination images stored in the discrimination image storage unit 34. For each of the discrimination images D1 to Dn, it repeatedly extracts the lowest pixel value from the pixels arranged in a row in the horizontal direction (along the baseline W) and then in the vertical direction, obtaining the average value of the lowest pixel values ​​in the discrimination images. The pixel value acquisition unit 25 also excludes outliers with pixel values ​​of 180 or more from the pixel values ​​and obtains the average value. The pixel value acquisition unit 25 obtains the past average value H by averaging the average values ​​G of each of the discrimination images Dn-8 to Dn-1.

[0034] The facial recognition unit 26 performs facial recognition using the facial image stored in the facial image storage unit 32. The facial recognition unit 26 recognizes the positions of the eyes, nose, mouth, etc., in the facial image and compares them with the facial image in the facial recognition image storage unit 35 to authenticate that it is a specific person, performing known 2D facial recognition.

[0035] The functional configuration of the control unit 20 has been described above. Below, the blink detection-enabled facial recognition process performed by the image processing device 10 will be explained using a flowchart.

[0036] (Blink detection embedded facial recognition processing) Referring to Figure 6, the flow of the blink detection-integrated facial recognition process is explained. The blink detection-integrated facial recognition process can suppress unauthorized authentication bypasses of facial recognition. The image processing device 10 performs access control by combining facial recognition and blink detection. Before performing facial recognition, blink detection uses the image P1, which is also used in facial recognition, to determine whether the person to be authenticated T is a living being. If the person to be authenticated T is not determined to be a living being, facial recognition is not performed, and the person to be authenticated T is not permitted to enter or exit.

[0037] First, in step S1, the image processing device 10 starts capturing a video of the face of the person to be authenticated T, with the imaging unit 40 set to a frame rate of f=15fps and a discrimination period Q=30 seconds. Let n=1.

[0038] The image acquisition unit 21 acquires image P1 from the video in which the face of the person to be authenticated T is captured, and stores it in the image storage unit 31 (step S2).

[0039] The process proceeds to step S3, where the detection unit 22 performs face detection using the image P1 stored in the image storage unit 31.

[0040] The process proceeds to step S4, where the discrimination image generation process is performed. As will be described in detail later, in the discrimination image generation process, organ points are detected from the face detected in image P1, and a discrimination image D1 is generated that includes the assumed position of the colored part of the eyeball.

[0041] Step S5 involves blink detection. As will be explained in detail later, the blink detection process determines whether or not the authenticated subject T blinked. After this determination, the process proceeds to step S6.

[0042] In step S6, if the eyes of the subject T blink (step S6: Yes), proceed to step S7; if the eyes of the subject T do not blink (step S6: No), proceed to step S11.

[0043] In step S7, the facial recognition unit 26 compares the facial image of the person to be recognized T shown in image P1 with the facial image stored in the facial recognition image storage unit 35.

[0044] The process proceeds to step S8. If facial authentication of the subject T is successful (step S8: Yes), the display unit 60 displays "Authentication successful" (step S9) and the process terminates. If facial authentication of the subject T fails (step S8: No), the display unit 60 displays "Authentication failed" (step S10) and the process terminates.

[0045] In step S11, it is determined whether the imaging time of the object to be authenticated T exceeds the discrimination period Q. If the imaging time of the object to be authenticated T exceeds the discrimination period Q (step S11: Yes), it is determined that the object to be authenticated T is not a living organism because no opening or closing of the eyes or blinking of the eyes can be confirmed even after the discrimination period Q has passed, and the display unit 60 displays "Invalid" (step S12) and the process ends. If the imaging time of the object to be authenticated T is less than or equal to the discrimination period Q (step S11: No), the process proceeds to step S13, where n=n+1, and the process returns to step S2 to acquire image P2.

[0046] The blink detection embedded facial recognition process has been described above. Next, the discrimination image generation process performed in step S4 of the blink detection embedded facial recognition process will be explained using Figure 7.

[0047] First, in step S20, the detection unit 22 obtains the organ point coordinates of the image Pn in which the face of the person to be authenticated T is shown.

[0048] Next, we proceed to step S21, where the cropped rectangle editing process is performed. As will be explained in detail later, the rectangular image to be cropped as the image for discrimination is edited.

[0049] Proceed to step S22. If n=1 (step S22: Yes), proceed to step S23. If n is not 1 (step S22: No), proceed to step S24.

[0050] In step S23, based on the cropped data obtained in the cropping rectangle editing process in step S21, a rectangular region that will be used as the discrimination image is cropped from image Pn, and the process ends.

[0051] In step S24, the coordinates of the rectangle extracted from image Pn-1 are obtained (step S24).

[0052] The process proceeds to step S25, where the discrimination unit 23 determines whether the amount of movement from the organ point coordinates of the inner and outer corners of the eye in image Pn-1 to the organ point coordinates of the inner and outer corners of the eye in image Pn is within the movement threshold. If the amount of movement from the organ point coordinates of the inner and outer corners of the eye in image Pn to the organ point coordinates of the inner and outer corners of the eye in image Pn is within the movement threshold (step S25: Yes), the editing unit 24 extracts a rectangular area which is the average value of the rectangle coordinates in image Pn-1 and the rectangle coordinates in image Pn (step S26), and then terminates. If the amount of movement from the organ point coordinates of the inner and outer corners of the eye in image Pn to the organ point coordinates of the inner and outer corners of the eye in image Pn exceeds the movement threshold (step S25: No), the rectangular area in image Pn is extracted (step S23), and then terminates.

[0053] The image generation process for discrimination has been explained above. Next, the cropping rectangle editing process performed in step S21 of the image generation process for discrimination will be explained using Figure 8.

[0054] First, in step S40, the editorial department 24 places vertical lines relative to the baseline W connecting the inner corner E1 and the outer corner E2 of the eye. As shown in Figure 3(a), a vertical line J1 divides the baseline W into an inner corner E1 side and an outer corner E2 side in a 1:3 ratio, and a vertical line J2 divides the baseline W into an inner corner E1 side and an outer corner E2 side in a 3:1 ratio.

[0055] Next, in step S41, the editorial department 24 sets the lengths of the vertical lines J1 and J2 to be the same as the length of the baseline W.

[0056] Proceed to step S42, and editorial staff 24 adjusts the vertical lines J1 and J2 so that the upper eyelid side is 6 and the lower eyelid side is 4, with the baseline W as the boundary.

[0057] In step S43, the editorial staff 24 divides vertical lines J1 and J2 into 33 sections each with dividing lines B parallel to the baseline W, and places reference points K1, K2, K3, and K4 at the 9th intersection of dividing lines B and vertical lines J1 and J2, both vertically and horizontally. Figure 3(b) illustrates the positional relationship between vertical lines J1 and J2, the 9 dividing lines B on the upper eyelid side (the remaining dividing lines B are omitted for clarity), and reference points K1 and K2.

[0058] Proceeding to step S44, editorial staff 24 adjusts the rectangle with the four reference points K1, K2, K3, and K4 as vertices into an L×L square using an affine transformation, and then rotates the base formed by K3 and K4 so that it is parallel to the base of image Pn, thus completing the process.

[0059] The above explains the rectangular cropping editing process. Next, we will explain the blink detection process performed in step S5 of the blink detection-enabled face recognition process using Figure 9.

[0060] First, in step S50, the eye open / closed state determination process is performed. As will be described in detail later, the determination unit 23 determines in the image Pn whether the eyes of the subject T are open or closed.

[0061] Next, the process proceeds to step S51. If it is determined that the eyes of the subject T being authenticated are closed in image Pn (step S51: Yes), the process proceeds to step S52. If it is not determined that the eyes of the subject T being authenticated are closed (step S51: No), the process terminates.

[0062] In step S52, it is determined whether or not the eyes of the subject T being authenticated were open in image Pn-1. If the eyes of the subject T being authenticated were open in image Pn-1 (step S52: Yes), the discrimination unit 23 determines that blinking occurred (step S53) and terminates. If the eyes of the subject T being authenticated were not open in image Pn-1 (step S52: No), the process terminates.

[0063] The blink detection process has been explained above. Next, the eye opening / closing state detection process, which is performed in step S50 of the blink detection process, will be explained using Figure 10.

[0064] First, in step S60, the editorial department 24 converts the discrimination image Dn to grayscale.

[0065] Next, in step S61, m=1, and proceed to step S62, where the lowest pixel value is extracted from a row of pixels arranged horizontally in the L×L pixel discrimination image Dn (along the line segment connecting the reference points K1 and K2 (or reference points K3 and K4) in Figure 4, or along the line segment connecting the inner corner E1 and outer corner E2 of the eye).

[0066] The process proceeds to step S63 to determine whether m=L or not. If m=L (step S63: Yes), the average value G of the discrimination image Dn is obtained by excluding outliers from the pixel values ​​extracted for m=1 to L (step S65). If m=L is not the case (step S63: No), m=m+1 (step S64) and the process returns to step S62. In this embodiment, outliers are defined as 180 or greater.

[0067] The process proceeds to step S66. If m < 8, it is determined whether the average value G is less than the allowable value. If m < 8 and the average value G is less than the allowable value (step S66: Yes), the process ends. If m is 8 or greater, or if m < 8 but the average value G is greater than or equal to the allowable value (step S66: No), the process proceeds to step S67. The allowable value is used in step S67 to prevent the average value G of images with closed eyes from being included in the past average value H, which would reduce the accuracy of eye open / closed detection. In this embodiment, the allowable value is the average value G of the discrimination images acquired from the face images stored in the face recognition image storage unit 35 multiplied by 1.2.

[0068] The process proceeds to step S67, where the pixel value acquisition unit 25 acquires the past average value H, which is the average of the average values ​​G of the seven discrimination images prior to discrimination image Dn, stored in the average value storage unit 33.

[0069] In step S68, it is determined whether the ratio G / H of the average value G to the past average value H is 1.33 or greater, which is the closed-eye threshold (see Figure 11). If G / H is 1.33 or greater (step S68: Yes), the discrimination unit 23 determines that the eyes of the subject T are closed in image Pn (step S69) and terminates. If G / H is less than 1.33 (step S68: No), the discrimination unit 23 determines that the eyes of the subject T are open in image Pn (step S70), incorporates the average value G of image Pn into the average value H, excludes the average value G of the oldest image (image Pn-8 or earlier) from those incorporated into the average value H, stores the new average value H in the average value storage unit 33 (step S71) and terminates.

[0070] The above explains the process for determining the open / closed state of the eyes.

[0071] In this way, through the blink detection-integrated face recognition process, the image processing device 10 extracts discrimination images Dn-8 to Dn-1 and discrimination image Dn, which include the assumed position of the colored part of the eyeball, from images Pn-8 to Pn-1 and image Pn, which include the eyes of the person to be authenticated T, based on the positions of the inner corner E1 and outer corner E2 of the eye, and obtains pixel values ​​n-8 to n-1 and pixel value n. Subsequently, based on the pixel values ​​n-8 to n-1 and pixel value n, it determines whether the eyes of the person to be authenticated T are open or closed and whether or not they are blinking. Since the discrimination images are extracted based on the positions of the inner corner E1 and outer corner E2 of the eye, which are easy to detect as organ points, even if the person to be authenticated T is wearing glasses or the like, it is possible to extract discrimination images that are suitable for determining whether or not the eyes are open or closed and whether or not they are blinking. Therefore, it is possible to detect and suppress fraudulent authentication bypasses of face recognition.

[0072] Furthermore, the image processing device 10 repeatedly extracts the lowest pixel value from a row of pixels arranged horizontally along the baseline W connecting the inner corner E1 and outer corner E2 of the eye, for each of the discrimination images Dn-8 to Dn-1 and discrimination image Dn, and calculates the average value of the lowest pixel value for each discrimination image. Therefore, even if a part of the colored part of the eyeball is obscured by, for example, illumination reflected in the glasses, the possibility of extracting the pixel value of the colored part of the eyeball can be increased, and the presence or absence of blinking can be suitably determined.

[0073] Furthermore, since the image processing device 10 calculates and uses the historical average value H of the lowest pixel value of the discrimination images Dn-8 to Dn-1, even if there is a temporary change in ambient brightness, the influence on the historical average value H can be kept to a minimum, and the presence or absence of blinking can be suitably determined.

[0074] The image processing device 10 calculates the ratio of the average value G of the discrimination image Dn to the historical average value H of the lowest pixel value in discrimination images Dn-8 to Dn-1. If this ratio is greater than the closed-eye threshold, it determines that the eyes of the subject T are closed in image Pn. Since the open / closed state of the eyes is determined by the ratio of past frames to the current frame, individual differences (differences in eye size) can be absorbed, and the presence or absence of blinking can be suitably determined (see Figure 11).

[0075] Furthermore, the image processing device 10 calculates the average value by excluding outliers among the pixels whose pixel value is 180 or higher. Therefore, if there is a reflection of lighting on glasses or the like, the pixels with the reflection can be excluded. Consequently, the presence or absence of blinking can be appropriately determined.

[0076] The image processing device 10 calculates pixel values ​​after converting the discrimination image to grayscale, thereby reducing the computation load on pixel values ​​and speeding up processing. Furthermore, by converting the extracted discrimination image to a square shape using affine transformation, the computation load on pixel values ​​is also reduced, and processing speed can be increased.

[0077] Furthermore, the image processing device 10 sets the length of the discrimination image in the direction along the baseline W to half the length of the baseline W, so that the midpoint coincides with the midpoint of the baseline W, and sets the length in the direction perpendicular to the baseline to 15 / 33 of the length of the baseline W. This allows the colored part of the eyeball to be suitably included in the discrimination image, and the average value to be calculated more accurately. Therefore, the presence or absence of blinking can be suitably determined.

[0078] The image processing device 10 determines whether or not the subject T blinks by determining whether the subject T blinks in image Pn and in image Pn-1, thereby suppressing unauthorized authentication bypasses such as the temporary insertion of photographs with closed eyes or still images.

[0079] (modified version) In the above embodiment, the image processing device 10 calculated the lowest pixel value for each row of pixels arranged in a line along the baseline W, but it may also calculate the lowest pixel value for each row of pixels arranged in a line perpendicular to the baseline W.

[0080] Furthermore, in the above embodiment, the image processing device 10 had an imaging unit 40, a display unit 60, etc., but these may be replaced by external devices.

[0081] In the above embodiment, the image processing device 10 performed blink detection and facial recognition, but facial recognition may be performed by an external device.

[0082] In the above embodiment, the closed-eye threshold was set to 1.33 in the image processing device 10, but a value of around 1.2 to 1.4 is preferable, and it is advisable to set an appropriate threshold according to various conditions such as the distance between the authentication target T and the imaging unit 40, and the illumination around the imaging unit 40. In addition, it is advisable to appropriately set the discrimination period Q, movement threshold, etc.

[0083] In the above embodiment, the average value G was compared with the past average value H of seven discrimination images. However, for example, to improve processing speed, it may be compared with the past average value of discrimination image Dn-1, or for example, to improve accuracy, it may be compared with the past average value of 10 discrimination images. The number of discrimination images used to calculate the past average value may be increased or decreased as appropriate.

[0084] Furthermore, in the above embodiment, the discrimination image included the assumed positions of the pupil and iris as the colored portion of the eyeball, but the discrimination image may include the assumed position of at least one of the pupil and iris as the colored portion of the eyeball. In addition, although the discrimination image was cropped based on the positions of the inner corner E1 and outer corner E2 of the eye of the subject to be authenticated T, it may also be cropped based on organ points representing the outer shape of the eye of the subject to be authenticated T, organ points representing the outer shape of the iris, organ points representing the eyebrows, etc.

[0085] In the above embodiment, the image processing device 10 treated pixels with a pixel value of 180 or higher as outliers. However, for example, pixels that are 20 percent or more higher than the pixel value indicating the skin color of the object to be authenticated T may be treated as outliers, or pixels that are 20 or more higher than the pixel value indicating the skin color of the object to be authenticated T may be treated as outliers. It is advisable to set an appropriate outlier value according to various conditions such as the distance between the object to be authenticated T and the imaging unit 40, and the illumination around the imaging unit 40. Furthermore, the allowable inclusion value may be appropriately changed via the input unit 70 according to various conditions.

[0086] Furthermore, in the above embodiment, the length of the discrimination image in the direction perpendicular to the baseline W was set to 15 / 33 of the length of the baseline W. However, for example, to speed up processing, it may be set to 40% of the length of the baseline W, or to be the same as the immediately preceding discrimination image, or to be the average length of the discrimination images that constitute the past average value.

[0087] In the above embodiment, the discrimination image was converted to a square by affine transformation, but it may also be converted to a rectangle.

[0088] In the above embodiment, the image processing device 10 performed face recognition after blink detection, but blink detection may be performed after face recognition, or blink detection and face recognition may be performed in parallel.

[0089] In the above embodiment, the image processing device 10 performed 2D facial recognition, but it may also perform other facial recognition methods, such as facial recognition using an infrared sensor.

[0090] In the above embodiment, the image processing device 10 was used for access control, but it may also be used for attendance management, PC security, logon authentication for tablets and smartphones, etc., or it may be incorporated into robots or lockers.

[0091] In the above embodiment, the image processing device 10 converted the discrimination image to grayscale, but it is also possible to adjust the G / H values ​​as appropriate without performing grayscale conversion.

[0092] Each function of the image processing device 10 of this invention can also be performed by a regular PC (Personal Computer) or tablet. Specifically, in the above embodiment, the programs for the blink detection embedded face recognition processing, discrimination image generation processing, cropped rectangle editing processing, blink discrimination processing, and eye open / closed state discrimination processing performed by the image processing device 10 were described as being pre-stored in the ROM of the storage unit 30. However, a computer capable of realizing the above functions may be configured by distributing the programs on computer-readable recording media such as flexible disks, CD-ROMs (Compact Disc Read Only Memory), DVDs (Digital Versatile Discs), and MOs (Magneto-Optical Discs), and then loading and installing those programs into a computer.

[0093] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these specific embodiments, and the present invention includes the invention described in the claims and its equivalents. The invention described in the original claims of this application is listed below.

[0094] (Note) (Note 1) The editorial department extracts first and second discrimination images from first and second images, which include the eye to be authenticated and are captured in chronological order, based on organ points representing the external shape of the eye to be authenticated, including the assumed position of at least one of the pupil and iris. An acquisition unit that acquires the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, A discrimination unit that determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, An image processing device equipped with the following features.

[0095] (Note 2) The editorial department extracts a third discrimination image, including the assumed position based on the organ point, from a third image which is the image taken immediately before the first image and includes the eye to be authenticated. The acquisition unit acquires the pixel values ​​of the third discrimination image, The discrimination unit determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image, the pixel values ​​of the second discrimination image, and the pixel values ​​of the third discrimination image. The image processing device described in Appendix 1.

[0096] (Note 3) The editorial department extracts the first, second, and third discrimination images based on the organ points that indicate the inner and outer corners of the eye to be authenticated. The image processing apparatus described in Appendix 2.

[0097] (Note 4) The acquisition unit extracts the lowest pixel value from among the pixels arranged in a horizontal line approximately parallel to the baseline connecting the inner and outer corners of the eye to be authenticated, for all pixels of the first, second, and third discrimination images, and calculates the average value of the lowest pixel value for each of the first, second, and third discrimination images. The image processing apparatus described in Appendix 2.

[0098] (Note 5) The acquisition unit calculates a past average value which is the average of the average of the lowest pixel value of the first discrimination image and the average of the lowest pixel value of the third discrimination image. The image processing apparatus described in Appendix 4.

[0099] (Note 6) The acquisition unit calculates the ratio of the average value of the lowest pixel value in the second discrimination image to the past average value, The discrimination unit determines that the eyes of the subject being authenticated are closed in the second image when the ratio is greater than the closed-eye threshold. The image processing apparatus described in Appendix 5.

[0100] (Note 7) The acquisition unit calculates the average value by excluding outliers from the pixels whose pixel value is 20 or more higher than the pixel value indicating the skin color of the subject to authentication. The image processing apparatus described in Appendix 4.

[0101] (Note 8) The acquisition unit acquires pixel values ​​after converting the first, second, and third discrimination images to grayscale. The image processing apparatus described in Appendix 2.

[0102] (Note 9) The acquisition unit sets the length of the first, second, and third discrimination images along the baseline to half the length of the baseline, and the midpoints to coincide with the midpoint of the baseline. The image processing apparatus described in Appendix 4.

[0103] (Note 10) The acquisition unit transforms the extracted first, second, and third discrimination images into rectangles using affine transformation, and makes the bases of the first, second, and third discrimination images parallel to the bases of the first, second, and third images. The image processing apparatus described in Appendix 2.

[0104] (Note 11) The discrimination unit determines whether or not the subject blinks by determining whether the subject blinks in the third image and whether the subject blinks in the second image. The image processing apparatus described in Appendix 6.

[0105] (Note 12) From the first and second images, which are captured in chronological order and include the eye to be authenticated, first and second discrimination images are extracted, including the assumed position of at least one of the pupil and iris, based on organ points representing the external shape of the eye to be authenticated. The pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image are obtained. Based on the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, the presence or absence of blinking of the subject to be authenticated is determined. Image processing methods.

[0106] (Note 13) On the computer, From the first and second images, which are captured in chronological order and include the eye to be authenticated, first and second discrimination images are extracted, including the assumed position of at least one of the pupil and iris, based on organ points representing the external shape of the eye to be authenticated. The pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image are obtained. Based on the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, the presence or absence of blinking of the subject to be authenticated is determined. A program designed to make it work in that way. [Explanation of Symbols]

[0107] 10…Image processing device, 20…Control unit, 21…Image acquisition unit, 22…Detection unit, 23…Discrimination unit, 24…Editing unit, 25…Pixel value acquisition unit (acquisition unit), 26…Face recognition unit, 30…Storage unit, 31…Image storage unit, 32…Face image storage unit, 33…Average value storage unit, 34…Discrimination image storage unit, 35…Face recognition image storage unit, 36…Setting storage unit, 40…Imaging unit, 41…Imaging device, 42…Drive unit, 50…Communication unit, 51…Communication device, 60…Display unit, 61…Display device, 70…Input unit, 71…Input device, B…Dividing line, E1…Inner corner of eye, E2…Outer corner of eye, F…Face, G…Average value, H…Past average value, J1, J2…Vertical line, K1~K4:Reference point, T…Authentication target, V…Organ point, W…Baseline

Claims

1. The editorial department extracts first and second discrimination images from first and second images, which include the eye to be authenticated and are captured in chronological order, based on organ points representing the external shape of the eye to be authenticated, including the assumed position of at least one of the pupil and iris. An acquisition unit that acquires the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, A discrimination unit that determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, Equipped with, The editorial department extracts a third discrimination image, including the assumed position based on the organ point, from a third image, which is the image taken immediately before the first image and includes the eye to be authenticated. The acquisition unit acquires the pixel values ​​of the third discrimination image, The discrimination unit determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image, the pixel values ​​of the second discrimination image, and the pixel values ​​of the third discrimination image. The acquisition unit extracts the lowest pixel value from a row of pixels arranged horizontally in a direction substantially parallel to the baseline connecting the inner and outer corners of the eye to be authenticated, for all pixels of the first, second, and third discrimination images, and calculates the average value of the lowest pixel value for each of the first, second, and third discrimination images. Image processing device.

2. The acquisition unit calculates a past average value which is the average of the average of the lowest pixel value of the first discrimination image and the average of the lowest pixel value of the third discrimination image. The image processing apparatus according to claim 1.

3. The acquisition unit calculates the ratio of the average value of the lowest pixel value in the second discrimination image to the past average value, The discrimination unit determines that the eyes of the subject being authenticated are closed in the second image when the ratio is greater than the closed-eye threshold. The image processing apparatus according to claim 2.

4. The acquisition unit calculates the average value by excluding outliers from the pixels whose pixel value is 20 or more higher than the pixel value that indicates the skin color of the subject to authentication. The image processing apparatus according to claim 1.

5. The acquisition unit sets the length of the first, second and third discriminant images in the direction along the baseline to half the length of the baseline, and the midpoints to coincide with the midpoints of the baseline. The image processing apparatus according to claim 1.

6. The discrimination unit determines whether or not the subject blinks by determining whether the eyes of the subject to be authenticated are open or closed in the third image and by determining whether the eyes of the subject to be authenticated are open or closed in the second image. The image processing apparatus according to claim 3.

7. An image processing method performed by an image processing device, An editing process that extracts first and second discrimination images from first and second images, which include the eye to be authenticated and are captured in chronological order, based on organ points representing the external shape of the eye to be authenticated, including the assumed position of at least one of the pupil and iris. An acquisition process for acquiring the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, A discrimination process that determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, Includes, The editing process extracts a third discrimination image, including the assumed position based on the organ point, from a third image, which is the image captured immediately before the first image and includes the eye to be authenticated. The acquisition process acquires the pixel values ​​of the third discrimination image, The discrimination process determines whether or not the subject to authentication blinks based on the pixel values ​​of the first discrimination image, the pixel values ​​of the second discrimination image, and the pixel values ​​of the third discrimination image. The acquisition process extracts the lowest pixel value from a row of pixels arranged horizontally in a direction substantially parallel to the baseline connecting the inner and outer corners of the eye to be authenticated, for all pixels of the first, second, and third discrimination images, and calculates the average value of the lowest pixel value for each of the first, second, and third discrimination images. Image processing methods.

8. A computer, Editing means for extracting first and second discrimination images from first and second images, which include the eye to be authenticated and are captured in a time series, based on organ points representing the external shape of the eye to be authenticated, including the assumed position of at least one of the pupil and iris. Acquisition means for acquiring the pixel values ​​of the first discrimination image and the pixel values ​​of the second discrimination image, A determination means for determining whether or not the subject to be authenticated blinks, based on the pixel values ​​of the first determination image and the pixel values ​​of the second determination image. To make it function as, The editing means extracts a third discrimination image, including the assumed position, from a third image which is an image captured immediately before the first image and includes the eye to be authenticated, based on the organ point. The acquisition means acquires the pixel values ​​of the third discrimination image, The determination means determines whether or not the subject to be authenticated blinks based on the pixel values ​​of the first determination image, the pixel values ​​of the second determination image, and the pixel values ​​of the third determination image. The acquisition means extracts the lowest pixel value from among the pixels arranged in a horizontal line substantially parallel to the baseline connecting the inner and outer corners of the eye to be authenticated, for all pixels of the first, second, and third discrimination images, and calculates the average value of the lowest pixel value for each of the first, second, and third discrimination images. program.

Images