Image Processing Apparatus, Image Processing Method, and Computer-Readable Storage Medium

Abstract

The present invention relates to an image processing apparatus, an image processing method, and a computer-readable storage medium. A storage unit of the image processing apparatus stores a plurality of time-series images obtained by photographing a subject when instructing the subject to change the orientation of the face. An action determination unit extracts a face region from each of the plurality of images, obtains a change characteristic of pixel values of a plurality of pixels arranged in a predetermined direction in the face region, and determines an action of the subject based on a time-series change of the change characteristics respectively obtained from the plurality of images. A determination unit determines the authenticity of the subject based on the action of the subject.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and an image processing program. Background Art

[0002] Biometric authentication technology is a technology for authenticating a person using biometric information of the authentication target. Biometric authentication technology includes face authentication technology, fingerprint authentication technology, vein authentication technology, voiceprint authentication technology, etc.

[0003] In face authentication technology, when performing person authentication, biometric information obtained from a face image of the authentication target is used. In this case, another person can use the person's photo to undergo person authentication, and thus can impersonate the person. Therefore, an impersonation attack can be easily performed compared with other biometric authentication technologies. To prevent an impersonation attack, it is preferable to perform authenticity determination of whether the face image obtained from the authentication target is a real person's face.

[0004] Regarding authenticity determination of a face image, the following technology is known. Facial feature points indicating the positions of parts representing a face are detected from the face image, and authenticity determination is performed using the movement of the facial feature points when the authentication target performs a specified action (for example, refer to Patent Document 1 and Patent Document 2).

[0005] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2008-71179;

[0006] Patent Document 2: Japanese Unexamined Patent Application Publication No. 2003-99763.

[0007] In the authenticity determination based on existing facial feature points, it is difficult to completely prevent an impersonation attack using a photo, and sometimes a correct determination result cannot be obtained.

[0008] In addition, this problem is not limited to person authentication using a face image, but occurs in various information processes using a face image. Summary of the Invention

[0009] In one aspect, an object of the present invention is to determine the authenticity of a subject using an image obtained by photographing the subject.

[0010] In one technical solution, an image processing apparatus includes a storage unit, a motion determination unit, and a determination unit. The storage unit stores a plurality of time-series images obtained by photographing a subject when instructing the subject to change the orientation of the face.

[0011] The motion determination unit extracts face regions from multiple images, obtains the change characteristics of the pixel values of multiple pixels arranged in a specified direction in the face regions, and determines the motion of the subject based on the time series change of the change characteristics obtained from the multiple images. The determination unit determines the authenticity of the subject based on the motion of the subject.

[0012] According to one method, it is possible to determine the authenticity of a subject using an image obtained by photographing the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a configuration diagram of the functions of an image processing apparatus.

[0014] Figure 2 It is a flowchart of image processing.

[0015] Figure 3 It is a configuration diagram of the functions showing a specific example of an image processing apparatus.

[0016] Figure 4 It is a diagram showing a face image.

[0017] Figure 5 It is a diagram showing the time series change of left - right symmetry.

[0018] Figure 6 It is a diagram showing the time series change of aspect ratio.

[0019] Figure 7 It is a flowchart showing a specific example of image processing.

[0020] Figure 8 It is a flowchart of motion determination processing.

[0021] Figure 9 It is a flowchart of authenticity determination processing.

[0022] Figure 10 It is a configuration diagram of an information processing apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

[0024] In the technology of Patent Document 1, as face feature points, the positions of the eyes, mouth, and nose are used. When registering a face image, the person to be registered is made to perform a specified face motion, and the motions of the eyes, mouth, and nose at this time are registered. Moreover, at the time of authentication, the person to be authenticated is made to perform the same motion, and it is determined whether the face image is a real face based on the motions of the eyes, mouth, and nose at this time.

[0025] In the technology of Patent Document 2, as facial feature points, the positions of the eyes and nostrils are used, and the authentication subject is made to perform an arbitrary or a specified-direction action. Whether the facial image is a real human face is determined based on the actions of the eyes and nostrils during the action.

[0026] However, in the authenticity determination based on facial feature points, it is difficult to completely prevent an impersonation attack using a photo, and sometimes a correct determination result cannot be obtained. For example, when someone holds up a photo, facial feature points can also be detected from the image taken of the photo, and by applying changes such as parallel movement, rotational movement, and bending to the photo, the actions of the facial feature points can be reproduced.

[0027] In addition, facial feature points cannot always be correctly detected. For example, it is difficult to correctly detect one eye when the face is large and facing left or right, or nostrils when the face is large and facing down. Therefore, even if the person performs the instructed action, sometimes the action cannot be correctly determined.

[0028] Figure 1 A configuration example showing the functions of the image processing device according to the embodiment. Figure 1 The image processing device 101 includes a storage unit 111, an action determination unit 112, and a determination unit 113. The storage unit 111 stores a plurality of time-series images obtained by photographing the subject when instructing the subject to change the orientation of the face. The action determination unit 112 and the determination unit 113 perform image processing using the images stored in the storage unit 111.

[0029] Figure 2 It is shown by Figure 1 A flowchart of an example of the image processing performed by the image processing device 101. First, the action determination unit 112 extracts the facial region from each of the plurality of images (step 201).

[0030] Next, the action determination unit 112 obtains the change characteristics of the pixel values of a plurality of pixels arranged in a specified direction in the facial region (step 202), and determines the action of the subject based on the time-series change of the change characteristics obtained from each of the plurality of images (step 203).

[0031] Then, the determination unit 113 determines the authenticity of the subject based on the action of the subject (step 204).

[0032] According to Figure 1 the image processing device 101, it is possible to determine the authenticity of the subject using the image obtained by photographing the subject.

[0033] Figure 3 It shows Figure 1 A configuration example of the functions of a specific example of the image processing device 101.Figure 3 The image processing device 301 includes a storage unit 311, an image acquisition unit 312, a display unit 313, an action instruction unit 314, an action determination unit 315, a determination unit 316, a selection unit 317, a feature extraction unit 318, a registration unit 319, and an authentication unit 320. The storage unit 311, the action determination unit 315, and the determination unit 316 respectively correspond to Figure 1 the storage unit 111, the action determination unit 112, and the determination unit 113 of

[0034] The imaging device 302 is, for example, a camera having an imaging element such as a CCD (Charged-Coupled Device) or a CMOS (Complementary Metal-Oxide-Semiconductor), and captures an image of a subject. The image captured by the imaging device 302 includes a plurality of images in time series. The image at each moment is sometimes also referred to as a frame.

[0035] The image processing device 301 is, for example, a biometric authentication device that performs biometric information processing based on the image of a registration subject or an authentication subject. In the case where the biometric information processing is a registration process for registering the biometric information of a registration subject, the subject is the registration subject. In the case where the biometric information processing is an authentication process for authenticating an authentication subject, the subject is the authentication subject. On the other hand, in the case where an impostor uses the photo of the user himself / herself for an impersonation attack, the subject is the photo of the user himself / herself.

[0036] During the registration process or the authentication process, the action instruction unit 314 instructs the subject to perform a certain action such as changing the orientation of the face. The change in the orientation of the face includes actions such as turning the face towards the imaging device 302 so that the face faces forward, turning the face to the left or right, and turning the face up or down. The action instruction unit 314 can use a text message, an illustration, a voice message, etc. to instruct the subject to perform a certain action.

[0037] The display unit 313 notifies the subject of the instruction by displaying the text message or illustration output from the action instruction unit 314 on the screen.

[0038] During the period when the subject performs the instructed action, the imaging device 302 captures the face of the subject and outputs the captured image to the image processing device 301. The image acquisition unit 312 acquires N (N is an integer of 2 or more) images 331 in time series from the image output by the imaging device 302 and stores them in the storage unit 311.

[0039] The action determination unit 315 detects the position of the face photographed in each image 331, extracts a rectangular face area including the face, and generates a face image, which is an image of the face area. N face images are generated from N images 331. At this time, the action determination unit 315 detects the positions of the left and right eyes and the nose from each image 331, and generates a face image in which the positions of the left and right eyes are horizontal and the position of the nose is at the center of the face image.

[0040] Figure 4 An example of a face image generated from the image 331 is shown. In Figure 4 For the face image, the x-axis and y-axis are set as coordinate axes. The x-axis represents the direction of the short side of the face image (horizontal direction), and the y-axis represents the direction of the long side of the face image (vertical direction). The direction of the short side is the left-right direction of the face, and the direction of the long side is the up-down direction of the face.

[0041] The nose is located on the center line 401 passing through the midpoint of the short side, and the left and right eyes are arranged on a straight line parallel to the short side. The face image is divided by the center line 401 into a left region 411 and a right region 412.

[0042] The action determination unit 315 obtains the change characteristics of the pixel values of a plurality of pixels arranged in the left-right direction in each of the N face images, and obtains the time-series change of the change characteristics of the pixel values obtained from the N face images. In addition, the action determination unit 315 obtains the aspect ratio of each of the N face images, and obtains the time-series change of the aspect ratios obtained from the N face images.

[0043] Then, the action determination unit 315 uses the time-series change of the change characteristics of the pixel values and the time-series change of the aspect ratios to determine the action of the subject, generates action information 333 indicating the determined action, and stores it in the storage unit 311. For example, the action information 333 indicates an action of turning the face forward, an action of turning the face to the left, an action of turning the face to the right, an action of turning the face upward, an action of turning the face downward, etc.

[0044] The determination unit 316 updates the correct answer counter 334 and the incorrect answer counter 335 using the action information 333, and updates the abnormality counter 336 using the time-series change of the change characteristics of the pixel values and the time-series change of the aspect ratios.

[0045] The correct answer counter 334, the incorrect answer counter 335, and the abnormality counter 336 are used when the subject is instructed to change the face orientation multiple times. The correct answer counter 334 indicates the number of times the action indicated by the action information 333 is consistent with the action instructed to the subject. The incorrect answer counter 335 indicates the number of times the action indicated by the action information 333 is inconsistent with the action instructed to the subject.

[0046] In addition, the abnormality counter 336 indicates the number of times an inappropriate action of the subject has been detected. For example, actions such as someone else holding up a photo of the person, changing the orientation of the photo, or distorting the photo and then attempting to reproduce the indicated action are detected as inappropriate actions.

[0047] The determination unit 316 performs authenticity determination of the subject using the count values indicated by the correct answer counter 334, the incorrect answer counter 335, and the abnormality counter 336. In the authenticity determination, it is determined whether the subject is real, that is, whether the subject is a real person or a forgery.

[0048] For example, when the count value of the correct answer counter 334 is greater than a specified value, the determination unit 316 determines that the subject is a real person, and when the count value of the incorrect answer counter 335 is greater than the specified value, it determines that the subject is a forgery. Moreover, the determination unit 316 stores the determination result 337 indicating whether the subject is a real person or a forgery in the storage unit 311.

[0049] By instructing the subject to change the orientation of the face multiple times and counting the number of times the subject performs the indicated action, the determination accuracy is improved compared to the case of performing authenticity determination based on a single action.

[0050] When the determination result 337 indicates a real person, the selection unit 317 selects a processing target image from the N face images respectively generated from the N images 331. For example, as the processing target image, a face image determined to have the face facing forward is selected.

[0051] The feature extraction unit 318 extracts feature information of the face photographed in the processing target image. For example, as the feature information of the face, position information indicating the positions of the respective parts of the face can be used. By using a face image with the face facing forward as the processing target image, feature information suitable for registration processing and authentication processing can be extracted.

[0052] In the registration processing, the registration unit 319 stores the extracted feature information as the registered biometric information 338 of the registered subject in the storage unit 311. In the authentication processing, the authentication unit 320 performs authentication of the authentication target subject by comparing the extracted feature information with the registered biometric information 338.

[0053] It is difficult to reproduce the time-series change of the pixel value change characteristics in the face image and the time-series change of the aspect ratio of the face image even when using a photo or the like. Therefore, by performing authenticity determination using such information, it is possible to accurately identify a real person's face and a forgery, and achieve reliable and secure face authentication against spoofing attacks.

[0054] The action determination unit 315 may also use, instead of the facial image region being an image of the entire face, an image of a partial region of the face as the facial image. The partial region of the face may also be a region including the part from the eyes to the chin. By using the partial region of the face, it is possible to reduce the influence of parts that vary significantly depending on the person or the shooting time, such as hair.

[0055] In addition, the action determination unit 315 may also apply a prescribed image correction process to the pixels of the facial image, use the facial image to which the image correction process has been applied, and obtain the change characteristics of the pixel values. As the image correction process, filtering processing based on a frequency filter, brightness correction processing based on histogram equalization, etc. are used. For example, by applying a blur filter (low-pass filter), it is possible to reduce the influence of minute parts such as moles, and by applying histogram equalization processing, it is possible to reduce the influence of the light source.

[0056] As the change characteristics of the pixel values in the left-right direction of the facial image, the left-right symmetry of the pixel values can be used. The action determination unit 315 calculates the left-right symmetry Sym, for example, by the following formula.

[0057] [Equation 1]

[0058]

[0059] [Equation 2]

[0060]

[0061] W represents the width of the facial image, and H represents the height of the facial image. I(x, y) in Equation (1) represents the luminance value at the coordinate (x, y) of the facial image, and I(W - x + 1, y) represents the luminance value at the position symmetric to the coordinate (x, y) with respect to the bisector that bisects the facial image left and right.

[0062] weight(x) is a weighting coefficient depending on the coordinate x. The closer to the center of the facial image (x = W / 2), the more the luminance value is likely to be affected by parts with high left-right symmetry such as hair, so weight(x) is set to a larger value as the x coordinate gets closer to the center.

[0063] In this case, Diff in Equation (1) represents the sum obtained by adding the information related to the difference in the luminance values of two pixels existing at symmetric positions with respect to the bisector of the facial image for the entire facial image. The smaller Diff is, the larger Sym in Equation (2) is, and in the case where the facial image is completely left-right symmetric, Sym in Equation (2) is the largest. The maximum value of Sym is 1, and the minimum value is 0. The action determination unit 315 may also calculate Diff using RGB, color difference signals, etc. of each pixel instead of the luminance value of each pixel.

[0064] When the face of a person faces forward, the left - right symmetry increases, and when it faces left or right, the left - right symmetry decreases. When the face faces upward or downward, the left - right symmetry is maintained. Therefore, by instructing the subject to change the face orientation in a specified order, Sym is calculated based on the face image during the period when the subject performs the instructed action, and thus the time - series change of the left - right symmetry can be obtained.

[0065] Figure 5 An example of the time - series change of the left - right symmetry is shown when the subject changes the face orientation in the order of front, left, front, right, and front. The horizontal axis represents the angle of the face in the left - right direction. 0° represents the front, positive values represent the left orientation when viewed from the subject, and negative values represent the right orientation when viewed from the subject. The vertical axis represents the left - right symmetry Sym.

[0066] Figure 5 (a) shows an example of the time - series change of the left - right symmetry when the subject is a real person's face. In this case, the time - series change is represented by a smooth curve, and the greater the angle to the left or right, the smaller the left - right symmetry.

[0067] Figure 5 (b) shows an example of the time - series change of the left - right symmetry when the subject is a photo and someone tries to rotate the photo left - right to reproduce the action of a real person's face. In this case, the generated face image is obtained by affine transformation of the original photo, so the left - right symmetry hardly changes.

[0068] Figure 5 (c) shows an example of the time - series change of the left - right symmetry when the subject is a photo and someone applies shape changes such as bending when rotating the photo. In this case, by applying shape changes to the photo, the left - right symmetry becomes smaller. However, in the shape change of a photo as a two - dimensional image, a sharp change in the left - right symmetry occurs, so it is difficult to reproduce Figure 5 (a) such a smooth curve.

[0069] Therefore, in the case of detecting Figure 5 (a) such a time - series change, the action determination unit 315 determines that the subject performs an action to the left or right. For example, the action determination unit 315 can detect Figure 5 (a)'s time - series change by performing frequency analysis on the time - series change of Sym.

[0070] In addition, the motion determination unit 315 may also pre-record the statistical values of Sym at each angle when multiple indeterminate persons perform the same motion, evaluate the error between the Sym calculated from the face image of the subject and the statistical values, and thereby detect Figure 5 (a) The time series change. As the statistical value, the average value, the median value, the mode value, etc. are used.

[0071] As described above, by utilizing the time series change of the left-right symmetry of the pixel values, the state of the subject facing left or right can be detected. In the case where the state of the subject facing left or right is detected due to the reduction of the left-right symmetry, the motion determination unit 315 uses, for example, the following method to determine whether the face is facing left or right.

[0072] (A1) When the nose photographed in the face image is located at a position to the left of the center position between the left and right pupils, the motion determination unit 315 determines that the face of the subject faces right. When the nose is located at a position to the right of the center position between the left and right pupils, the motion determination unit 315 determines that the face of the subject faces left.

[0073] (A2) The motion determination unit 315 compares the average value LV of the luminance values in the left region of the face image with the average value RV of the luminance values in the right region of the face image. Then, when LV is larger than RV, the motion determination unit 315 determines that the face of the subject faces right. When RV is larger than LV, the motion determination unit 315 determines that the face of the subject faces left.

[0074] The motion determination unit 315 may also obtain the change characteristics of the pixel values of multiple pixels arranged in a direction other than left and right, such as the up-down direction of each face image, and use the time series change of the change characteristics to determine the motion of the subject.

[0075] The aspect ratio of the face image represents the ratio H / W of the height H to the width W of the face image, and can be obtained by dividing the height H by the width W. When the face faces up or down, the width W of the face image hardly changes compared to the face image with the face facing forward, but the height H is smaller than the face image with the face facing forward. Therefore, by instructing the subject to change the face orientation in a specified order and calculating the aspect ratio based on the face image during the period when the subject performs the instructed motion, the time series change of the aspect ratio can be obtained.

[0076] Figure 6 An example of the time series change of the aspect ratio when the subject changes the face orientation in the order of front, up, front, down, and front is shown. The horizontal axis represents the angle of the face in the up-down direction. 0° represents the front, positive values represent up, and negative values represent down. The vertical axis represents the aspect ratio.

[0077] For example, the motion determination unit 315 can detect Figure 6 such time-series changes by performing frequency analysis on the time-series changes in the aspect ratio. In addition, the motion determination unit 315 may also pre-record the statistical values of the aspect ratio at each angle when multiple uncertain persons perform the same action, and evaluate the error between the aspect ratio calculated from the face image of the subject and the statistical values, thereby detecting Figure 6 the time-series changes. In this way, by utilizing the time-series changes in the aspect ratio of the face image, it is possible to detect the state where the subject is facing upward or downward.

[0078] It is also possible to use the left-right symmetry and aspect ratio when the face is facing forward as reference values to evaluate the left-right symmetry and aspect ratio at each moment. In this case, the motion instruction unit 314 instructs the subject to face forward. When the left-right symmetry is within the specified range R1 and the aspect ratio is within the specified range R2, the motion determination unit 315 determines that the face is facing forward. Moreover, the left-right symmetry and aspect ratio at this time are stored in the storage unit 311 as the reference left-right symmetry and reference aspect ratio, respectively.

[0079] For example, the motion determination unit 315 sets an allowable error for the left-right symmetry calculated from the face image of the registration object facing forward during the registration process, or the left-right symmetry calculated from the face images of multiple persons facing forward, thereby determining the specified range R1. In addition, the motion determination unit 315 sets an allowable error for the aspect ratio calculated from the face image of the registration object facing forward during the registration process, or the aspect ratio calculated from the face images of multiple persons facing forward, thereby determining the specified range R2.

[0080] Next, the motion instruction unit 314 instructs the subject to face left, right, up, or down. The motion determination unit 315 calculates the difference D1 between the left-right symmetry obtained from the face image and the reference left-right symmetry, and calculates the difference D2 between the aspect ratio obtained from the face image and the reference aspect ratio. Moreover, the motion determination unit 315 determines whether the face is facing left or right based on the time-series change of the difference D1, and determines whether the face is facing up or down based on the time-series change of the difference D2.

[0081] The left-right symmetry and aspect ratio when the face is facing forward are affected by individual differences in the positions of facial parts, shooting environments, etc. However, by using the differences D1 and D2 to absorb these effects, it is possible to accurately determine the orientation of the face.

[0082] It is also possible to pre-record the number of times the action instruction unit 314 instructs the subject to perform an action, and based on the ratio of the count values of the correct answer counter 334 and the incorrect answer counter 335 to the number of instructions, determine the authenticity of the subject. The ratio of the count value of the correct answer counter 334 to the number of instructions represents the correct answer rate, and the ratio of the count value of the incorrect answer counter 335 to the number of instructions represents the incorrect answer rate.

[0083] For example, when the correct answer rate is greater than a specified value, the determination unit 316 determines that the subject is a real person. On the other hand, when the correct answer rate is less than or equal to the specified value and the incorrect answer rate is greater than the specified value, the determination unit 316 determines that the subject is a forgery.

[0084] The determination unit 316 can also calculate the sharpness of each of the N face images, and based on the time series change of the sharpness of these face images, determine the authenticity of the subject. The sharpness of an image represents the degree of blurring or shaking of the image, and can be calculated using, for example, DOG (Difference of Gaussian). The smaller the blurring or shaking, the greater the sharpness.

[0085] When the subject is a real person, the left-right symmetry, aspect ratio, and sharpness of the face image change smoothly over time. However, in the case of an inappropriate action of the subject based on a forgery, as Figure 5 shown in (c), rapid changes are likely to occur.

[0086] Therefore, the determination unit 316 calculates the difference between the maximum value and the minimum value for the left-right symmetry, aspect ratio, and sharpness of the past M face images, respectively. When any of the differences is greater than the specified value, it is determined that an inappropriate action has been performed. M can also be an integer greater than or equal to 2 and less than or equal to N. Moreover, when the abnormal counter 336 is incremented by 1 and the count value of the abnormal counter 336 exceeds the threshold, the determination unit 316 determines that the subject is a forgery. In this way, by utilizing the time series changes of the left-right symmetry, aspect ratio, and sharpness of the face image, inappropriate actions can be detected.

[0087] On the other hand, when there is almost no time series change in the left-right symmetry, aspect ratio, and sharpness, it is possible that the subject remains stationary without following the instructions, or there is no change because the subject is a forgery. In such a case, it is not preferable to determine that the subject is a real person.

[0088] Therefore, the determination unit 316 calculates the differences between the maximum value and the minimum value for the left-right symmetry, aspect ratio, and sharpness of the past M facial images, respectively. When all the differences are smaller than the specified value, the still state detection counter is incremented by 1. Moreover, when the count value of the still state detection counter exceeds the threshold, the determination unit 316 determines that the subject is a forgery.

[0089] The selection unit 317 may also select, as the processing target image, a facial image with sharpness greater than the specified value among the facial images determined to face the front when the determination result 337 indicates a real person.

[0090] Figure 7 It represents a Figure 3 Flowchart showing a specific example of the image processing performed by the image processing device 301. First, the action instruction unit 314 instructs the subject to perform actions such as changing the facial orientation (step 901), and the image acquisition unit 312 acquires N time-series images 331 from the video output by the imaging device 302 (step 902).

[0091] Next, the action determination unit 315 generates facial images based on the N images 331 respectively (step 903), determines the action of the subject using a part or all of the N facial images, and generates action information 333 representing the determined action (step 904).

[0092] Next, the determination unit 316 performs authenticity determination using the action information 333 (step 905). When the subject is a real person, the determination unit 316 generates a determination result 337 indicating that the subject is a real person, and when the subject is a forgery, it generates a determination result 337 indicating that the subject is a forgery. When it is not clear whether the subject is a real person or a forgery, the determination result 337 is not generated.

[0093] The selection unit 317 checks whether the determination result 337 indicates a real person (step 906). When the determination result 337 indicates a real person (step 906, Yes), it selects the processing target image from the N facial images (step 907). Moreover, the feature extraction unit 318 extracts the facial feature information from the selected processing target image.

[0094] Next, the registration unit 319 or the authentication unit 320 performs biometric information processing (step 908). When the biometric information processing is registration processing, the registration unit 319 registers the extracted feature information as the registered biometric information 338 of the subject in the storage unit 311. When the biometric information processing is authentication processing, the authentication unit 320 performs authentication of the subject by comparing the extracted feature information with the registered biometric information 338.

[0095] When the determination result 337 has not been generated or the determination result 337 does not indicate a real person (step 906, No), the action instruction unit 314 checks whether the determination result 337 indicates a forgery (step 909). When the determination result 337 indicates a forgery (step 909, Yes), the action instruction unit 314 performs an error process (step 910). In the error process, the action instruction unit 314 generates an error message indicating that the photographed object is not real, and the display unit 313 displays this error message on the screen.

[0096] When the determination result 337 has not been generated (step 909, No), the image processing apparatus 301 repeats the processes after step 901.

[0097] Figure 8 Is the indication Figure 7 The flowchart of an example of the action determination process in step 904. First, the action determination unit 315 selects one face image from N face images and calculates the left-right symmetry of the selected face image (step 1001). Moreover, the action determination unit 315 determines the orientation of the face based on the time-series change of the left-right symmetry (step 1002) and checks whether the face faces left or right (step 1003). When the face faces left or right (step 1003, Yes), the action determination unit 315 generates action information 333 indicating an action to make the face face left or right (step 1010).

[0098] When the face does not face left or right (step 1003, No), the action determination unit 315 calculates the aspect ratio of the selected face image (step 1004). Moreover, the action determination unit 315 determines the orientation of the face based on the time-series change of the aspect ratio (step 1005) and checks whether the face faces up or down (step 1006). When the face faces up or down (step 1006, Yes), the action determination unit 315 generates action information 333 indicating an action to make the face face up or down (step 1010).

[0099] When the face does not face up or down (step 1006, No), the action determination unit 315 determines the orientation of the face based on the time-series changes of the left-right symmetry and the aspect ratio (step 1007) and checks whether the face faces the front (step 1008). When the face faces the front (step 1008, Yes), the action determination unit 315 generates action information 333 indicating an action to make the face face the front (step 1010).

[0100] When the face is not facing forward (No in step 1008), the action determination unit 315 checks whether the selected face image is the last face image (step 1009). When there are unselected face images remaining (No in step 1009), the action determination unit 315 repeats the processing after step 1001 for the next face image. Moreover, when the selected face image is the last face image (Yes in step 1009), the action determination unit 315 ends the processing.

[0101] Figure 9 is a flowchart showing an example of the authenticity determination process in step 905. First, the determination unit 316 obtains the sharpness of each of the past M face images, and calculates the difference between the maximum value and the minimum value for the left-right symmetry, aspect ratio, and sharpness of these face images. Moreover, the determination unit 316 determines whether the subject has made an inappropriate action based on the calculated differences (step 1101). Figure 7

[0102] When the subject has not made an inappropriate action (No in step 1101), the determination unit 316 checks whether the action of the subject shown in the action information 333 follows the instruction of the action instruction unit 314 (step 1102). When the action of the subject follows the instruction (Yes in step 1102), the determination unit 316 increments the correct answer counter 334 by 1 (step 1103). On the other hand, when the action of the subject does not follow the instruction (No in step 1102), the determination unit 316 increments the incorrect answer counter 335 by 1 (step 1104).

[0103] Next, the determination unit 316 compares the count value of the correct answer counter 334 with the threshold TH1 (step 1105). When the count value of the correct answer counter 334 is greater than TH1 (Yes in step 1105), the determination unit 316 determines that the subject is a real person and generates a determination result 337 indicating that the subject is a real person (step 1107).

[0104] On the other hand, when the count value of the correct answer counter 334 is less than or equal to TH1 (No in step 1105), the determination unit 316 compares the count value of the incorrect answer counter 335 with the threshold TH2 (step 1106). When the count value of the incorrect answer counter 335 is greater than TH2 (Yes in step 1106), the determination unit 316 determines that the subject is a forgery and generates a determination result 337 indicating that the subject is a forgery (step 1108).

[0105] On the other hand, when the count value of the incorrect answer counter 335 is less than or equal to TH2 (No in step 1106), the determination unit 316 ends the processing.

[0106] When the subject makes an inappropriate motion (step 1101, Yes), the determination unit 316 increments the abnormality counter 336 by 1 (step 1109), and compares the count value of the abnormality counter 336 with the threshold value TH3 (step 1110). When the count value of the abnormality counter 336 is greater than TH3 (step 1110, Yes), the determination unit 316 determines that the subject is a forgery, and generates a determination result 337 indicating that the subject is a forgery (step 1111).

[0107] On the other hand, when the count value of the abnormality counter 336 is less than or equal to TH3 (step 1110, No), the determination unit 316 ends the process.

[0108] In addition, in the authenticity determination process, a timeout period may be set for the motion of the subject. In this case, after the motion instruction unit 314 instructs the subject to change the orientation of the face, when the motion of the subject that is consistent with the change in the instructed face orientation cannot be determined based on the image 331 captured within a specified time, the determination unit 316 determines that the subject is a forgery.

[0109] By setting a timeout period for the motion of the subject, error processing can be performed when the subject does not follow the instruction, and the registration of the biological information 338 or the authentication of the subject can be rejected.

[0110] Figure 1 The structure of the image processing device 101 and Figure 3 The structure of the image processing device 301 is merely an example, and some constituent elements may be omitted or changed according to the use or conditions of the image processing device. For example, in Figure 3 In the image processing device 301, when N images 331 in time series are pre-stored in the storage unit 311, the image acquisition unit 312, the display unit 313, and the motion instruction unit 314 can be omitted.

[0111] When the biological information processing is performed by an external device, the selection unit 317, the feature extraction unit 318, the registration unit 319, and the authentication unit 320 can be omitted. The image processing device 301 may perform other information processing using the face image instead of performing biological information processing using the face image.

[0112] Figure 2 and Figures 7 - 9 The flowchart is merely an example, and some processes may be omitted or changed according to the structure or conditions of the image processing device. For example, in Figure 7In the image processing, when N images 331 in a time series are pre-stored in the storage unit 311, the processes of step 901 and step 902 can be omitted.

[0113] When the biological information processing is performed by an external device, the processes of step 907 and step 908 can be omitted. In step 908, the image processing device 301 may perform other information processing instead of the biological information processing.

[0114] In Figure 8 In the action determination process, when determining the action of the subject only based on the left-right symmetry of the face image, the processes of step 1004 to step 1006 can be omitted.

[0115] In Figure 9 In the authenticity determination process, when not using the inappropriate actions of the subject for authenticity determination, the processes of step 1101 and step 1109 to step 1111 can be omitted. When not using the number of times the subject fails to perform the indicated action for authenticity determination, the processes of step 1104, step 1106, and step 1108 can be omitted.

[0116] Figure 4 The face image of Figure 5 The time-series change in the left-right symmetry of Figure 6 The time-series change in the aspect ratio of

[0117] Figure 10 Indicates Figure 1 The image processing device 101 of Figure 3 And the configuration example of the information processing device (computer) used as the image processing device 301 of Figure 10 The information processing device of Figure 3 The imaging device 302 of

[0118] The memory 1202 is, for example, a semiconductor memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a flash memory, and stores programs and data for processing. The memory 1202 can serve as Figure 1 the storage unit 111 of Figure 3 or the storage unit 311 of

[0119] The CPU 1201 (processor) executes programs using the memory 1202, for example, and thus operates as Figure 1 the action determination unit 112 and the determination unit 113 of Figure 3 The CPU 1201 also operates as

[0120] the image acquisition unit 312, the action instruction unit 314, the action determination unit 315, the determination unit 316, the selection unit 317, the feature extraction unit 318, the registration unit 319, and the authentication unit 320 of Figure 3 by executing programs using the memory 1202. The output device 1204 can serve as

[0121] the display unit 313 of Figure 1 The input device 1203 is, for example, a keyboard, a pointing device, etc., and is used for inputting instructions or information from an operator or a user. The output device 1204 is, for example, a display device, a printer, a speaker, etc., and is used for outputting inquiries or instructions to the operator or the user, as well as processing results. The instruction to the operator or the user can also be an action instruction for the subject to be photographed, and the processing result can also be the determination result 337. Figure 3 or the storage unit 311 of

[0122] The medium drive device 1206 drives the portable recording medium 1209 to access its recorded content. The portable recording medium 1209 can be a storage device, a floppy disk, an optical disk, an optical magnetic disk, etc. The portable recording medium 1209 can also be a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, etc. An operator or user can pre-store programs and data in the portable recording medium 1209 and load them into the memory 1202 for use.

[0123] In this way, the computer-readable recording medium storing the programs and data for processing is a physical (non-transitory) recording medium such as the memory 1202, the auxiliary storage device 1205, or the portable recording medium 1209.

[0124] The network connection device 1207 is a communication interface circuit that connects to communication networks such as a LAN (Local Area Network) and a WAN (Wide Area Network) and performs data conversion associated with communication. The information processing device can receive programs and data from an external device via the network connection device 1207 and load them into the memory 1202 for use.

[0125] In addition, the information processing device does not need to include Figure 10 all of the constituent elements, and some constituent elements can also be omitted according to the use or conditions. For example, in the case where an interface with an operator or user is not required, the input device 1203 and the output device 1204 can be omitted. In the case where the portable recording medium 1209 or the communication network is not used, the medium drive device 1206 or the network connection device 1207 can be omitted.

[0126] Although the embodiments of the invention and their advantages have been described in detail, those skilled in the art can make various changes, additions, and omissions without departing from the scope of the invention clearly recited in the claims.

Claims

1. An image processing apparatus, characterized in that, Comprising: a storage unit that stores a plurality of time-series images obtained by photographing the subject when instructing the subject to change the orientation of the face; a motion determination unit that extracts a face region from each of the plurality of time-series images, and obtains the change characteristics of the pixel values of the pixels arranged in a specified direction in the specified direction based on the difference in pixel values of two pixels where the bisector lines of the face images arranged in the specified direction in the face region are in symmetric positions, and determines the motion of the subject based on the time-series change of the change characteristics in the specified direction obtained from each of the plurality of time-series images; and a determination unit that determines the authenticity of the subject based on the motion of the subject.

2. The image processing apparatus according to claim 1, wherein the change in the orientation of the face instructed to the subject represents an action of turning the face to the left or to the right, the specified direction is the left-right direction of the face, the change characteristics of the pixel values of the pixels arranged in the specified direction in the specified direction represent the left-right symmetry of the pixel values of the plurality of pixels arranged in the specified direction, the motion of the subject turns the face to the left or to the right.

3. The image processing apparatus according to claim 1, wherein the motion determination unit further applies image correction processing to the pixels of the face region, and obtains the change characteristics of the pixel values of the pixels arranged in the specified direction based on the face region to which the image correction processing has been applied.

4. The image processing apparatus according to claim 1, wherein the motion determination unit further obtains the aspect ratio of the face region, and determines the motion of the subject based on the time-series change of the aspect ratio obtained from each of the plurality of time-series images.

5. The image processing apparatus according to claim 2, wherein the motion determination unit further obtains the aspect ratio of the face region, and when the left-right symmetry is within a specified range and the aspect ratio is within a specified range, determines that the face is facing the front with respect to the photographing apparatus that photographs the subject, and stores the left-right symmetry and the aspect ratio when the face is determined to be facing the front with respect to the photographing apparatus as a reference left-right symmetry and a reference aspect ratio in one or more memories, and determines whether the face turns to the left or to the right based on the time-series change of the difference between the left-right symmetry obtained after storing the reference left-right symmetry and the reference aspect ratio in the storage unit and the reference left-right symmetry.

6. The image processing apparatus according to claim 5, wherein the image processing apparatus further includes a selection unit that selects, as a face image to be registered or authenticated, an image of a face region determined to be facing the front from the face regions extracted from each of the plurality of time-series images.

7. The image processing apparatus according to claim 1, wherein When the change in the orientation of the face is indicated to the subject multiple times, and the number of times the determined action of the subject coincides with the change in the orientation of the face indicated to the subject is greater than a specified value, the determination unit further determines that the subject is real.

8. The image processing apparatus according to claim 1, wherein: After indicating the change in the orientation of the face to the subject, when an action that coincides with the change in the orientation of the face cannot be determined based on the images captured within a specified time, the determination unit further determines that the subject is not real.

9. An image processing method, which is executed by a computer to perform the following processing: Store a plurality of time-series images of the subject captured when indicating the change in the orientation of the face to the subject in one or more memories. Extract the face region from each of the plurality of time-series images. Obtain the change characteristics in a specified direction of the pixel values of the pixels arranged in the specified direction based on the difference in the pixel values of two pixels where the bisector lines of the face image arranged in the specified direction in the face region are located at symmetric positions. Determine the action of the subject based on the time-series change of the change characteristics in the specified direction obtained from each of the plurality of time-series images. Determine the authenticity of the subject based on the action of the subject.

10. A non-transitory computer-readable storage medium storing an image processing program for causing at least one computer to perform the following processing: Store a plurality of time-series images of the subject captured when indicating the change in the orientation of the face to the subject in one or more memories. Extract the face region from each of the plurality of time-series images. Obtain the change characteristics in a specified direction of the pixel values of the pixels arranged in the specified direction by obtaining the difference in the pixel values of two pixels where the bisector lines of the face image arranged in the specified direction in the face region are located at symmetric positions. Determine the action of the subject based on the time-series change of the change characteristics in the specified direction obtained from each of the plurality of time-series images. Determine the authenticity of the subject based on the action of the subject.

Images