Image processing device, image processing method, and program

Abstract

To provide an image processing device capable of efficiently executing biological determination in a situation in which a photographing condition changes.SOLUTION: An image processing device for determining whether a person in an image is a living body comprises: decision means for deciding a threshold value indicating a skin color range in a time-series image acquired at a prescribed time; acquisition means for acquiring color information about a skin color region included in the time-series image on the basis of the threshold value; and determination means for determining whether or not to continue biological determination processing for determining whether the person of the time-series image is a living body on the basis of the color information.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a technique for determining whether a person included in an image is a living body or not.

Background Art

[0002] There is a living body determination technique for determining whether a person shown in an image or video is a living body or not.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When the illumination color, illuminance, background color, etc. change during the living body determination process and the automatic correction function of the camera operates, noise other than the skin color change due to blood flow may enter, and the accuracy of the living body determination process may decrease. In Patent Document 1, when the estimation of the pulse is started, the image correction by the automatic correction unit of the camera is stopped. However, in the method of Patent Document 1, in a situation where the shooting conditions change, a situation may occur where the automatic correction of the camera does not function well, and the accuracy of the living body determination process may decrease.

[0005] The present invention has been made in view of the above problems, and an object thereof is to efficiently perform living body determination even in a situation where the shooting conditions change.

Means for Solving the Problems

[0006] The image processing apparatus according to the present invention is an image processing apparatus for determining whether a person in an image is a living being, and includes: determination means for determining a threshold value indicating the range of skin color in a time-series image acquired over a predetermined period of time; acquisition means for acquiring color information about the skin color region included in the time-series image based on the threshold value; and determination whether or not to continue the living being determination process for determining whether the person in the time-series image is a living being based on the color information. Based on the amount of change in the color information, it is determined whether or not to update the threshold used in the biological detection process. It is characterized by having a determination means for doing so. [Effects of the Invention]

[0007] This invention enables efficient biological identification even when imaging conditions change. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram showing an example of the hardware configuration of an image processing device. [Figure 2] This is a block diagram showing an example of the functional configuration of an image processing device. [Figure 3] This is a flowchart illustrating the processes performed by an image processing device. [Figure 4] This is a flowchart illustrating the processes performed by an image processing device. [Figure 5] This figure shows an example of a biometric identification process. [Figure 6] This is a flowchart explaining what the image processing unit does. [Figure 7] This figure shows an example of a biometric identification process. [Figure 8] This is a flowchart illustrating the processes performed by an image processing device. [Figure 9] This figure shows an example of a biological identification process. [Figure 10] This is a flowchart illustrating the processes performed by an image processing device. [Figure 11] This figure shows an example of a biological identification process. [Figure 12] This is an example of a GUI that shows the progress of the biometric identification process. [Modes for carrying out the invention]

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0010] One method of biometric identification involves extracting a region of the face, determining the change in skin color within that region, measuring the period of change, and estimating the pulse rate. Another method of biometric identification involves determining whether the change in skin color is due to a change in blood flow. In this embodiment, we will describe a biometric identification process that determines whether a person in an image is alive or not by estimating the pulse rate or detecting changes in blood flow of a subject (person) in the image.

[0011] <Embodiment 1> Figure 1 is a block diagram showing an example of the hardware configuration of the image processing device 1 (biological determination device) according to this embodiment. The central processing unit (CPU) 11 uses RAM 13 as work memory to read and execute the OS and other programs stored in ROM 12 and storage device 14, and controls each component connected to the system bus 19 to perform calculations and logical decisions for various processes. The processes executed by the CPU 11 include the biological determination process of this embodiment. The storage device 14 is a hard disk drive or external storage device, and stores programs and various data related to the image processing of this embodiment. The input unit 15 is an imaging device such as a camera, and an input device such as a button, keyboard, or touch panel for inputting user instructions. The storage device 14 is connected to the system bus 19 via an interface such as SATA, and the input unit 15 is connected via a serial bus such as USB, but the details of these connections are omitted. The communication interface 16 communicates with external devices wirelessly. The display unit 17 is a display. The sensor 18 is an image sensor or a distance sensor. The biological determination device does not necessarily have to have some of the hardware described here. For example, if the biometric detection device is a network camera, then input devices such as a keyboard and the display unit 17 may or may not be present.

[0012] FIG. 2 is a block diagram showing a functional configuration example of the image processing apparatus according to Embodiment 1. An image processing apparatus (biometric determination apparatus) 1 performs "biometric determination (spoofing determination)" to determine whether a subject is a living body using color information detected from an image. The image processing apparatus 1 includes an image acquisition unit 201, a face detection unit 202, a skin color threshold determination unit 203, a skin color region detection unit 204, a skin color change measurement unit 205, an update determination unit 206, a hue calculation unit 207, a hue change management unit 208, a biometric determination unit 209, and a result output unit 210.

[0013] The image acquisition unit 201 acquires an image of a target person. Here, a time-series image (video) captured by an imaging device capable of communicating via a network is acquired. Here, the time-series image indicates a plurality of image groups captured over a certain period.

[0014] The face detection unit 202 detects a face from the acquired image. An existing image recognition method is used for face detection. Specifically, a face detector trained with face images using a neural network is used to detect a rectangle indicating the position and size of the face included in the image. Alternatively, the position of the face included in the image is detected by matching processing using a face image template. Alternatively, a screen for guiding the target person to enter a predetermined shooting angle may be generated, and a face image may be acquired by extracting a face region from an image when a person appears within the frame of the screen. Here, when referring to a face image, it refers to a partial image obtained by extracting a rectangle including the face from the image. Therefore, the description is simplified by assuming that the color information and feature amounts extracted from the face image are information indicating the face of a person.

[0015] The threshold determination unit 203 determines the threshold of the target color in the input video based on the color information in the face region. The threshold determined here is called the "skin color threshold" because the target color is the color of human skin. The skin color threshold indicates the definition of skin color in an image and defines the conditions for identifying skin color pixels in the image based on the hue, saturation, and lightness included in the color information. The method for determining the skin color threshold will be described later. The skin color region detection unit 204 acquires a partial image (skin color region) used for biological determination from the image based on the determined threshold of the target color. The hue calculation unit 207 calculates the hue based on the color information of each pixel in the determined skin color region and obtains the average hue value of all pixels (hereinafter referred to as the hue average value). The skin color change measurement unit 205 measures the change amount of the number of pixels in the skin color region detected by the skin color region detection unit 204 and the change amount of the hue average value obtained by the hue calculation unit 206. The update determination unit 206 determines whether to update the skin color threshold based on the change amount obtained by the skin color change measurement unit 205. To determine to update the skin color threshold, the skin color threshold determination unit 203 is instructed to redetermine the skin color threshold based on the image after the determination, and the hue change management unit 208 is instructed to exclude the change amount of the hue average value up to that point from the target of the biological determination process.

[0016] The hue change management unit 208 calculates and records the change amount of the hue average value based on the hues calculated in a plurality of frames. The biological determination unit 209 (biological determination unit) determines whether it is a living body from the calculated change amount of the hue average value. Note that the biological determination unit 209 may execute a pulse estimation process for estimating the pulse rate of the subject based on the color information. Alternatively, the biological determination unit 209 may determine whether the subject is a living body by detecting the blood flow of the subject. The result output unit 210 outputs a determination result as to whether the subject (person) shown in the image is a living body. That is, it outputs a determination result as to whether the subject is a "disguise" using a photograph, a display terminal, or the like.

[0017] Note that the color information used for biological determination and the color information used for skin color change are not limited to H (hue) in the HSV color space, and may be G in RGB or those in other color spaces.

[0018] Next, we will explain the processes performed by each functional unit using the flowchart in Figure 3. In the following explanation, we will omit the notation of each process (step) by prefixing it with "S". However, it is not necessary to perform all the steps described in this flowchart.

[0019] In S301, the image processing device 1 performs initialization. Specifically, the skin tone threshold determination unit 203 clears the skin tone threshold setting. Furthermore, the hue change management unit 208 sets the start time for biological detection. In S302, the image acquisition unit 201 acquires an image. In S303, the face detection unit 202 detects a face from the image. In S304, the face detection unit 202 determines whether a face was detected from the image. If no face is detected, the process proceeds to S314, the termination process is executed, and the flow ends. In the termination process, if no face is detected in the first image, the result output unit 210 outputs a result indicating that biological detection was not started. Alternatively, it may notify that no person to be detected was found. If no face is detected in an image during the detection process for a certain person, it outputs a result indicating that the detection process has been interrupted. Note that if a series of images in which the presence of a face is guaranteed are input, processes S304 and S314 may be skipped.

[0020] If a face is detected by the face detection unit 202 in S304, the skin tone threshold determination unit 203 determines in S305 whether a skin tone threshold has been set. If a skin tone threshold has not been set, the process proceeds to S306 to set the skin tone threshold. The skin tone threshold defines the range that HSV can take, based on the color information in the HSV color space obtained from the RGB values ​​of pixels in the skin tone region included in the face image. Here, H represents hue, V represents lightness, and S represents saturation. The method for setting the skin tone threshold may be determined using the mean and variance of each RGB signal in the region where the face was detected, or it may be the converted HSV region. Note that the skin tone threshold is set for each detected face. Here, for the sake of simplicity, the face of one person is detected and the skin tone threshold is set for that person. If it is determined in S305 that a skin tone threshold has been set, the process proceeds directly to S308 without going through S306.

[0021] Next, the process proceeds to S308, where the hue calculation unit 207 detects the skin-colored region using the determined skin-color threshold and measures the number of pixels in the skin-colored region. That is, it counts the number of pixels in the image whose HVS color space information falls within the range of the skin-color threshold.

[0022] Next, the process proceeds to S309, where the hue calculation unit 207 calculates the average hue value based on the color information of the pixels in the skin tone region.

[0023] In S310, the update determination unit 206 determines whether or not to update the threshold based on the color information. Specifically, it uses the change in the number of skin-colored pixels obtained in S308 and the change in the average hue value obtained in S309 to determine whether the set skin-color threshold is appropriate. Details of this process will be explained later using Figure 4.

[0024] If it is determined in S310 that the threshold does not need to be updated (qualified), the process proceeds to S311. In S311, the biological determination unit 209 determines whether the subject is a living person or not based on the time-series images taken over a predetermined period and the set threshold. That is, it performs biological determination processing (biological determination). The biological determination processing may be performed using a biological score that indicates the likelihood of the person in the image being a living person. The biological determination processing determines that the person is a living person if the biological score is above a certain threshold value, and determines that the person is not a living person if the score is below a certain threshold value. The biological score is estimated, for example, based on the pulse rate estimated based on the average hue value included in the color information. Specifically, the biological determination processing may perform pulse rate estimation processing. Pulse rate estimation processing is a method of estimating the pulse rate of the subject based on the color information obtained from the time-series images. For example, the amount of change over time of the average hue value is transformed using a Fast Fourier Transform, and the frequency of the peak in the spectrum is obtained as the pulse rate. The biological score is calculated by dividing the power of the peak frequency by the average power of the other frequencies. If the biological score is above a threshold, the subject is determined to be alive; if it is below a threshold, it is determined to be non-living. Another example of biological determination processing is blood flow detection using the amplitude and time variation of multiple vibrating color information. This method of estimating blood flow can be determined with fewer frames than pulse rate estimation. In this case, if the required time for biological determination to be made in S312 has not elapsed, the biological score at a certain point in the time-series image may be output. Furthermore, after the required time has elapsed, if the statistical value of the biological score output within the required time (e.g., maximum value, mean value, or median value) is above a certain value, the subject may be determined to be living.

[0025] In S312, the result output unit 209 calculates the elapsed time from the start time of the biometric determination and determines whether the required time for biometric determination has elapsed. If it has elapsed, in S313, the result output unit 209 outputs the determination result of the biometric determination process for the person in the series of time-series images and terminates the process. If it has not elapsed, the process returns to S302 and repeats the biometric determination process. Note that the biometric determination used in this example is performed even if the time change amount of the average hue value does not reach the required time (PT1), but the reliability is low when the required time is not reached and high when the required time is reached. If biometric determination can only be performed after the required time has elapsed, the order of S311 and S312 is reversed.

[0026] If the skin tone threshold is determined to be inadequate in S310, the image processing device 1 reinitializes in S307. That is, the skin tone threshold setting is cleared, the start time of the biological detection is updated, and the amount of time change in the average hue value up to that point is excluded from the biological detection process. Then, an image is acquired in S302 and the process is repeated. Immediately after reinitialization, the skin tone threshold setting is cleared, so the skin tone threshold is reset, i.e., updated, with the next image. In other words, in S306, the skin tone threshold is updated based on the image acquired after the image in which the skin tone threshold was determined to be inadequate. In this way, by determining whether the threshold is appropriate for each image, the accuracy of biological detection can be prevented from decreasing even when the shooting conditions change.

[0027] In this flowchart, the system outputs the results of the biometric detection and terminates after the required time has elapsed. However, it may be configured to continue outputting the results of the impersonation detection. Alternatively, the system may prompt the user to input whether or not to continue the detection process when the impersonation detection result is output. Furthermore, the output unit may output the estimated pulse wave rate instead of the biometric detection result. This allows the pulse rate to be used not only for biometric detection but also as a factor in determining the subject's health status.

[0028] Figure 4 is a flowchart showing an example of the skin tone threshold eligibility determination process in S310 of Figure 3. In this flowchart, the skin tone threshold is determined to be unsuitable if there is a change greater than a predetermined amount in both the skin tone pixel count and the average hue value in a time shorter than the time required for the biological determination process (hereinafter referred to as the change observation time). Here, the time required for biological determination is PT1(s), and the time required for threshold update determination (change observation time) is PT2(s). When performing biological determination, PT1>PT2>0. However, this does not apply when pulse rate estimation is performed, as the change observation time may be longer or shorter than the time required for the biological determination process.

[0029] In S401, the update determination unit 206 acquires the number of pixels (P0) in the skin-colored region of the image to be compared. As an example, an image taken a predetermined period (PT2) before the current time (let's call it the first time) is used for comparison. If the predetermined period (PT2) has not elapsed since the start of the biological determination, the number of pixels in the skin-colored region of the image at the start of the biological determination (T=0) is acquired. Note that an image in which the skin-colored region has been determined based on a pre-set threshold is used.

[0030] In S402, the update determination unit 206 obtains the number of pixels (P1) in the skin-colored region of the current image.

[0031] In S403, the update determination unit 206 determines whether the change in the number of pixels in the skin-colored region of the image acquired during a predetermined period (PT2) (Pd) is greater than a first predetermined value (skin-colored pixel count change threshold PdT). The change in the number of pixels in the skin-colored region of the image acquired before the predetermined period (PT2) (Pd) is calculated. An example of the calculation formula is as follows.

[0032] Pd = abs(P0 - P1) / min(P0, P1) The reason we're using absolute values ​​for the amount of change here is that even if the skin-colored area increases, there's a possibility of noise other than skin color changes due to blood flow, such as the disappearance of shadows on the face or the background being confused with the skin.

[0033] If the change in the number of pixels is less than a predetermined value, the process proceeds to S408 and terminates, determining that the skin tone threshold is acceptable. If, in S403, it is determined that the change in the number of pixels is greater than a predetermined value, the process proceeds to S404.

[0034] In S404, the update determination unit 206 obtains the average hue value of the skin-colored region of the image to be compared. Here, it obtains the average hue value (H0) of the image from a predetermined period (PT2) prior to the current time (first time). If the predetermined period (PT2) has not elapsed since the start of the biological determination, it obtains the average hue value at the start of the biological determination. In S405, the update determination unit 206 obtains the average hue value (H1) of the skin-colored region of the image obtained at the first time.

[0035] In S406, the update determination unit 206 determines whether the change amount (Hd) is greater than a second predetermined value (hue change threshold HdT). It calculates the change amount Hd compared to the average hue value of the skin tone region of the image acquired before a predetermined period (PT2). An example of the calculation formula is as follows.

[0036] Hd = abs(H0 - H1) If the change in the average hue value is less than a predetermined value, the process proceeds to S408 and terminates S310, determining that the skin tone threshold is acceptable. If it is determined in S406 that the change in the average hue value is greater than a predetermined value, the process proceeds to S407 and terminates S310, determining that the skin tone threshold is unacceptable.

[0037] In this example, the amount of change is compared based on the difference in color information between the image at the current time (first time) and the image from a predetermined period (PT2) earlier, but other methods are also acceptable. For example, the amount of change may be obtained based on the color information of the image acquired at the start of the predetermined period (PT2) and the color information of the image acquired at the end of the predetermined period (PT2). Alternatively, the amount of change may be obtained based on the difference between the maximum and minimum number of pixels in the skin-colored region among the color information of the time-series images acquired during the predetermined period (PT2). Regarding hue, it may also be determined whether the change in the average hue of the skin-colored region is greater than or equal to a second predetermined value, based on the average hue of the skin-colored region in the time-series images at the start of the predetermined period (PT2) and the average hue of the skin-colored region in the time-series images at the end of the predetermined period (PT2).

[0038] This section describes variations in reinitialization when the skin tone threshold is determined to be unsuitable in S307. Here, we explain a method to suppress the decrease in accuracy of biological detection by changing the target period for biological detection. Even if the shooting conditions change, if time passes and the conditions return to their original state, and the various evaluation values ​​calculated from the color information become close to those at the start of biological detection, it is highly likely that the skin tone threshold has returned to a suitable state. However, for images acquired during periods of large changes in color information, the change in the average hue value contains noise other than skin tone changes due to blood flow, so excluding them from the biological detection processing target may improve detection accuracy. In this case, the reinitialization process in S307 does not update the skin tone threshold, but updates the start time of biological detection. For example, in reinitialization, the start time of the biological detection process is pushed back by a certain period (for example, the number of frames from the start to the time when reinitialization occurs). The amount of time change in the average hue value up to that point may be excluded from the biological detection processing target. Updating the start time of biological detection means resetting the time counter for n frames of processing to 0. This means extending the required time by n. It also means excluding the amount of time variation in the average hue value up to that point from the processing target.

[0039] Figure 5 compares the processing steps when processing continues even if the skin tone threshold becomes unsuitable, and when reinitialization is performed.

[0040] Figure 5A shows the acquired images. As an example, it illustrates a situation where the brightness changes and becomes brighter immediately before biological detection is performed. In this case, the camera's automatic correction works with a time lag according to the amount of change in the shooting environment. For example, the camera determines that the brightness of the first image is excessive and controls the image to reduce its brightness. Images 501-503 are images taken during shooting and show how the image brightness changes due to the camera's automatic correction. Note that the person in the images is the same person, and the shooting environment is assumed to have not changed significantly after the skin tone threshold was determined and the environment became brighter.

[0041] Figures 5B and 5C show examples where processing continues even when the skin tone threshold becomes unsuitable. Figure 5B shows the detection of skin tone areas using the skin tone threshold set in the first image (501) of the acquired images. Images 504, 505, and 506 show the detection results of skin tone areas in images 501, 502, and 503, where the white areas are determined to be skin tone. In image 504, the skin tone area determined by the threshold set in image 501 includes the entire face, but in images 505 and 506, the skin area is smaller.

[0042] Figure 5C shows graphs of the time evolution of color information and biological detection. Graph 507 represents the number of skin tone pixels, graph 508 represents the average hue, and graph 509 represents the biological score.

[0043] Time axis 510 indicates the required time of 5 seconds for biometric detection in this case. When pulse rate estimation is performed and the result is used for biometric detection, approximately 5 seconds is required to achieve accuracy. The required time will differ and is not limited to 5 seconds when biometric detection is performed using the correlation of the amplitude and time change of multiple vibrating color information without determining the pulse rate.

[0044] In graph 507, the number of skin-colored pixels decreases up to point 511, 0.3 seconds after the start. In graph 508, the average hue decreases up to point 512, 0.3 seconds after the start. In graph 509, the biological score at point 513, after 5 seconds, is 2.66, which does not reach the arbitrary threshold of 3.4 (shown by the dotted line in 514) for determining a living organism, and is therefore determined to be a spoof. This indicates that images 502 and 503 are in the process of undergoing a series of automatic corrections from image 501, resulting in a failure to extract the skin-colored region and consequently a decrease in the accuracy of estimating the amount of change in color information.

[0045] Figures 5D and 5E show an example of reinitialization when skin tone becomes unsuitable. In this example, reinitialization is performed if there is a change of 16% in the number of skin tone pixels or a change of more than 2 in the average hue value during a change observation time of 3 seconds. Here, the images to be judged are similarly to images 501-503 in Figure 5A. Images 504, 505, and 513 in Figure 5D are the results of extracting the skin tone region.

[0046] Figure 5D shows the skin tone region detection results, with 504 and 505 being the same as those in Figure 5B. In this case, the biological detection process is reinitialized midway through, so a different skin tone threshold is set for image 503 compared to image 501. Image 513 shows the skin tone region extracted from image 505. It can be seen that the skin tone region is larger than that of 506 in Figure 5B.

[0047] Figure 5E is a graph showing the time-dependent changes in skin tone and biological detection. Graph 514 shows the number of skin tone pixels, graph 515 shows the average hue, and graph 516 shows the biological score. Graph 514 shows the average number of pixels decreasing to 75% at point 517, and graph 515 shows the average hue decreasing by 2.1 at point 518. When both the change in skin tone pixels and the average hue meet the conditions, as in this case, reinitialization is performed. Specifically, the skin tone threshold setting is cleared, the start time of biological detection is updated, and the amount of time-dependent change in the average hue up to that point is excluded from the biological detection process. The next image shows the resetting of the skin tone threshold. Graph 513 shows the detection result of the skin tone region using this reset. Graph 514 shows the number of skin tone pixels approaching its original value after point 517, and graph 515 also approaches its original value after point 518. Graph 516 shows the biological score becoming 0 at point 519 because the amount of hue change up to that point is excluded from the calculation. The biological detection process takes 0.3 seconds, at which point the start time is reset, and the result is output at 5.3 seconds. The biological score for 521 is 3.69, which is greater than the dotted threshold of 3.4 for 522, so it is determined to be a living organism.

[0048] If re-initialization occurs, the time required for impersonation will be added to the initialization time when outputting the result. However, if the biometric score reaches a sufficient value before the added time is reached, the result may be output at that point. This reduces the time required for the judgment process.

[0049] As described above, by reinitializing the skin tone threshold for extracting skin tone regions when it becomes inappropriate, biological detection can be efficiently performed even when the shooting conditions change.

[0050] <Embodiment 2> In Embodiment 1, re-initialization was performed only when both the change in the number of skin color pixels and the change in the average hue value were greater than a predetermined value. In this embodiment, an example of changing the re-initialization condition and the interval at which re-initialization is executed will be described. As the re-initialization condition, in Embodiment 1, re-initialization was determined only when the change amounts of both the number of skin color pixels and the average hue value were large, but re-initialization may be performed when the change amount of either one is greater than a predetermined value. Also, if the frequency of re-initialization becomes too high, the required time for the biological determination process will become long, so an example of providing a waiting time for re-initialization according to the situation of the change amount of color information will be described.

[0051] FIG. 6 is a flowchart for explaining the processing executed by the image processing apparatus 1 in Embodiment 2. It is assumed that the image processing apparatus 1 has the same hardware configuration and functional configuration as in Embodiment 1. In FIG. 6, the processing of S401, S402, S404, and 405 is the same as the flowchart of FIG. 4, so the description thereof will be omitted. In S601, when the update determination unit 206 determines that the change amount of the number of skin color pixels is greater than the skin color pixel change threshold PdT, it determines as ineligible in S604. If it is determined otherwise in S601, the change amount of the average hue value is examined, and in S604, if the update determination unit 206 determines that the change amount of the average hue value is greater than the hue change threshold HdT, it determines as ineligible in S604. Otherwise, it determines as eligible in S603.

[0052] Also, in order to combine the processing of FIGS. 4 and 6, the change observation time (PT₂) and the magnitude of the first predetermined value (skin color pixel change threshold PdT) used for determining the change amount may be as follows.

[0053] Condition 1) When, within the first change observation time (PT₂), both the change in the number of skin color pixels and the change in the average hue value are greater than the first change threshold (the first predetermined value or the second predetermined value).

[0054] Condition 2) When, after elapse of a second change observation time (PT₃ < PT₂) shorter than the first change observation time, either the change in the number of skin color pixels or the change in the average hue value is greater than the second change threshold (the third predetermined value) greater than the first change threshold.

[0055] Typically, both changes in skin tone pixel count and changes in the average hue are examined, and if there is a sudden and large change, one of the changes is then monitored.

[0056] Furthermore, in this embodiment, when reinitialization is performed, a waiting time is provided for the next reinitialization. The update determination unit 209 of the image processing device 1 determines that if the biological detection process is initialized, it will continue the biological detection process without performing reinitialization until the waiting time has elapsed. For example, if reinitialization is performed at time T1, reinitialization will not be performed during the waiting time IT. Specifically, for images acquired between time T1+IT, the determination process in S310 in Figure 3 is skipped, and the process proceeds to S311. As an exception, if the amount of change in color information is quite large and the effect of image quality adjustment cannot be ignored, reinitialization may be performed even during the reinitialization waiting time. In addition, the determination process in S310 may be changed in combination with the above conditions, depending on the condition of the amount of change in color information and the waiting time. A specific example is described below.

[0057] Figure 7 compares the processing steps when the process continues even if the skin tone threshold becomes unsuitable, and when reinitialization is performed.

[0058] Figure 7A shows the acquired image set. This is an example where the brightness and lighting color changed immediately before biometric analysis was performed. The camera's automatic correction function activated, determining that the initial image was too dark and changing it to a brighter image. The change in lighting color also caused a change in skin tone. Images 701-704 are extracted images taken during the process of this change.

[0059] Figure 7B shows the detection of skin-colored areas using the skin-color threshold set in the first image acquired. The detection results for images 701, 702, 703, and 704 are 705, 706, 707, and 708, with the white areas being the regions determined to be skin-colored. In image 705, the skin-colored region determined by the threshold set in image 701 includes many shadowed areas. In images 706 and 707, the skin area increases and the shadowed area decreases, but in image 708, the threshold can no longer cope with the color change, and non-skin-colored areas appear on the face.

[0060] Figure 7C is a graph showing the time-dependent changes in skin tone and biometric detection. Graph 709 represents the number of skin tone pixels, graph 710 represents the average hue, and graph 711 represents the biometric score. Graph 716 indicates the time required for biometric detection, which is 5 seconds. In graph 709, the number of skin tone pixels increases from the start to point 712, which is 0.3 seconds later, and then decreases slightly to point 713.

[0061] In the graph of average hue for 710, the average hue increases up to 714, where 1 second has elapsed from the start, and then decreases down to 715. In the graph for 711, the bioscore for 717, where 5 seconds have elapsed, is 2.12, which falls short of the threshold of 3.4 shown by the dotted line for 718, and is therefore judged to be an impersonator.

[0062] Figures 7D and 7E show an example of reinitialization when the skin tone becomes unsuitable. In this example, reinitialization is performed under the following two conditions:

[0063] Condition 1) During a change observation time of 3 seconds (waiting time 1), if the number of skin-tone pixels is greater than 16%, and the average hue is greater than 2, then both changes occur. Condition 2) When the change observation time (waiting time 2) of 1 second has elapsed, if either the skin tone pixel count is greater than 30% or the average hue is greater than 3, Figure 7D shows the skin tone region detection results, with 705, 706, and 707 being the same as those in Figure 7B. Figure 7E is a graph showing the time changes in skin tone and biological detection. Graph 720 shows the number of skin tone pixels, graph 721 shows the average hue value, and graph 722 shows the biological score. Graph 720 shows the average number of pixels, which has increased to 90% at 723 after 0.3 seconds, and graph 721 shows the average hue value, which has increased by 4.9 at 725 after 0.3 seconds. Assuming that both the change in skin tone pixels and the average hue value meet the conditions at 0.3 seconds, a reinitialization is performed. The skin tone threshold setting is cleared, the start time of biological detection is updated, and the amount of time change in the average hue value up to that point is excluded from the biological detection processing. The skin tone threshold is reset in the next image. Graph 718 shows the skin tone region detection result using the reset skin tone threshold. Graph 722 shows the biological score, which has dropped to 0 at 726 after 0.3 seconds.

[0064] After reinitialization in 0.3 seconds, the average number of pixels in 720 initially decreases, but the change from there to 724 at 1.3 seconds is not significant. The average hue value of 721 changes significantly even after reinitialization. The change in the number of skin tones in 720 at 1.3 seconds, 1 second after reinitialization, is 9% of that of 724, but the change in the average hue value of 721 is 4.2. Reinitialization is performed assuming that the change in the average hue value 1 second after reinitialization satisfies the condition. The skin tone threshold is reset in the next image. The detection result of the skin tone area using this is 719. The biological score of 722 also decreases to 0 at 728 after 1.3 seconds. The change in the average hue value of 721 after reinitialization becomes smaller. The required time for biological detection is 1.3 seconds and the start time is reset, so the biological detection result is output at 6.3 seconds for 729. The biological score of 730 for 722 is 3.48, which is greater than the threshold of 3.4 shown by the dotted line for 731, so it is determined to be a living organism.

[0065] In this way, by changing the conditions for the amount of change in color information and the period during which reinitialization is performed, the biological detection process can be performed more efficiently. In this embodiment, condition 1: If the change in both the number of skin-color pixels and the average hue value is greater than a certain value, reinitialization is performed any number of times within the change observation time. Condition 2: If the change in either the number of skin-color pixels or the average hue value is greater than a certain value, reinitialization is performed when the change observation time has elapsed. Both conditions 1 and 2 may be performed any number of times within the change observation time, or they may not be performed until the change observation time has elapsed.

[0066] As described above, by combining re-initialization conditions, biological detection can be efficiently performed even when the shooting conditions change.

[0067] <Embodiment 3> In embodiments 1 and 2, reinitialization is performed when the change in the number of skin-color pixels and the average hue value exceeds a certain value, thereby extending the time required for impersonation detection. However, depending on the elapsed time and the biological detection result, the time required for detection may be shortened without reinitialization, and the output of the detection result may be brought forward. That is, the storage unit stores the biological score for the time-series image at a predetermined time. If it is determined that the skin-color threshold should be updated before the time required for biological detection processing has elapsed, the biological detection processing determines whether the person is a living being based on the stored score if the first time has elapsed. If the first time has not elapsed, the determination of whether the person is a living being is made after updating the threshold. Note that the first time is less than the time required for biological detection processing.

[0068] Figure 9 is a graph showing the time evolution of skin tone change and biometric assessment in an example where automatic correction works slowly, and the changes in skin tone pixel count and average hue are gradual. The graph for skin tone pixel count is 901, the graph for average hue is 902, and the graph for biometric score is 903. The point where the skin tone threshold is not met, due to the gradual change in skin tone pixel count and average hue, is at line 904, 4.9 seconds before the biometric assessment time of 5 seconds. At this point, the biometric score 910 is 4.6, which exceeds the threshold of 3.4 shown by the dotted line 911. Rather than reinitializing and extending the assessment time by 5 seconds to 9.9 seconds, it is better to output the biometric assessment result at this stage.

[0069] Figure 8 is a flowchart illustrating the processes performed by the image processing device 1 in Embodiment 3. The process of canceling the unsuitability judgment of the skin tone threshold is performed after processes S407 and S604 in Figures 4 and 6, which determine the skin tone threshold to be unsuitable. The image processing device 1 has the same hardware and functional configuration as in Embodiment 1.

[0070] In S801, the biological determination unit 209 acquires a biological score. In S802, if the biological determination unit 209 finds that the biological score is greater than the biological threshold * RS1 (RS1 > 1), it proceeds to S803. In S803, the biological determination unit 209 acquires the elapsed time. In S804, the biological determination unit 209 acquires the required time for biological determination. In S805, if the elapsed time is greater than the required time * RT (RT < 1), it proceeds to S807. In S807, the biological determination unit 209 shortens the required time to the elapsed time. In S808, the biological determination unit 209 changes to a skin color threshold suitability determination and terminates.

[0071] In S802, the biological determination unit 209 determines that the value is less than or equal to the biological threshold * RS1 (RS1 > 1), or in S805, the elapsed time is less than or equal to the required time * RT, and if so, it determines in S806 that the device is ineligible and terminates.

[0072] If the flowchart determines that the process is eligible, the required time is shortened to the elapsed time, and it is determined that the required time has elapsed at S312 in Figure 3, so the result output is brought forward.

[0073] Note that the skin tone threshold in S310 is set before the biological determination in S311, and the biological score used in S310 is the value from the previous image. Alternatively, the order of S310 and S311 can be reversed to use the current biological score.

[0074] While a biometric score is used to determine the reliability of the biometric determination results, other indicators of reliability may also be used. For example, if a binary classifier for living / non-living organisms is used and the likelihood cannot be measured, the number of living / non-living organisms in consecutive determination results can be compared to determine if there is a significant difference.

[0075] As described above, for example, it is possible to suppress cases where the biometric detection process is reinitialized when 80% of the required time has elapsed, thereby reducing the time required for the detection process and enabling efficient execution of the biometric detection process.

[0076] <Embodiment 4> This embodiment describes an example of dynamically changing the threshold for the amount of change in color information used to determine whether reinitialization is necessary. Generally, setting a small threshold for the amount of change in color information allows for flexible biological detection processing in response to the camera's automatic correction, but it may also lead to unnecessary reinitialization due to factors such as facial movement. Conversely, setting a large threshold for the amount of change in color information may introduce noise other than skin color changes due to blood flow changes into the biological detection process. Therefore, in this embodiment, after reinitialization is performed, the threshold for the amount of change in color information (PdT or HdT) is set to a value greater than the initial value (PdT1 > PdT or HdT1 > HdT). This suppresses reinitialization. Furthermore, if it is determined that reconsideration of biological detection is necessary after a certain period of time has elapsed, the processing time for biological detection is extended.

[0077] Figure 11 shows an example of the time-dependent changes in skin tone and biological score in Embodiment 4. Graph 1101 shows the number of skin tone pixels, graph 1102 shows the average hue value, and graph 1103 shows the biological score. In this example, the following two conditions are used as reinitialization conditions.

[0078] Condition 1) If, within a predetermined time of 3 seconds, both the number of skin-tone pixels and the average hue value change to greater than 2. Condition 2) When a predetermined time of 1 second has elapsed, either the skin tone pixel count has changed by 30%, or the average hue has changed to a value greater than 5. The change in the number of skin-tone pixels from 1101 to 1105 after 0.3 seconds is 6%, and the change in the average hue of 1102 from the start to 1106 is 3.5, so condition 1 is not met. The change in the number of skin-tone pixels from 1101 to 1107 after 1 second is 5%, and the change in the average hue of 1102 to 1108 after 1 second is 3.23, so condition 2 is also not met. The biological score of 1109 after 5 seconds has elapsed is 3.18, which is lower than the threshold of 3.4 shown by the dotted line of 1112, and will be judged as non-living if left as is.

[0079] Figure 10 is a flowchart of the processes executed by the image processing device 1 in Embodiment 4. The process of determining whether to extend the time for biological detection can be inserted after the decision to proceed is made as yes at S312 in the flowchart of Figure 3. If the required time is extended, the process returns to 302 in Figure 3; otherwise, the process proceeds to the result output at S313. The image processing device 1 is similar to the hardware and functional configuration of Embodiment 1, except for some processes.

[0080] If reinitialization does not occur and the required time of 5 seconds for impersonation has elapsed, the process shown in Figure 10 begins. In S1001, the biometric determination unit 209 acquires a biometric score. In S1002, the biometric determination unit 209 determines whether the biometric score warrants reconsideration. For example, it determines whether the biometric score is smaller than a standard value. Cases where the biometric score is within a predetermined standard range may be included, or cases where the biometric score is too low may be excluded. If it is determined that the biometric determination does not warrant reconsideration, the process ends. In Figure 11, the biometric score at 1104 is 3.18, which is slightly smaller than the threshold of 3.4 shown by the dotted line at 1112, so in this example, it is determined that reconsideration is warranted.

[0081] When it is determined that the biological score is worth reconsidering the biological determination, in S1103, the biological determination unit 209 acquires the history of color information in the skin color region used for the biological determination. The values from the number of skin color pixels of 1101 in FIG. 11 to 1104 indicating the required time for biological determination from the start of the hue average value of 1102 are acquired. In S1104, the biological determination unit 209 sets the value at the start position as a comparison value. Also, at this time, the threshold value of the change amount of color information (PdT1 > PdT or HdT1 > HdT) is changed to a smaller value (PdT2 < PdT < PdT1 or HdT2 < HdT < HdT1). In this example, the change threshold value of the hue average value of condition 2 is lowered from 5 to 3.

[0082] In S1105, the biological determination unit 209 performs a process of comparing the values of the color information history from the start time to the elapse of a predetermined time as the current value with the aforementioned comparison value. In S1106, the biological determination unit 209 determines whether the change amount is greater than PdT2 by comparing the comparison value and the current value. This process can be substituted by the processes of FIGS. 4 and 6. The case of threshold disqualification determination is regarded as a large change, and the case of qualification is regarded as not a large change. In the example of FIG. 11, when the threshold value (HdT1) regarded as a large change is 3.0 in terms of the hue average value, it is determined that there is a large change in the hue average value 1 second after condition 1.

[0083] If it is determined as a large change in S1006, in S1007, the biological determination unit 209 records that time. In S1008, the biological determination unit 209 resets the hue average value and the number of skin color pixels at that time as the comparison value and continues the process. If multiple locations with large changes are found, the last time is recorded.

[0084] Even if it is determined as no change in S1006, when the change measurement time has elapsed, in S1008, the biological determination unit 209 resets the comparison value to the value before the predetermined change measurement time.

[0085] After processing all the history of changes in the required time, the process proceeds to S1009, where the biological determination unit 209 determines whether a time record exists. If so, the process proceeds to S1011, where the biological determination unit 209 extends the required time for biological determination by the recorded time and terminates. The amount of time change in the average hue used for biological determination remains constant even if the required time for impersonation is extended. By extending the required time for biological determination, parts containing noise are excluded from the biological determination, and the correct result is obtained. In Figure 11, the biological score 1111 for 1103 at an extended determination time of 1110 of 6.0 seconds is 4.82, which is greater than the threshold of 3.4 shown by the dotted line in 1112, and the result is that of a living being.

[0086] As described above, by dynamically changing the threshold color information used as a condition for reinitialization, the biometric detection process can be performed more efficiently.

[0087] Furthermore, all image data used for biometric identification can be saved, and if it is determined that a reassessment of the biometric identification is necessary, the reinitialization threshold can be lowered and the biometric identification can be performed again.

[0088] <Embodiment 5> This embodiment describes an example of a GUI in the biometric detection process. The image processing device 1 described above may display the time progress from the start to the end of the biometric detection process to the user. Furthermore, if reinitialization occurs, the time progress will be reversed. If reinitialization is performed, the cause may also be displayed to the user. This improves convenience as the user can understand the progress of the biometric detection process and the factors that are hindering it.

[0089] Figure 12 is an example of the presentation. The image processing device 1 outputs the judgment result of the biological detection process and the time progress of the biological detection process via the result output unit 209. Figure 12A is an example of the progress bar display when reinitialization occurs. 1202 is when biological detection has started, and 1202 and 1203 are when biological detection is in progress. Reinitialization occurs in the middle of the display at 1303, and 1204 indicates that the 5-second re-detection has started due to the resetting of the skin color threshold, with the message "Impersonation re-detection started (skin color threshold reset)". 1205 displays the progress status within the required time for re-detection. Alternatively, the total display time range when re-detection occurs may be set to reinitialization time + required time, and the progress of the progress bar at 1204 may be shown as elapsed time / (reinitialization time + required time) instead of resetting to 0.

[0090] Figure 12B shows an example of the progress bar display when the required time for biological assessment is shortened. 1206 is when the biological assessment has started, and 1207 is when the biological assessment is in progress. The required time is shortened midway through the display at 1207, and at 1208, it shows "Biological assessment completed (assessment time shortened)," indicating that the required time has been shortened and the result output has been brought forward.

[0091] Figure 12C shows an example of how the progress bar is displayed when the required time for biological detection is extended. 1209 is when biological detection has started, and 1210 is when biological detection is in progress. If the required time is reached while 1210 is displayed and an extension of the detection time is decided, "Biological detection complete" is not displayed, and instead, "Biological detection in progress (detection time extended)..." is displayed at 1211, and (required time - extended time) / required time is displayed as the current progress. In addition, the cause of the change in progress time may be clearly indicated as a change in the number of skin tone pixels or a change in the average hue value.

[0092] (Other embodiments) The present invention can also be realized by performing the following process: supplying software (program) that realizes the functions of the above-described embodiment to a system or device via a data communication network or various storage media; and having the computer (or CPU, MPU, etc.) of that system or device read and execute the program. Alternatively, the program may be recorded on a computer-readable recording medium and provided.

[0093] Furthermore, for the face detection unit 102 and other processing units described above, a pre-trained model developed through machine learning may be used instead. In this case, for example, multiple combinations of input and output data for the processing unit are prepared as training data, knowledge is acquired from these through machine learning, and a pre-trained model is generated that outputs output data for the input data based on the acquired knowledge. The pre-trained model can be configured as, for example, a neural network model. The pre-trained model then operates in cooperation with a CPU or GPU, etc., as a program to perform processing equivalent to that of the processing unit, thereby performing the processing of the processing unit. Furthermore, the pre-trained model may be updated after a certain amount of processing as needed. [Explanation of Symbols]

[0094] 200 Biological Identification Devices 201 Image Acquisition Unit 202 Face detection unit 203 Skin Tone Threshold Determination Unit 204 Skin tone area detection unit 205 Skin tone change measurement section 206 Update determination section 207 Hue calculation section 208 Hue Change Management Department 209 Biological Testing Unit 210 Result Output Section

Claims

1. An image processing device that determines whether a person in an image is a living being, A determination means for determining a threshold that indicates the range of skin color in a time-series image acquired over a predetermined period of time, An acquisition means for acquiring color information about the skin-colored region included in the time-series image based on the threshold, An image processing apparatus characterized by having determination means for determining whether or not to continue the biological detection process that determines whether or not the person in the time-series image is a living being based on the color information, and determining whether or not to update the threshold used in the biological detection process based on the amount of change in the color information.

2. An image processing device for determining whether a person in an image is a living organism, A determination means for determining a threshold that indicates the range of skin color in a time-series image acquired over a predetermined period of time, An acquisition means for acquiring color information about the skin-colored region included in the time-series image based on the threshold, The system includes a determination means for determining whether or not to continue the biological detection process that determines whether the person in the time-series image is a living being, based on the aforementioned color information. The image processing apparatus is characterized in that, when updating the threshold used in the biological determination process, the determination means updates the threshold based on an image acquired after the time-series image.

3. The image processing apparatus according to claim 1, characterized in that the determination means determines to change the time-series image that is the target of the biological determination process based on the amount of change in the color information.

4. The aforementioned color information includes the number of pixels included in the skin tone region or the hue value of the pixels included in the skin tone region. The image processing apparatus according to any one of claims 1 to 3, characterized in that the determination means determines to initialize the biological detection process when the change in at least one of the number of pixels included in the skin-color region or the hue value of the pixels included in the skin-color region during the predetermined time is greater than or equal to a predetermined value.

5. The image processing apparatus according to claim 4, characterized in that the determination means determines whether the amount of change in the number of pixels in the skin-colored region is greater than or equal to a first predetermined value, based on the maximum and minimum values ​​of the number of pixels in the skin-colored region among the time-series images acquired at the predetermined time.

6. The image processing apparatus according to claim 4, characterized in that the determination means determines whether the amount of change in the number of pixels in the skin-colored region of the time-series image is greater than or equal to a first predetermined value, based on the number of pixels in the skin-colored region of the time-series image at the start of the predetermined time and the number of pixels in the skin-colored region of the time-series image at the end of the predetermined time.

7. The determination means determines the average hue of the skin-colored region of the time-series image at the start of the predetermined time and the average hue of the skin-colored region of the time-series image at the end of the predetermined time. The image processing apparatus according to any one of claims 4 to 6, characterized in that it determines whether the amount of change in the average hue value of the skin-color region is greater than or equal to a second predetermined value.

8. The image processing apparatus according to claim 4, characterized in that the determination means determines whether the amount of change in the number of pixels in the skin-color region included in the color information is greater than a first predetermined value in a predetermined time shorter than the predetermined time.

9. The image processing apparatus according to any one of claims 4 to 8, characterized in that the determination means changes the predetermined value when the biological determination process is initialized.

10. The image processing apparatus according to any one of claims 1 to 9, characterized in that the biological determination process estimates the pulse rate of the person based on the amount of change in the color information.

11. The image processing apparatus according to any one of claims 9 to 10, wherein the biological determination process estimates a score indicating the likelihood of a person being a living organism based on the color information, determines that the person is a living organism if the score is above a certain value, and determines that the person is not a living organism if the score is below a certain value.

12. The image processing apparatus according to claim 11, characterized in that the score is estimated based on the pulse rate estimated based on the average hue value included in the color information.

13. The system further includes recording means for recording the score for the time-series images during the predetermined time, and if it is determined that the threshold should be updated before the required time for the biological determination process has elapsed, The biological determination process, if the first time has elapsed, determines whether the person is a living being or not based on the stored score. The image processing apparatus according to claim 11 or 12, characterized in that, if the first time has not elapsed, it is determined whether or not the person is a living organism after updating the threshold.

14. The image processing apparatus according to any one of claims 1 to 13, characterized in that the determination means determines that if the biological determination process is initialized, the biological determination process should be continued until a waiting time has elapsed.

15. The image processing apparatus according to any one of claims 1 to 14, further comprising output means for outputting the determination result of the biological determination process.

16. The image processing apparatus according to claim 15, characterized in that the output means outputs the time progress of the biological determination process.

17. An image processing method for determining whether a person in an image is a living being, A determination step of determining a threshold that indicates the range of skin color in time-series images acquired over a predetermined period of time, An acquisition step of acquiring color information about the skin-colored region included in the time-series image based on the threshold, An image processing method characterized by comprising: a determination step of determining whether or not to continue the biological detection process that determines whether or not the person in the time-series image is a living being based on the color information; and a determination step of determining whether or not to update the threshold used in the biological detection process based on the amount of change in the color information.

18. An image processing method for determining whether a person in an image is a living being, A determination step of determining a threshold that indicates the range of skin color in time-series images acquired over a predetermined period of time, An acquisition step of acquiring color information about the skin-colored region included in the time-series image based on the threshold, The process includes a determination step of determining whether or not to continue the biological detection process that determines whether the person in the time-series image is a living being, based on the aforementioned color information. The aforementioned determination step is an image processing method characterized in that, when updating the threshold used in the biological determination process, the threshold is updated based on an image acquired after the time-series image.

19. A program for causing a computer to function as one of the means of the image processing apparatus described in any one of claims 1 to 16.

Images