Biological image acquisition assistance device and biological image acquisition assistance method
The biometric image acquisition support device addresses posture-related challenges in fingerprint recognition by guiding users to maintain optimal finger angles, ensuring accurate and consistent biometric data capture for improved authentication.
Patent Information
- Application Number
- PCT/JP2024/017594
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-29
- Filing Date
- 2024-05-13
- Publication Date
- 2024-12-05
AI Technical Summary
Conventional fingerprint recognition methods face challenges in ensuring accurate biometric authentication due to variations in finger posture during imaging, leading to potential focus issues and reduced authentication accuracy.
A biometric image acquisition support device and method that includes an acquisition unit for capturing images, a calculation unit to determine the inclination of biological parts relative to a reference angle, and a control unit generating posture guidance screens to adjust the user's posture for optimal image capture.
This approach supports the acquisition of biometric data suitable for non-contact biometric authentication by ensuring the biological parts are within a predetermined angle range, reducing variations and enhancing authentication accuracy.
Smart Images

Figure JP2024017594_05122024_PF_FP_ABST
Abstract
Description
Biometric image acquisition support device and biometric image acquisition support method
[0001] The present disclosure relates to a biometric image acquisition assistance device and a biometric image acquisition assistance method.
[0002] Patent Literature 1 discloses a method for performing fingerprint recognition using a mobile device, which extracts features of each of a plurality of fingers of a subject, generates a biometric identifier including the extracted features, and stores the generated biometric identifier in a memory. The method for performing fingerprint recognition includes capturing an image of the subject's plurality of fingers using a camera, a storage medium, instructions stored in the storage medium, and a mobile device that executes the instructions, detecting the plurality of fingers depicted in the image using a fingertip detection algorithm, identifying fingertip segments of each of the plurality of fingers, extracting features of the plurality of fingers from each identified fingertip segment, and generating a biometric identifier including the extracted features and storing the generated biometric identifier in a memory.
[0003] Japan Special Table No. 2020-501264
[0004] In the configuration of Patent Document 1, markings or guides are displayed on a display to instruct the user on how to position their fingers to be captured, thereby guiding the user in how to position their fingers. However, because finger sizes vary from user to user depending on factors such as gender and age, there is a possibility that the fingerprints in the images of multiple fingers captured by the method of Patent Document 1 may not be in focus.
[0005] The present disclosure has been devised in consideration of the above-described conventional circumstances, and aims to provide a biometric image acquisition assistance device and a biometric image acquisition assistance method that assist in the acquisition of biometric data that is more suitable for biometric authentication in a non-contact state.
[0006] The present disclosure provides a biometric image acquisition support device including: an acquisition unit that acquires an image of a biological part of a subject; a calculation unit that calculates the inclination of the biological part with respect to a reference angle based on the acquired image; and a control unit that, if it is determined that the inclination of the biological part is not within a predetermined range with respect to the reference angle, generates a posture guidance screen including information regarding the inclination of the biological part and the captured image, and outputs the screen to a monitor.
[0007] The present disclosure also provides a biometric image acquisition support method performed by a computer, which acquires an image of a biological part of a subject, calculates the inclination of the biological part with respect to a reference angle based on the acquired image, and, if it is determined that the inclination of the biological part is not within a predetermined range with respect to the reference angle, generates a posture guidance screen including information regarding the inclination of the biological part and the acquired image, and outputs the screen to a monitor.
[0008] According to the present disclosure, it is possible to assist in obtaining biometric data that is more suitable for biometric authentication in a contactless state.
[0009] FIG. 1 is a block diagram showing an example of the internal configuration of a terminal device according to a first embodiment; FIG. 2 is a flowchart showing an example of an operation procedure of a terminal device according to the first embodiment; FIG. 3 is a diagram explaining an example of an operation procedure of a terminal device according to the first embodiment; FIG. 4 is a diagram explaining an example of a first feedback display screen in a first display example; FIG. 5 is a diagram explaining an example of a second feedback display screen in a first display example; FIG. 6 is a diagram explaining an example of a first feedback display screen and an example of a second feedback display screen in a second display example;
[0010] (Background to the present disclosure) Conventionally, there is a fingerprint registration method in which a fingerprint is registered on a terminal device owned by a user. However, when a user captures an image of their fingerprint themselves, the way in which the user holds their hand over the terminal device, i.e., the posture of the hand held over the terminal device, changes with each image capture. Therefore, the way the fingerprint appears in the captured image during authentication differs from the way the fingerprint appears during registration, which may result in a decrease in the accuracy of biometric authentication using fingerprints.
[0011] Therefore, a conventional method for acquiring a fingerprint image has been proposed, which allows a user to check whether the fingerprint on the fingertip is in focus by displaying an enlarged image of the fingertip captured in a captured image on the screen of a terminal device. However, it has been difficult for a user to determine whether the current way of holding their hand is appropriate for biometric authentication based on the enlarged image displayed on the screen of a terminal device, or to adjust the way their hand is held while viewing the enlarged image.
[0012] Therefore, in the following embodiments, examples of a biometric image acquisition support device and a biometric image acquisition support method that can support the acquisition of biometric data more suitable for biometric authentication in a non-contact state will be described.
[0013] Hereinafter, with reference to the drawings as appropriate, detailed descriptions of embodiments that specifically disclose the configuration and operation of a biometric image acquisition assistance device and a biometric image acquisition assistance method according to the present disclosure will be provided. However, unnecessary detailed descriptions may be omitted. For example, detailed descriptions of well-known matters or redundant descriptions of substantially identical configurations may be omitted. This is to avoid unnecessary redundancy in the following description and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure and are not intended to limit the subject matter recited in the claims.
[0014] (First embodiment) An example of the internal configuration of a terminal device P1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the internal configuration of a terminal device P1 according to the first embodiment.
[0015] A terminal device P1, as an example of a biometric image acquisition support device, captures an image of a user's entire hand or a part used for biometric authentication (hereinafter referred to as a "biometric part") and detects the joint points of the user's biometric part that appear in the captured image. Based on the detected joint points of the user's biometric part, the terminal device P1 calculates the pitch angle and roll angle of the biometric part as the orientation of the user's biometric part that appears in the captured image (i.e., the inclination of the user's biometric part with respect to the light-receiving surface of the image sensor (not shown) of the camera 13). Based on the calculated pitch angle and roll angle of the biometric part, the terminal device P1 generates and outputs a second feedback display screen (see FIGS. 5 to 10 ) for adjusting the orientation (inclination) to a position that allows biometric data suitable for biometric authentication to be acquired.
[0016] In the following description, an example of acquiring biometric data used for biometric authentication using a fingerprint will be described, but the present invention is not limited to this. For example, the biometric authentication may be authentication using veins or palm prints. In the case of biometric authentication using veins, the terminal device P1 captures an image of an area including at least one joint other than the fingertip as a biometric part, and in the case of biometric authentication using palm prints, the terminal device P1 captures an image of the palm as a biometric part and detects the posture (tilt) of the biometric part.
[0017] The terminal device P1 includes a communication unit 10, a processor 11, a memory 12, a camera 13, and a monitor 14. Note that, although the terminal device P1 shown in Fig. 1 is configured separately from the fingerprint database DB1, this is not limiting. For example, the terminal device P1 may be configured integrally with the fingerprint database DB1.
[0018] The communication unit 10, which is an example of an acquisition unit, is configured using a communication interface circuit for transmitting and receiving data. The communication unit 10 is connected to the fingerprint database DB1 so as to be able to communicate with the fingerprint database DB1 wirelessly or via wire, and transmits and receives data.
[0019] The processor 11, which is an example of a calculation unit, a control unit, and a detection unit, is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU"), a Graphics Processing Unit (hereinafter referred to as "GPU"), or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 references programs and data stored in the memory 12 and executes the programs to realize various functions (e.g., a function to detect the inclination of a biological part, a support function to allow the user to adjust the inclination of a biological part, etc.) that are executed to acquire biometric data suitable for biometric authentication.
[0020] The processor 11 associates the captured image (hereinafter referred to as "biometric image") or biometric data determined to be suitable for biometric authentication with user information (e.g., the user's name, gender, date of birth, etc.) and stores (registers) it in the fingerprint database DB1.
[0021] The biometric data referred to here may be a pattern of a biological part extracted from a biometric image (for example, a fingerprint pattern, a vein pattern, or a palm print pattern), or a feature that indicates the individuality of a user extracted from a biometric image.
[0022] The memory 12 includes, for example, a random access memory (hereinafter referred to as "RAM") that serves as a work memory used when executing each process of the processor 11, and a read only memory (hereinafter referred to as "ROM") that stores programs and data that define the operation of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM stores programs that define the operation of the processor 11.
[0023] The camera 13 is configured to have at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a charged-coupled device (hereinafter referred to as "CCD") or a complementary metal oxide semiconductor (hereinafter referred to as "CMOS") solid-state image sensor, and converts an optical image formed on the imaging surface into an electrical signal.
[0024] The monitor 14 is configured using, for example, a Liquid Crystal Display (hereinafter referred to as "LCD") or an organic electroluminescence (hereinafter referred to as "EL"). The monitor 14 outputs and displays images captured by the camera 13 or various screens generated by the processor 11 (such as the first feedback display screen or the second feedback display screen in the first to eighth display examples shown in FIGS. 4 to 10, respectively).
[0025] The fingerprint database DB1 is a so-called storage device, and is configured using a storage medium such as a flash memory, a hard disk drive (hereinafter referred to as an "HDD"), or a solid state drive (hereinafter referred to as an "SSD"). The fingerprint database DB1 stores (registers) biometric images generated by the processor 11 in association with user information. The fingerprint database DB1 may also store (register) biometric data extracted from the biometric images in association with user information. Furthermore, when storing (registering) multiple biometric images, the fingerprint database DB1 may also store (register) each of the multiple biometric images in association with information on the biometric parts corresponding to each of the biometric images (e.g., information on the middle finger and index finger) and user information.
[0026] Next, an example of an operation procedure of the terminal device P1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of an operation procedure of the terminal device P1 according to embodiment 1. Note that in the example of the operation procedure of the terminal device P1 shown in Fig. 2, steps St16 and St19 are not essential and may be omitted.
[0027] The processor 11 acquires a captured image Img (see FIG. 3) of the entire hand or a body part of the user captured by the camera 13 (St11).
[0028] The processor 11 detects the three-dimensional coordinates of each joint point JNT (see FIG. 3 ) based on the acquired captured image Img (St12). The processor 11 estimates the skeletal structure SL (see FIG. 3 ) of the user's fingers based on each calculated joint point JNT. Note that the joint points detected here may be only those biological parts corresponding to at least one biological part, or may be the entire hand. Skeletal parts that are the target of detection in each biometric authentication (fingerprint authentication, vein authentication, palm print authentication, etc.) will be described later.
[0029] The processor 11 calculates the pitch angle θp and roll angle θr (see FIG. 3 ) of the user's fingers (biological parts) shown in the captured image Img based on the calculated three-dimensional coordinates of each joint point JNT (St13). Based on the calculated pitch angle θp and roll angle θr, the processor 11 calculates the angular difference between the pitch angle θp and a reference angle (e.g., 0 (zero) degrees) and the angular difference between the roll angle θr and the reference angle (St14).
[0030] The processor 11 determines whether the angular differences between the calculated pitch angle θp and roll angle θr and the reference angle are both within a predetermined range (e.g., −5° to +5°) (St15). That is, the processor 11 determines whether the tilt of the biological part calculated as the pitch angle θp and the roll angle θr with respect to the light receiving surface of the image sensor (not shown) of the camera 13, which is the reference angle, is within a predetermined range. The pitch angle θp is an angle centered on the pitch axis PLp (an example of the first direction, see FIG. 3). The roll angle θr is an angle centered on the roll axis PLr (an example of the second direction, see FIG. 3). The pitch axis PLp and the roll axis PLr are each two axes that can define a reference angle with respect to the user's biological part (fingers) captured in the captured image Img, i.e., a horizontal plane parallel to the light receiving surface of the image sensor (not shown) of the camera 13.
[0031] The predetermined range here refers to the range of the angle difference between the pitch angle θp and the roll angle θr, which indicate the inclination of a biometric part that can be used as biometric data for biometric authentication. Different ranges (values) may be set for each biometric part used for biometric authentication.
[0032] If the processor 11 determines in step St15 that the angular differences between the pitch angle θp and the roll angle θr and the reference angle are both within a predetermined range (St15, YES), it calculates the distance between two adjacent joint points corresponding to each finger. For example, in the case of the middle finger, the processor 11 calculates the distance between joint point "12" and joint point "11" among joint points "9" to "12," the distance between joint point "11" and joint point "10," and the distance between joint point "10" and joint point "9." The processor 11 further determines whether all of the calculated distances between two joint points are equal to or greater than a predetermined value (predetermined distance) (St16). Note that the processor 11 may also determine whether all of the distance ratios are equal to or greater than a predetermined value in step St16.
[0033] The predetermined value here is a value for determining whether a finger is bent or not, and indicates the minimum distance between joint points or a ratio to the distance between each joint point. The predetermined value may be the same or different for each finger, such as the middle finger or index finger, and for each joint, such as the first joint or second joint.
[0034] Specifically, the predetermined value may be set for each finger and may be the minimum distance between two adjacent joint points, such as the minimum distance between joint point "12" and joint point "11" on the tip of the middle finger, the minimum distance between joint point "11" and joint point "10" on the middle finger, the minimum distance between joint point "8" and joint point "7" on the tip of the index finger, or the minimum distance between joint point "16" and joint point "15" on the index finger. The predetermined value may also be set for each finger and may be the ratio of the distance between two adjacent joint points (i.e., the lengths of the joints), such as the ratio of the length of the first joint of the middle finger (the distance between joint point "12" and joint point "11") to the length of the second joint of the middle finger (the distance between joint point "11" and joint point "10").
[0035] On the other hand, if the processor 11 determines in the processing of step St15 that the angular differences between the pitch angle θp and the roll angle θr and the reference angle are not within the predetermined range (St15, NO), it generates second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 (examples of posture guidance screens, see Figures 5 to 10) that visualize the current tilt of the body parts based on the tilt of the body parts indicated by the calculated pitch angle θp and roll angle θr, and assist the user in adjusting the tilt of the body parts. The processor 11 outputs the generated second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 to the monitor 14 for display (St17).
[0036] Furthermore, if the processor 11 determines in the processing of step St16 that all of the calculated distances between two joint points are equal to or greater than a predetermined value (St16, YES), it determines that the user's fingers are not bent (i.e., not clenched), and generates first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, and Sc82 (examples of notification screens; see FIGS. 4 and 6 to 10) that notify the user that the current inclination of the biometric part is appropriate for biometric authentication. The processor 11 outputs the generated first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, and Sc72 to the monitor 14 for display (St18). In addition, instead of the first feedback display screen, the processor 11 may generate an audio signal to notify the user that the current inclination of the body part is suitable for biometric authentication, and output the audio signal from a speaker (not shown).
[0037] The processor 11 determines whether the pitch angle θp and the roll angle θr, which indicate the inclination of the user's body parts, remain within a predetermined range for three seconds or more (St20).
[0038] On the other hand, if the processor 11 determines in the processing of step St16 that all of the calculated distances between the two joint points are not equal to or greater than a predetermined value (St16, NO), it determines that the user's fingers are bent (i.e., clenched), and generates a second feedback display screen including a control command or message (e.g., "Please spread your hand and fingers") instructing the user to spread their fingers apart, and outputs the second feedback display screen to the monitor 14 (St19).
[0039] If, in the processing of step St20, the processor 11 determines that the angular difference between each of the pitch angle θp and the roll angle θr and the reference angle remains within a predetermined range for three seconds or more (St20, YES), the processor 11 acquires the captured image Img captured by the camera 13 as a biometric image of the user to be used for biometric authentication (St21).
[0040] The duration of "3 seconds" is merely an example and is not limited to this. The processor 11 may determine whether the pitch angle θp and the roll angle θr are each within a predetermined range, that is, whether the inclination of the biometric part is maintained, while notifying the user that the current inclination of the biometric part is suitable for biometric authentication.
[0041] On the other hand, if the processor 11 determines in the processing of step St20 that the angular difference between the pitch angle θp and the roll angle θr does not remain within the predetermined range for more than 3 seconds (St20, NO), it returns to the processing of step St11.
[0042] As described above, the terminal device P1 according to the first embodiment can determine, through the processing of step St15, whether the tilt (angle difference) of the user's biometric part in the captured image allows a biometric image suitable for biometric authentication to be acquired. If the terminal device P1 determines that the tilt (angle difference) of the biometric part is equal to or greater than a predetermined range and that a biometric image suitable for biometric authentication cannot be acquired, the terminal device P1 generates a second feedback display screen that associates information about the calculated tilt of the biometric part with the captured image Img in which the biometric part is captured. This allows the terminal device P1 to notify the user of the tilt of the biometric part and to provide support for acquiring a biometric image suitable for biometric authentication. Therefore, the terminal device P1 can reduce variation in how the biometric part appears in the captured image and more effectively support the acquisition of a biometric image more suitable for biometric authentication.
[0043] Furthermore, the terminal device P1 according to the first embodiment can determine whether each finger is bent by clenching through the process of step St16. This allows the terminal device P1 to correct the bent fingers and more effectively support the acquisition of biometric images that are more suitable for biometric authentication using fingerprints or veins.
[0044] Next, a method for calculating the pitch angle θp and the roll angle θr will be described with reference to Fig. 3. Fig. 3 is a diagram illustrating an example of an operation procedure of the terminal device P1 according to embodiment 1. Note that in Fig. 3, reference numerals are not assigned to all of the joint points in order to make the diagram easier to understand.
[0045] The processor 11 calculates the three-dimensional coordinates of multiple joint points JNT (each of the 21 joint points "0" to "20") of the user's hand H from the captured image Img in which the entire user's hand H is captured.
[0046] The processor 11 calculates the pitch angle θp and the roll angle θr based on the calculated three-dimensional coordinates of each joint point JNT. Note that the pitch axis PLp shown in FIG. 3 has a pitch angle θp = 0 (zero) degrees, which indicates the reference angle of the pitch angle. The roll axis PLr has a roll angle θr = 0 (zero) degrees, which indicates the reference angle of the roll angle.
[0047] For example, when capturing an image of a fingertip suitable for fingerprint authentication as an example of a biological part, the processor 11 calculates the pitch angle θp of the user's hand H based on the three-dimensional coordinates of each of the joint points "9" at the base of the middle finger to "12" at the tip of the middle finger. Also, when capturing an image suitable for fingerprint authentication, for example, the processor 11 calculates the roll angle θr of the user's hand H based on the three-dimensional coordinates of each of the joint points "8" at the tip of the index finger and "16" at the tip of the ring finger.
[0048] Here, other examples of calculating the pitch angle θp and the roll angle θr will be described. Note that the methods of calculating the pitch angle θp and the roll angle θr corresponding to each body part described below are merely examples, and are not limited to these.
[0049] When capturing an image of a finger suitable for fingerprint authentication as an example of a biological part, for example, the processor 11 calculates the pitch angle θp of the finger from which the fingerprint is to be captured based on the three-dimensional coordinates of the joint point at the fingertip of the finger from which the fingerprint is to be captured and the three-dimensional coordinates of any one joint point other than the fingertip of the finger. Specifically, if the finger from which the fingerprint is to be captured is the middle finger, for example, the processor 11 calculates the pitch angle θp of the finger from which the fingerprint is to be captured based on the three-dimensional coordinates of the joint point "12" at the fingertip of the middle finger and the three-dimensional coordinates of any one joint point among the joint points "9" to "11" other than the tip of the middle finger.
[0050] Furthermore, for example, the processor 11 calculates the roll angle θr of the finger from which the fingerprint is to be obtained based on the three-dimensional coordinates of the joint point "8" at the tip of the index finger and the three-dimensional coordinates of the joint point "16" at the tip of the ring finger or the joint point "20" at the tip of the little finger.
[0051] Furthermore, when capturing an image of a finger suitable for vein authentication as an example of a biological part, for example, the processor 11 calculates the pitch angle θp of the finger from which veins are to be obtained based on the three-dimensional coordinates of a joint point near the tip of the finger from which veins are to be obtained and the three-dimensional coordinates of a joint point near the base of the finger. Specifically, if the finger from which veins are to be obtained is the middle finger, for example, the processor 11 calculates the pitch angle θp of the middle finger from which veins are to be obtained based on the three-dimensional coordinates of joint point "11" or joint point "12" near the tip of the middle finger and the three-dimensional coordinates of joint point "9" or joint point "10" near the base of the middle finger.
[0052] Furthermore, for example, the processor 11 calculates the roll angle θr of the finger from which veins are to be acquired based on the three-dimensional coordinates of the second joint point "6" at the tip of the index finger and the three-dimensional coordinates of the joint point "14" at the second joint of the ring finger.
[0053] Furthermore, when capturing an image of a palm suitable for palmprint authentication as an example of a biological part, for example, the processor 11 calculates the pitch angle θp of the finger from which the palmprint is to be acquired based on the three-dimensional coordinates of the wrist joint point "0" and the three-dimensional coordinates of any one of the middle finger joint points "9" to "12." The processor 11 may calculate the pitch angle θp based on the three-dimensional coordinates of the wrist joint point "0" and the three-dimensional coordinates of any one of the ring finger joint points "13" to "16," or may calculate the pitch angle θp based on the three-dimensional coordinates of any two of the index finger joint points "5" to "8."
[0054] The processor 11 also calculates the roll angle θr based on the three-dimensional coordinates of the joint point "17" at the base of the little finger and the three-dimensional coordinates of the joint point "5" at the base of the index finger. The processor 11 may also calculate the roll angle θr based on the three-dimensional coordinates of the joint point "17" at the base of the little finger and the three-dimensional coordinates of any one of the joint points "1" to "4" of the thumb.
[0055] As described above, the terminal device P1 can calculate the pitch angle θp and roll angle θr, which indicate the inclination of the user's biometric parts, based on the three-dimensional coordinates of the joint points corresponding to the biometric parts used in biometric authentication. This allows the terminal device P1 to more effectively suppress variations in how the biometric parts are captured and acquire biometric images that are more suitable for biometric authentication. Therefore, the terminal device P1 can achieve more accurate biometric authentication.
[0056] <First Display Example> Next, the first display example will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram illustrating an example of a first feedback display screen Sc11 in the first display example. Fig. 5 is a diagram illustrating examples of second feedback display screens Sc12 to Sc15.
[0057] 4 and 5 show examples of visualizing the current inclination of the user's fingers by depicting a circular level CL. Each of the first feedback display screen Sc11 and the second feedback display screens Sc12 to Sc15 visualizes the current inclination of the user's fingers by generating a circular level CL in which a bubble AB, which is smaller than the level HL and indicates the current inclination of the user's fingers, is superimposed on the level HL, which indicates the reference angle.
[0058] 4 and 5 will be described as an example in which the predetermined range is -5° to +5°. Also, in each of the second feedback display screens Sc12 to Sc15 shown in Fig. 5, the symbols for the pitch reference line, roll reference line, pitch axis, and roll axis are omitted in order to make it easier to see the positional relationship between the level HL and the air bubble AB. In this disclosure, the pitch angle θp and roll angle θr are defined such that the direction of the arrow tip shown in Fig. 3 is represented by a + (positive sign) and the direction opposite to the arrow tip is represented by a - (negative sign).
[0059] The processor 11 generates a first feedback display screen Sc11 including an image Img11 on which the detected joint point JNT and a circular spirit level CL based on the pitch angle θp and the roll angle θr are superimposed, and a message Msg1 "Please stop your hand" requesting the user to maintain the current inclination of their fingers.
[0060] Specifically, the processor 11 generates a spirit level HL whose center is the intersection of a pitch reference line LN11A corresponding to the pitch axis PLp and a roll reference line LN12A corresponding to the roll axis PLr. The processor 11 also generates an air bubble AB that indicates the tilt of the user's fingers and whose center is the intersection of a pitch axis LN11B corresponding to the pitch axis PLp and a roll axis LN12B corresponding to the roll axis PLr.
[0061] The processor 11 calculates the superimposition position of the bubble AB to be superimposed on the level HL based on each of the pitch angle θp and the roll angle θr relative to the reference angle. The processor 11 superimposes the bubble AB on the level HL based on the calculated superimposition position, thereby generating a circular level CL. The processor 11 superimposes the detected joint points JNT (each joint point) and the circular level CL based on each of the pitch angle θp and the roll angle θr on the captured image Img11.
[0062] 4 shows an example of a message requesting the user to maintain the current inclination of the fingers, but it may also be a message informing the user that the current inclination of the fingers is suitable for biometric authentication. Moreover, the message Msg1 may not be included on the first feedback display screen Sc11, but may be output as voice.
[0063] As a result, the terminal device P1 in the first display example can notify the user that the inclination of the user's fingers is appropriate for biometric authentication and that no adjustment is necessary. Furthermore, the user can intuitively understand that the current inclination of the fingers should be maintained based on the message Msg1 included in the first feedback display screen Sc11 or the fact that the pitch reference line LN11A and roll reference line LN12A of the spirit level HL substantially coincide with the pitch axis LN11B and roll axis LN12B of the bubble AB.
[0064] The processor 11 superimposes the detected joint points JNT (each joint point) and a circular spirit level CL based on each of the pitch angle θp and roll angle θr onto the captured images Img12, Img13, Img14, and Img15, and generates each of the second feedback display screens Sc12 to Sc15.
[0065] The second feedback display screen Sc12 shown in FIG. 5 shows the respective positions of the level HL and the air bubble AB when the calculated pitch angle θp is greater than +5° and the roll angle θr is less than −5°. The second feedback display screen Sc13 shows the respective positions of the level HL and the air bubble AB when the calculated pitch angle θp and roll angle θr are both greater than +5°. The second feedback display screen Sc14 shows the respective positions of the level HL and the air bubble AB when the calculated pitch angle θp and roll angle θr are both less than −5°. The second feedback display screen Sc15 shows the respective positions of the level HL and the air bubble AB when the calculated pitch angle θp is less than −5° and the roll angle θr is greater than +5°.
[0066] Also, although not shown in Figure 5, if it is determined that the tilt of the user's fingers is not within a specified range, the processor 11 may generate a second feedback display screen Sc12 to Sc15 that includes a message requesting the user to adjust the tilt of their fingers, such as "Tilt your hand so that the bubble is in the center of the circle."
[0067] As a result, the terminal device P1 in the first display example can assist the user in adjusting the tilt of the user's fingers by generating and outputting second feedback display screens Sc12 to Sc15 in which the circular level CL is superimposed on the user's fingers. Based on the second feedback display screens Sc12 to Sc15, the user can intuitively adjust the tilt of the user's fingers so that the position of the center point of the bubble AB relative to the center point of the level HL and the position of the pitch axis or roll axis of the bubble AB relative to the pitch reference line or roll reference line of the level HL are aligned, respectively.
[0068] In addition, in the first feedback display screen Sc11 and the second feedback display screens Sc12 to Sc15, the superimposition process of the skeleton SL and the joint points JNT (each joint point) is not essential and may be omitted.
[0069] <Second Display Example> Next, a second display example will be described with reference to Fig. 6. Fig. 6 is a diagram illustrating an example of a first feedback display screen Sc22 and an example of a second feedback display screen Sc21 in the second display example.
[0070] Also, although not shown in Figure 6, the processor 11 may generate a second feedback display screen Sc21 containing a message requesting the user to adjust the inclination of their fingers, or may generate a first feedback display screen Sc22 containing a message requesting the user to maintain the current inclination of their fingers, based on the determination result of whether the inclination of the user's fingers is within a predetermined range.
[0071] In the second display example, spheres HS21 and HS22 having their centers at the intersection of the pitch axis PLp and the roll axis PLr are superimposed on captured images Img21 and Img22 of the user's fingers. The second feedback display screen Sc21 in the second display example rotates each of the spheres HS21 based on the angular differences between the reference angle and each of the pitch angle θp and roll angle θr, and superimposes the rotated spheres HS21 on the captured image Img21, thereby visualizing the tilt of the user's fingers.
[0072] The processor 11 detects the joint points JNT of the user's fingers from the captured images Img21 and Img22, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.
[0073] The processor 11 generates spheres HS21 and HS22 whose centers are the intersection points of the pitch axis PLp and the roll axis PLr, and generates curves LN21A and LN22A, which are curves obtained by projecting the pitch axis PLp onto the surfaces of the spheres HS21 and HS22 and which pass through two intersection points between the pitch axis PLp and the spheres, and curves LN21B and LN22B, which are curves along the surfaces of the spheres and which pass through two intersection points between the roll axis PLr and the spheres. The processor 11 calculates the angular differences between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angular differences between the calculated pitch angle θp and roll angle θr are within a predetermined range.
[0074] If the processor 11 determines that the calculated angular difference between the pitch angle θp and the roll angle θr is not within a predetermined range, the processor 11 rotates the sphere HS21 based on the calculated angular difference. The processor 11 superimposes the rotated sphere HS21, the curves LN21A and LN21B corresponding to the rotated sphere HS21, the skeleton SL, and the joint points JNT on the captured image Img21 to generate a second feedback display screen Sc21. The processor 11 outputs the generated second feedback display screen Sc21 to the monitor 14 for display.
[0075] As a result, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by using the second feedback display screen Sc21, which visualizes the angular differences of the pitch angle θp and the roll angle θr relative to the reference angle, i.e., the adjustment direction of the tilt of the user's fingers, using the two curves LN21A and LN21B on the second feedback display screen Sc21. Furthermore, the user can intuitively grasp the tilt of the user's fingers and the direction in which they should tilt their fingers using the two curves LN21A and LN21B on the second feedback display screen Sc21.
[0076] On the other hand, if the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 omits the rotation of the sphere HS22. The processor 11 superimposes the sphere HS22, the curve LN22A along the pitch axis PLp, the curve LN22B along the roll axis PLr, the skeleton SL, and the joint points JNT on the captured image Img22 to generate a first feedback display screen Sc22. The processor 11 outputs the generated first feedback display screen Sc22 to the monitor 14 for display.
[0077] The first feedback display screen Sc22 shows the sphere HS22 in a so-called front view. Therefore, the curves LN22A and LN22B displayed on the first feedback display screen Sc22 are displayed as straight lines that intersect at right angles with the cross at the center point of the sphere HS22.
[0078] As a result, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by displaying the two curves LN22A and LN22B as being approximately perpendicular to each other. Furthermore, the user can intuitively understand from the two curves LN22A and LN22B on the first feedback display screen Sc22 that the current inclination of the user's fingers should be maintained. Therefore, the terminal device P1 can assist in acquiring a biometric image that is more suitable for biometric authentication.
[0079] The first feedback display screen Sc22 in the second display example may include a message notifying the user that the current inclination of the user's fingers is suitable for biometric authentication and that no adjustment is necessary. The processor 11 may also change the line type, line thickness, or line color of the sphere HS22 displayed on the first feedback display screen Sc22 and the line type, line thickness, or line color of the sphere HS21 displayed on the second feedback display screen Sc21.
[0080] <Third Display Example> Next, a third display example will be described with reference to Fig. 7. Fig. 7 is a diagram illustrating an example of a first feedback display screen Sc32 and an example of a second feedback display screen Sc31 in the third display example.
[0081] Also, although not shown in Figure 7, the processor 11 may generate a second feedback display screen Sc31 containing a message requesting the user to adjust the inclination of their fingers, or may generate a first feedback display screen Sc32 containing a message requesting the user to maintain the current inclination of their fingers, based on the determination result of whether the inclination of the user's fingers is within a specified range.
[0082] In the third display example, spheres HS31 and HS32 having their centers at the intersection of the pitch axis PLp and the roll axis PLr are superimposed on captured images Img31 and Img32 of the user's fingers. The second feedback display screen Sc31 in the third display example rotates each of the spheres HS31 based on the angular differences between the reference angle and each of the pitch angle θp and the roll angle θr, and superimposes vectors VT31A and VT31B corresponding to the angular differences between the rotated sphere HS31 and each of the pitch angle θp and the roll angle θr on the captured image Img31, thereby visualizing the adjustment direction of the tilt of the user's fingers.
[0083] The processor 11 detects the joint points JNT of the user's fingers from the captured images Img31 and Img32, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.
[0084] The processor 11 generates spheres HS31 and HS32 whose centers are the intersection points of the pitch axis PLp and the roll axis PLr, and pitch axes LN31A and LN32A and roll axes LN31B and LN32B that are superimposed on the spheres HS31 and HS32. The processor 11 calculates the angular differences between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angular differences between the calculated pitch angle θp and roll angle θr are within predetermined ranges.
[0085] When the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, the processor 11 generates a vector VT31A based on the calculated angular difference of the pitch angle θp. Specifically, the processor 11 determines the direction of the vector VT31A in which the user should adjust the inclination of the fingers (i.e., the direction in which the fingers should be inclined) based on the sign (positive or negative) of the angular difference of the calculated pitch angle θp, and determines the length of the vector VT31A based on the magnitude of the calculated angular difference.
[0086] Furthermore, the processor 11 generates a vector VT31B based on the calculated angular difference of the roll angle θr. Specifically, the processor 11 determines the direction of the vector VT31B in which the user should adjust the inclination of the fingers (i.e., the direction in which the fingers should be inclined) based on the sign of the calculated angular difference of the roll angle θr, and determines the length of the vector VT31B based on the magnitude of the calculated angular difference.
[0087] The processor 11 superimposes the sphere HS31, the pitch axis LN31A and the roll axis LN31B, the vectors VT31A and VT31B, the skeleton SL, and the joint points JNT on the captured image Img31 to generate a second feedback display screen Sc21. The processor 11 outputs the generated second feedback display screen Sc21 to the monitor 14 for display.
[0088] As a result, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by using the second feedback display screen Sc31, which visualizes the angular differences of the pitch angle θp and the roll angle θr relative to the reference angle, i.e., the inclination of the user's fingers, in the direction along the pitch axis PLp (i.e., the horizontal direction) and the direction along the roll axis PLr (the vertical direction), using each of the two vectors VT31A and VT31B. Furthermore, the user can intuitively grasp the inclination of the user's fingers from the directions and lengths of the vectors VT31A and VT31B displayed on the second feedback display screen Sc31, and can intuitively grasp in which direction the fingers should be inclined.
[0089] On the other hand, if the processor 11 determines that the angular differences between the calculated pitch angle θp and roll angle θr are within a predetermined range, the processor 11 omits generating vectors. The processor 11 superimposes the sphere HS32, the pitch axis LN32A and the roll axis LN32B, the skeleton SL, and the joint points JNT on the captured image Img32 to generate a first feedback display screen Sc32. The processor 11 outputs the generated first feedback display screen Sc32 to the monitor 14 for display.
[0090] As a result, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by omitting the generation and display of vectors. Furthermore, the user can intuitively understand that the current inclination of the user's fingers should be maintained based on the pitch axis LN32A and the roll axis LN32B superimposed on the sphere HS32 on the first feedback display screen Sc32. Therefore, the terminal device P1 can assist in acquiring a biometric image that is more suitable for biometric authentication.
[0091] The first feedback display screen Sc32 in the third display example may include a message notifying the user that the current inclination of the user's fingers is suitable for biometric authentication and that no adjustment is necessary. The processor 11 may also change the line type, line thickness, or line color of the sphere HS32 displayed on the first feedback display screen Sc32 and the line type, line thickness, or line color of the sphere HS31 displayed on the second feedback display screen Sc31.
[0092] <Fourth Display Example> Next, a fourth display example will be described with reference to Fig. 8. Fig. 8 is a diagram illustrating an example of a first feedback display screen Sc43 and examples of second feedback display screens Sc41 and Sc42 in the fourth display example.
[0093] Also, although not shown in Figure 6, the processor 11 may generate second feedback display screens Sc41 and Sc42 containing a message requesting the user to adjust the inclination of their fingers, or may generate a first feedback display screen Sc43 containing a message requesting the user to maintain the current inclination of their fingers, based on the determination result of whether the inclination of the user's fingers is within a predetermined range.
[0094] In the fourth display example, spheres HS41, HS42, and HS43 having their centers at the intersection of the pitch axis PLp and the roll axis PLr are superimposed on captured images Img41, Img42, and Img43 of the user's fingers. The second feedback display screens Sc41 and Sc42 in the fourth display example visualize the adjustment direction of the tilt of the user's fingers by superimposing, on the captured images Img41 and Img42, vectors corresponding to the angular differences between the reference angle and the pitch angle θp and between the reference angle and the roll angle θr, respectively.
[0095] The processor 11 detects the joint points JNT of the user's fingers from the captured images Img41 to Img43, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.
[0096] Processor 11 generates a sphere whose center is the intersection of the pitch axis PLp and the roll axis PLr, and generates pitch axes LN41A, LN42A, and LN43A which are curves obtained by projecting the pitch axis PLp onto the surfaces of spheres HS41 to HS43, and roll axes LN41B, LN42B, and LN43B which are curves obtained by projecting the roll axis PLr onto the surfaces of spheres HS41 to HS43. Processor 11 calculates the angular differences between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angular differences between the calculated pitch angle θp and roll angle θr are within a predetermined range.
[0097] The processor 11 further calculates the angular difference between the calculated pitch angle θp and the pitch axis PLp (i.e., the reference angle), and generates a vector indicating the angular difference of the pitch angle θp based on the sign (positive or negative) and magnitude of the calculated angular difference.
[0098] In the example shown on the second feedback display screen Sc41, the processor 11 determines the direction of a vector VT41 along which the user should tilt their fingers based on the calculated angular difference between the pitch angle θp and the roll angle θr and the signs of the pitch angle θp and the roll angle θr, and determines the length of the vector VT41 based on the magnitude of the calculated angular difference. Here, the vector VT41 is equal to a vector generated by combining a vector based on the angular difference and sign of the pitch angle θp and a vector based on the angular difference and sign of the roll angle θr.
[0099] The processor 11 superimposes the skeleton SL, the joint points JNT, the sphere HS41, the pitch axis LN41A and the roll axis LN41B, and the vector VT41 on the captured image Img41 to generate a second feedback display screen Sc41, which is output and displayed on the monitor 14. Note that here, the processor 11 superimposes the starting position of the vector VT41 on the palm side of the user (i.e., the front side of the page), at the zenith position of the sphere HS41.
[0100] As a result, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by using the second feedback display screen Sc41 that visualizes the adjustment direction of the tilt of the user's fingers using the vector VT41. Furthermore, the user can intuitively grasp the tilt of the user's fingers from the direction and length of the vector VT41 displayed on the second feedback display screen Sc41, and can also intuitively grasp in which direction the user should tilt their fingers.
[0101] In addition, in the example shown on the second feedback display screen Sc42, the processor 11 determines the direction of the vector VT42A along the direction in which the user should tilt their fingers (i.e., the opposite direction to the direction indicated by the sign of the pitch angle θp) based on the angle difference and sign of the calculated pitch angle θp, and determines the length of the vector VT42A based on the magnitude of the angle difference of the calculated pitch angle θp.
[0102] Based on the angle difference and sign of the calculated roll angle θr, processor 11 determines the direction of vector VT42B along the direction in which the user should tilt their fingers (i.e., the opposite direction to the direction indicated by the sign of the roll angle θr), and determines the length of vector VT42B based on the magnitude of the angle difference of the calculated roll angle θr.
[0103] The processor 11 superimposes the skeleton SL, the joint points JNT, the sphere HS42, the pitch axis LN42A and the roll axis LN42B, and the vectors VT42A and VT42B on the captured image Img42 to generate a second feedback display screen Sc42, which is output to and displayed on the monitor 14. Note that here, the processor 11 superimposes the starting positions of the vectors VT42A and VT42B on the palm side of the user (the front side of the page) and at the zenith position of the sphere HS42.
[0104] As a result, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by using the second feedback display screen Sc42, which visualizes the adjustment direction of the tilt of the user's fingers using each of the vectors VT42A and VT42B. Furthermore, the user can intuitively grasp the tilt of the user's fingers based on the direction and length of each of the vectors VT42A and VT42B displayed on the second feedback display screen Sc42, and can also intuitively grasp in which direction the user should tilt their fingers.
[0105] On the other hand, if the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 omits generating the vectors. The processor 11 superimposes the skeleton SL, the joint points JNT, the sphere HS43, and each of the pitch axis LN43A and the roll axis LN43B on the captured image Img43 to generate a first feedback display screen Sc43, and outputs and displays it on the monitor 14.
[0106] As a result, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by omitting the generation and display of vectors. Furthermore, the user can intuitively understand that the current inclination of the user's fingers should be maintained based on the pitch axis LN43A and the roll axis LN43B superimposed on the sphere HS43 on the first feedback display screen Sc43. Therefore, the terminal device P1 can assist in acquiring a biometric image that is more suitable for biometric authentication.
[0107] The first feedback display screen Sc43 in the fourth display example may include a message notifying the user that the current inclination of the user's fingers is suitable for biometric authentication and that no adjustment is necessary. The processor 11 may also change the line type, line thickness, or line color of the sphere HS43 displayed on the first feedback display screen Sc43 and the line type, line thickness, or line color of the spheres HS41 and HS42 displayed on the second feedback display screens Sc41 and Sc42.
[0108] Next, a fifth display example and a sixth display example will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating examples of first feedback display screens Sc52 and Sc62 and examples of second feedback display screens Sc51 and Sc61 in the fifth and sixth display examples.
[0109] Also, although not shown in Figure 9, if the tilt of the user's fingers is not within a specified range, the processor 11 may generate a second feedback display screen Sc51, Sc61 including a message requesting the user to adjust the tilt of their fingers.
[0110] <Fifth Display Example> In the fifth display example, squares LN51 and LN52 are superimposed on the user's biological parts that appear in captured images Img51 and Img52 in which the user's fingers are captured. The second feedback display screen Sc51 in the fifth display example visualizes the adjustment direction of the tilt of the user's fingers by deforming the square LN51 based on the angular differences between the reference angle and each of the pitch angle θp and roll angle θr and superimposing it on the captured image Img51.
[0111] The processor 11 detects the joint points JNT of the user's fingers from the captured images Img51 and Img52, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.
[0112] The processor 11 calculates the angular difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.
[0113] If the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, it deforms the shape of the square based on the angular difference between the calculated pitch angle θp and roll angle θr.
[0114] Specifically, the processor 11 detects an area including at least a portion of the user's biological part (the palm in the example shown in FIG. 9 ) in the captured image Img51. The processor 11 generates a square in the detected area to be superimposed on a horizontal plane defined by the pitch axis PLp and the roll axis PLr. The processor 11 forms a horizontal plane defined by the calculated pitch angle θp and roll angle θr. The processor 11 calculates a transformation formula (transformation matrix) for superimposing the square (not shown) superimposed on the horizontal plane defined by the pitch axis PLp and the roll axis PLr on the horizontal plane defined by the pitch angle θp and the roll angle θr. Based on the calculated transformation formula (transformation matrix), the processor 11 transforms the square (not shown) superimposed on the horizontal plane defined by the pitch axis PLp and the roll axis PLr into a square LN51 that can be superimposed on the horizontal plane defined by the pitch angle θp and the roll angle θr.
[0115] The processor 11 generates a second feedback display screen Sc51 by superimposing, on the captured image Img51, a square LN51 that is a region including at least a part of the user's biological part (the palm in the example shown in FIG. 9 ) and that has been transformed onto a horizontal plane defined by the pitch angle θp and the roll angle θr. The processor 11 outputs the generated second feedback display screen Sc51 to the monitor 14 to display it.
[0116] This allows the terminal device P1 to generate a second feedback display screen Sc51 that visualizes the tilt of the user's fingers in the shape of a square LN51 superimposed on the user's biometric parts shown in the captured image Img51. By generating the second feedback display screen Sc51, the terminal device P1 can assist in the acquisition of a biometric image suitable for biometric authentication. Furthermore, the shape of the square LN51 displayed on the second feedback display screen Sc51 allows the user to intuitively grasp the tilt of the user's fingers and intuitively grasp in which direction the user should tilt the fingers to return the deformed square LN51 to the original square.
[0117] On the other hand, if the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, it omits the shape deformation process of the square LN52.
[0118] The processor 11 superimposes, on the captured image Img52, a square LN52 that has been deformed onto a horizontal plane defined by the pitch axis PLp and the roll axis PLr, in an area that includes at least a portion of the user's biometric part (the palm in the example shown in FIG. 9 ) that appears in the captured image Img52. The processor 11 generates a first feedback display screen Sc52 that includes the captured image Img52 on which the square LN52 has been superimposed and a message Msg2 "OK!" that notifies the user that the current inclination of the user's fingers is appropriate for biometric authentication. The processor 11 outputs the generated first feedback display screen Sc52 to the monitor 14 for display.
[0119] As a result, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by the shape of the square LN52 superimposed on the user's biometric part in the captured image Img52 or the message Msg2. Furthermore, the shape of the square LN52 or the message Msg2 allows the user to intuitively understand that the current inclination of the user's fingers should be maintained. Therefore, the terminal device P1 can assist in acquiring a biometric image that is more suitable for biometric authentication.
[0120] In addition, in the fifth display example, the processor 11 may change the line type, line thickness or line color of the square LN52 displayed on the first feedback display screen Sc52 and the line type, line thickness or line color of the transformed square LN51 displayed on the second feedback display screen Sc51.
[0121] <Sixth display example> In the sixth display example, a sphere BL generated by Augmented Reality (AR) is superimposed on the user's biological parts shown in the captured images Img61 and Img62 in which the user's fingers are captured, thereby visualizing the tilt of the user's fingers.
[0122] The processor 11 detects the joint points JNT of the user's fingers from the captured images Img61 and Img62, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.
[0123] The processor 11 calculates the angular difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.
[0124] If the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, the processor 11 tilts the pitch axis PLp and the roll axis PLr based on the calculated angular difference between the pitch angle θp and the roll angle θr. Furthermore, the processor 11 calculates the position (coordinates) of the ball BL when it is rolled from the intersection position of the pitch axis PLp and the roll axis PLr based on the calculated angular difference between the pitch angle θp and the roll angle θr. The processor 11 generates a second feedback display screen Sc61 by superimposing the pitch axis LN61A and the roll axis LN61B after the tilting process and the ball BL virtually rolled on the user's palm on the captured image Img61. The processor 11 outputs the generated second feedback display screen Sc61 to the monitor 14 for display.
[0125] Here, the processor 11 may change the size of the sphere BL superimposed on the captured image Img61 based on the distance between the intersection of the pitch axis PLp and the roll axis PLr and the position of the sphere BL, and the direction in which the sphere BL is located relative to the intersection of the pitch axis PLp and the roll axis PLr. This allows the processor 11 to express a sense of perspective using the size of the sphere BL.
[0126] As a result, the terminal device P1 can generate a second feedback display screen Sc61 that visualizes the tilt of the user's fingers based on the rolling direction, position, or size of the ball BL. By generating the second feedback display screen Sc61, the terminal device P1 can assist in acquiring a biometric image suitable for biometric authentication. Furthermore, the user can intuitively grasp the tilt of the user's fingers displayed on the second feedback display screen Sc61 and can intuitively grasp in which direction to tilt the fingers so that the position of the ball BL is at the intersection of the pitch axis PLp and the roll axis PLr.
[0127] On the other hand, if the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 superimposes a sphere BL at the intersection position (reference position) of the pitch axis PLp and the roll axis PLr in the captured image Img62. The processor 11 generates a first feedback display screen Sc62 that includes the captured image Img62 with the sphere BL superimposed thereon and a message Msg3 "OK!" notifying the user that the current inclination of the user's fingers is appropriate for biometric authentication. The processor 11 outputs the generated first feedback display screen Sc62 to the monitor 14 for display.
[0128] As a result, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by the position of the sphere BL in the captured image Img62 or by the message Msg3. Furthermore, the user can intuitively understand from the position of the sphere BL or the message Msg3 that the current inclination of the user's fingers should be maintained. Therefore, the terminal device P1 can assist in acquiring a biometric image that is more suitable for biometric authentication.
[0129] In addition, in the sixth display example, the processor 11 may change the color of the ball BL displayed on the first feedback display screen Sc62 and the color of the ball BL displayed on the second feedback display screen Sc61.
[0130] Next, the seventh and eighth display examples will be described with reference to Fig. 10. Fig. 10 is a diagram illustrating examples of first feedback display screens Sc72 and Sc82 and examples of second feedback display screens Sc71 and Sc81 in the seventh and eighth display examples.
[0131] Also, although not shown in Figure 10, if the tilt of the user's fingers is not within a specified range, the processor 11 may generate a second feedback display screen Sc71, Sc81 including a message requesting the user to adjust the tilt of their fingers.
[0132] <Seventh Display Example> In the seventh display example, a ring RG is superimposed on captured images Img71 and Img72 of the user's fingers. The second feedback display screen Sc71 in the seventh display example visualizes the adjustment direction of the tilt of the user's fingers by superimposing, on the ring RG, a ring bar RG71A based on the angle differences between the reference angle and each of the pitch angle θp and the roll angle θr.
[0133] The processor 11 detects the joint points JNT of the user's fingers from the captured images Img71 and Img72, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.
[0134] The processor 11 calculates the angular difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.
[0135] If the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, it generates a ring RG to be superimposed on the captured image Img71, and generates a ring bar RG71A based on the angular difference between the pitch angle θp and roll angle θr and superimposes it on the ring RG.
[0136] Specifically, the processor 11 superimposes the ring bar RG71A on the ring RG in a direction (angle) based on the angular difference between the pitch angle θp and the roll angle θr. The processor 11 also determines the circumferential length of the superimposed ring bar RG71A in the circumferential direction based on the magnitude of the angular difference between the pitch angle θp and the roll angle θr.
[0137] The processor 11 generates a second feedback display screen Sc71 by superimposing the ring RG with the ring bar RG71A superimposed thereon on the captured image Img71. The processor 11 outputs the generated second feedback display screen Sc71 to the monitor 14 to display it.
[0138] As a result, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by generating the second feedback display screen Sc71 that visualizes the tilt of the user's fingers with the ring bar RG71A. Furthermore, the user can intuitively grasp the tilt of the user's fingers from the position and circumference of the ring bar RG71A displayed on the second feedback display screen Sc71, and can also intuitively grasp in which direction to tilt the fingers to erase the ring bar RG71A.
[0139] On the other hand, if the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 generates a ring RG to be superimposed on the captured image Img72 and superimposes it on the captured image Img72. The processor 11 generates a first feedback display screen Sc72 that includes the captured image Img72 with the ring RG superimposed thereon and a message Msg4 "OK!" notifying the user that the current inclination of the user's fingers is appropriate for biometric authentication. The processor 11 outputs the generated first feedback display screen Sc72 to the monitor 14 to display it.
[0140] This allows the terminal device P1 to notify the user that the current inclination of the user's fingers is suitable for biometric authentication by using the ring RG or message Msg4 displayed in the captured image Img72. Furthermore, the ring RG or message Msg4 allows the user to intuitively understand that the current inclination of the user's fingers superimposed on the first feedback display screen Sc72 should be maintained. Therefore, the terminal device P1 can assist in acquiring a biometric image that is more suitable for biometric authentication.
[0141] In addition, in the seventh display example, the processor 11 may change the line type, line thickness or line color of the ring RG displayed on the first feedback display screen Sc72 and the line type, line thickness or line color of the ring RG displayed on the second feedback display screen Sc71.
[0142] <Eighth Display Example> In the eighth display example, pitch sliders SL81A and SL82A and roll sliders SL81B and SL82B are displayed along two adjacent sides of captured images Img81 and Img82 of the user's fingers. The second feedback display screen Sc81 in the eighth display example visualizes the adjustment direction of the tilt of the user's fingers by superimposing pitch pointers PT81A and PT82A based on the angular difference between a reference angle and the pitch angle θp and roll pointers PT81B and PT82B based on the angular difference between the reference angle and the roll angle θr on the pitch slider SL81A and the roll slider SL81B, respectively.
[0143] The processor 11 detects the joint points JNT of the user's fingers from the captured images Img81 and Img82, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.
[0144] The processor 11 calculates the angular difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.
[0145] If the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, the processor 11 generates a pitch slider SL81A and a roll slider SL81B based on the calculated angular difference.
[0146] Specifically, the processor 11 generates a pitch slider SL81A including a pitch area RG81A indicating a tilt within a predetermined range suitable for biometric authentication and a pitch pointer PT81A indicating the current pitch angle θp based on the angular difference of the calculated pitch angle θp. The processor 11 generates a roll slider SL81B including a roll area RG81B indicating a tilt within a predetermined range suitable for biometric authentication and a roll pointer PT81B indicating the current roll angle θr based on the angular difference of the calculated roll angle θr.
[0147] The processor 11 generates a second feedback display screen Sc81 including the generated pitch slider SL81A, roll slider SL81B, and captured image Img81. The processor 11 outputs the generated second feedback display screen Sc81 to the monitor 14 for display.
[0148] As a result, the terminal device P1 can generate a second feedback display screen Sc81 that visualizes the tilt of the user's fingers based on the position of the pitch pointer PT81A relative to the pitch area RG81A and the position of the roll pointer PT81B relative to the roll area RG81B. By generating the second feedback display screen Sc81, the terminal device P1 can assist in the acquisition of a biometric image suitable for biometric authentication. Furthermore, the user can intuitively grasp the tilt of the user's fingers based on the position of the pitch pointer PT81A relative to the pitch area RG81A and the position of the roll pointer PT81B relative to the roll area RG81B displayed on the second feedback display screen Sc81, and can intuitively grasp in which direction the user should tilt their fingers so that the pitch pointer PT81A is included in the pitch area RG81A and the roll pointer PT81B is included in the roll area RG81B.
[0149] If the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, it generates a pitch slider SL82A and a roll slider SL82B based on the calculated angular difference.
[0150] Specifically, the processor 11 generates a pitch slider SL82A including a pitch area RG82A indicating a tilt within a predetermined range suitable for biometric authentication and a pitch pointer PT82A indicating the current pitch angle θp based on the angular difference of the calculated pitch angle θp. The processor 11 generates a roll slider SL82B including a roll area RG82B indicating a tilt within a predetermined range suitable for biometric authentication and a roll pointer PT82B indicating the current roll angle θr based on the angular difference of the calculated roll angle θr.
[0151] The processor 11 generates a message Msg5 "OK!" notifying the user that the current inclination of the user's fingers is appropriate for biometric authentication. The processor 11 generates a first feedback display screen Sc82 including the generated message Msg5, a pitch slider SL82A, a roll slider SL82B, and a captured image Img82. The processor 11 outputs the generated first feedback display screen Sc82 to the monitor 14 for display.
[0152] As a result, the terminal device P1 can generate a first feedback display screen Sc82 that visualizes that the inclination of the user's fingers is suitable for biometric authentication based on the position of the pitch pointer PT82A relative to the pitch area RG82A and the position of the roll pointer PT82B relative to the roll area RG82B, or based on the message Msg5. By generating the first feedback display screen Sc82, the terminal device P1 can assist in the acquisition of a biometric image suitable for biometric authentication. Furthermore, the pitch pointer PT82A, the roll pointer PT82B, or the message Msg5 allow the user to intuitively understand that the current inclination of the user's fingers superimposed on the first feedback display screen Sc82 should be maintained. Therefore, the terminal device P1 can assist in the acquisition of a biometric image more suitable for biometric authentication.
[0153] In the eighth display example, the processor 11 may change the colors of the pitch pointer PT82A and the roll pointer PT82B displayed on the first feedback display screen Sc82 and the colors of the pitch pointer PT82A and the roll pointer PT82B displayed on the second feedback display screen Sc81.
[0154] (Variation of First Embodiment) In the terminal device P1 according to the first embodiment, in determining the tilt of a biometric part suitable for biometric authentication, the pitch angle θp = 0 (zero)° and the roll angle θr = 0 (zero)° are set as reference angles, and a captured image determined to have a tilt suitable for biometric authentication based on these reference angles is acquired as a biometric image. In the biometric authentication system 100 according to the variation of the first embodiment, the tilt of a biometric part when a pre-registered biometric image or biometric data is acquired is set as the reference angle, and an example is described below in which a captured image determined to have a tilt suitable for biometric authentication based on this reference angle is acquired as a biometric image.
[0155] The terminal device P1A in the modified example of the first embodiment has the same configuration as the terminal device P1 in the first embodiment. Therefore, in the following description of the biometric authentication system 100 according to the modified example of the first embodiment, the same components as those in the terminal device P1 in the first embodiment are denoted by the same reference numerals, and the description thereof will be omitted.
[0156] An example of the internal configuration of a biometric authentication system 100 according to a modification of the first embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the internal configuration of a biometric authentication system 100 according to a modification of the first embodiment.
[0157] The biometric authentication system 100 described below performs user authentication by, for example, face authentication based on a face image of the user and multimodal biometric authentication based on a biometric image of the user's body parts (for example, fingerprints, veins, palm prints, etc.), but is not limited to this. The biometric authentication system 100 may perform two or more user authentication processes, such as multi-factor authentication.
[0158] The biometric authentication system 100 according to the modified example of the first embodiment identifies an individual user in advance through a first optional authentication, and then acquires a biometric image to be used in a second biometric authentication based on the biometric image of the identified user or tilt information of the biometric part at the time of enrollment associated with the biometric data.
[0159] Terminal device P1A, as an example of a biometric image acquisition support device, captures a user's facial image to be used in the first user authentication and transmits it to server S1. Furthermore, based on the facial authentication result transmitted from server S1, terminal device P1A determines a reference angle for capturing an image of the user's biometric parts to be used in the second user authentication, and performs capturing (acquiring) the image of the user's biometric parts. Terminal device P1A may extract user features from the facial image and transmit the extracted user features to server S1, or similarly, extract user features from a biometric image and transmit the extracted user features to server S1.
[0160] The terminal device P1A includes a communication unit 10A, a processor 11A, a memory 12A, a camera 13A, and a monitor 14A. When multi-factor authentication is used, the terminal device P1A may have any configuration that enables multi-factor authentication.
[0161] The communication unit 10A, which is an example of an acquisition unit, is configured using a communication interface circuit for transmitting and receiving data to and from the server S1 via wired or wireless communication. The communication unit 10A transmits a facial image or biometric image output from the processor 11A to the server S1. The communication unit 10A acquires information on the face authentication result transmitted from the server S1 and outputs it to the processor 11A. Note that the wireless communication referred to here refers to communication via a wireless local area network (LAN) such as Wi-Fi (registered trademark).
[0162] The processor 11A, which is an example of a calculation unit, a control unit, and a detection unit, is configured using, for example, a CPU or FPGA, and performs various processes and controls in cooperation with the memory 12A. Specifically, the processor 11A references programs and data stored in the memory 12A and executes the programs to realize various functions related to imaging of biological parts.
[0163] The memory 12A includes, for example, a RAM as a work memory used when executing each process of the processor 11A, and a ROM for storing programs and data that define the operation of the processor 11A. The RAM temporarily stores data or information generated or acquired by the processor 11A. The ROM stores programs that define the operation of the processor 11A.
[0164] The camera 13A includes at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a CCD or CMOS solid-state image sensor, and converts an optical image formed on an imaging surface into an electrical signal.
[0165] The monitor 14A is configured using, for example, an LCD or an organic EL display, and outputs and displays images captured by the camera 13A, various screens generated by the processor 11A (such as the first feedback display screen or the second feedback display screen in the first to eighth display examples shown in FIGS. 4 to 10, respectively), or screens including the face authentication result or the biometric authentication result transmitted from the server S1.
[0166] The server S1 performs a first round of user authentication based on a facial image transmitted from the terminal device P1A. The server S1 generates a facial authentication result including tilt information of the identified biometric parts of the user as a result of the user authentication and transmits the result to the terminal device P1A. The server S1 performs a second round of user authentication based on the biometric image transmitted from the terminal device P1A. The server S1 includes a communication unit 20, a processor 21, a memory 22, a biometric database DB1A, and a facial image database DB2. Note that the biometric database DB1A and the facial image database DB2 may each be configured integrally with the server S1.
[0167] The communication unit 20 is configured using a communication interface circuit for transmitting and receiving data to and from the terminal device P1A via wired or wireless communication. The communication unit 20 transmits to the terminal device P1A either a facial authentication result using a facial image or a biometric authentication result using a biometric image output from the processor 21. The communication unit 20 acquires the facial image or biometric image transmitted from the terminal device P1A and outputs it to the processor 21.
[0168] The processor 21 is configured using, for example, a CPU or FPGA, and performs various processes and controls in cooperation with the memory 22. Specifically, the processor 21 refers to the programs and data stored in the memory 22 and executes the programs to realize various functions related to imaging of biological parts.
[0169] The memory 22 includes, for example, a RAM as a work memory used when the processor 21 executes each process, and a ROM for storing programs and data that define the operation of the processor 21. The RAM temporarily stores data or information generated or acquired by the processor 21. The ROM stores programs that define the operation of the processor 21.
[0170] The biometric database DB1A is a so-called storage device, and is configured using a storage medium such as a flash memory, HDD, or SSD. The biometric database DB1A stores (registers) a pre-registered biometric image of at least one person, information regarding the inclination of the biometric body parts shown in the biometric image, and user information, in association with each other. The biometric database DB1A may also store (register) biometric data extracted from the biometric image in association with the user information. Furthermore, when storing (registering) multiple biometric images, the biometric database DB1A may also store (register) each of the multiple biometric images, information regarding the biometric body parts corresponding to each of the biometric images, and user information in association with each other.
[0171] The facial image database DB2 is a so-called storage device, and is configured using a storage medium such as a flash memory, HDD, SSD, etc. The facial image database DB2 stores (registers) a facial image of at least one person registered in advance or feature amounts extracted from a facial image, in association with user information.
[0172] Next, an example of an operation procedure of the biometric authentication system 100 according to the modification of the first embodiment will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of an operation procedure of the biometric authentication system 100 according to the modification of the first embodiment.
[0173] In the sequence diagram shown in FIG. 12, the processes of steps St32 to St33 and steps St39 to St40 indicated by dashed lines are not essential and may be omitted.
[0174] The processor 11A of the terminal device P1A accepts a user operation to launch an application capable of capturing facial images and biological parts (Step 31). Based on the user operation, the processor 11A generates an operation instruction screen (not shown) requesting the user to perform a facial image capture operation, and outputs the operation instruction screen to the monitor 14A (Step 33). Here, the operation instruction screen includes, for example, a message such as "Please face the camera" (Step 32). The user faces the camera 13A (Step 34).
[0175] The processor 11A outputs a control command to the camera 13A to capture an image of the user's face. The processor 11A acquires the facial image output from the camera 13A (St35). The processor 11A outputs the acquired facial image to the communication unit 10A and causes it to be transmitted to the server S1 (St36). The processor 11A may also cause the server S1 to transmit biometric data of the user acquired from the facial image.
[0176] The processor 21 of the server S1 acquires the facial image transmitted from the terminal device P1A, compares the acquired facial image with the facial image of at least one person registered in the facial image database DB2, and performs facial authentication (St37).
[0177] The processor 21 acquires information about the inclination of the biological part associated with the user information (e.g., information about the reference angle corresponding to the registered biological image, the pitch angle θp, the roll angle θr, etc.) from the biometric database DB1A based on the user information of the user authenticated (identified) by facial recognition. The processor 21 associates the information about the inclination of the biological part with the facial recognition result, outputs it to the communication unit 20, and causes it to be transmitted to the terminal device P1A (St38).
[0178] The processor 11A outputs the facial authentication result transmitted from the server S1 to the monitor 14A. Furthermore, if the acquired facial authentication result indicates successful user authentication, the processor 11A generates an operation instruction screen (not shown) requesting the user to capture an image of a biological part, and outputs the operation instruction screen to the monitor 14A (St40). Here, the operation instruction screen includes, for example, a message such as "Please point your hand toward the camera" (St39). The user holds their hand (biological part) over the camera 13A (St41).
[0179] The processor 11A outputs a control command to the camera 13A to capture an image of the user's body part. The processor 11A acquires a captured image Img (see FIG. 3) of the user's entire hand or a body part output from the camera 13A (St42).
[0180] The processor 11A detects the three-dimensional coordinates of each joint point JNT (see FIG. 3) of the user's fingers based on the acquired captured image Img. The processor 11A calculates the pitch angle θp and roll angle θr (see FIG. 3) of the user's fingers (biological parts) shown in the captured image based on the three-dimensional coordinates of each detected joint point JNT. Note that the joints for which the three-dimensional coordinates are calculated may be at least only those joints corresponding to the biological parts.
[0181] The processor 11A sets the reference angle based on the information about the tilt of the body part transmitted from the server S1. The processor 11A calculates the angle difference between the calculated pitch angle θp and roll angle θr and the set reference angle (St43).
[0182] The processor 11A determines whether the angular differences between the calculated pitch angle θp and roll angle θr and the reference angle are both within a predetermined range (e.g., −5° to +5°) (St44). Note that the predetermined range here refers to the range of pitch angle θp and roll angle θr that indicates the inclination of a biometric part suitable for biometric data used in biometric authentication. A different range (value) may be set for each biometric part used in biometric authentication.
[0183] If the processor 11A determines in step St44 that the angular differences between the pitch angle θp and the roll angle θr and the reference angle are both within a predetermined range (e.g., −5° to +5°) (St44, YES), the processor 11A generates first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, and Sc82 (see FIGS. 4 and 6 to 10) that notify the user that the current inclination of the biometric body part is appropriate for biometric authentication. The processor 11A outputs the generated first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, and Sc82 to the monitor 14A for display (St45). In addition, instead of the first feedback display screen, the processor 11A may generate an audio signal to notify the user that the current inclination of the body part is suitable for biometric authentication, and output the audio signal from a speaker (not shown).
[0184] On the other hand, if the processor 11A determines in the processing of step St44 that the angular difference between each of the pitch angle θp and roll angle θr and the reference angle is not within a predetermined range (e.g., −5° to +5°, etc.) (St44, NO), it generates second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 (see Figures 5 to 10) that assist in adjusting the inclination of the current biological part based on the angular difference corresponding to each of the pitch angle θp and roll angle θr. The processor 11A outputs the generated second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 to the monitor 14A for display (St46).
[0185] Based on the captured image output from the camera 13A, the processor 11A determines whether the pitch angle θp and roll angle θr, which are the inclinations of the user's body parts, remain within a predetermined range for three seconds or more (St47).
[0186] The duration of "3 seconds" is merely an example and is not limited to this. The processor 11A may determine whether each of the pitch angle θp and the roll angle θr is within a predetermined range, that is, whether the inclination of the biometric part is maintained, for a period of time during which the processor 11A notifies the user that the current inclination of the biometric part is suitable for biometric authentication. Furthermore, in the processing of step St47, the processor 11A may perform the process of determining whether each of the pitch angle θp and the roll angle θr is within a predetermined range at any interval (e.g., every second).
[0187] If, in the processing of step St47, the processor 11A determines that the angular difference between the pitch angle θp and the roll angle θr remains within a predetermined range for three seconds or more (St47, YES), it acquires a biometric image from the images captured by the camera 13A to be used for biometric authentication (St48).
[0188] The captured image acquired as the biometric image may be an image captured in the processes of steps St42 to St47, or may be an image captured after the process of step St47. The captured image acquired as the biometric image may be an image in which the angle difference between the pitch angle θp and the reference angle and the angle difference between the roll angle θr and the reference angle are smallest among the captured images, an image in which a biometric part such as a fingerprint is most in focus, or the like.
[0189] On the other hand, if the processor 11A determines in the processing of step St47 that the angular difference between the pitch angle θp and the roll angle θr does not remain within the predetermined range for more than 3 seconds (St47, NO), it returns to the processing of step St41.
[0190] The processor 11A outputs the acquired biometric image to the communication unit 10A and transmits it to the server S1 (St49). Note that the processor 11A may transmit biometric data extracted from the biometric image instead of the biometric image itself.
[0191] The processor 21 performs biometric authentication (user authentication) using the biometric image transmitted from the terminal device P1A (St50). The processor 21 performs biometric authentication by comparing the acquired biometric image with the biometric image of at least one person registered in the biometric database DB1A, generates a biometric authentication result, and transmits it to the terminal device P1A. Note that the processor 21 may also perform biometric authentication using the biometric data transmitted from the terminal device P1A and the biometric data registered in the biometric database DB1A.
[0192] As described above, even when a biometric image that is not acquired in a state where the pitch angle and roll angle are not set to 0° and 0°, i.e., a biometric image that is not acquired in a state substantially parallel to the light receiving surface of the image sensor (not shown) of the camera 13, is registered in advance, the biometric authentication system 100 according to the variation of the first embodiment can assist in acquiring a biometric image that is suitable for biometric authentication with the registered biometric image by setting the reference angle used for acquiring the biometric image to a reference angle that corresponds to the registered biometric image. This allows the biometric authentication system 100 to more effectively suppress a decrease in biometric authentication accuracy.
[0193] As described above, the terminal devices P1, P1A (an example of a biometric image acquisition support device, a computer) according to the first embodiment and the modified example of the first embodiment comprise a communication unit 10, 10A (an example of an acquisition unit) that acquires an image Img of a biometric part (e.g., a fingertip, a finger, or a palm) of a user (an example of a subject), a processor 11, 11A (an example of a calculation unit) that calculates the inclination of the biometric part with respect to a reference angle based on the acquired image Img, and the processor 11, 11A (an example of a control unit) that, if it is determined that the inclination of the biometric part is not within a predetermined range with respect to the reference angle, generates a second feedback display screen (an example of a posture guidance screen) that includes information regarding the inclination of the biometric part and the captured image Img and outputs the generated screen to the monitor 14, 14A.
[0194] As a result, when the terminal devices P1 and P1A according to embodiment 1 and the variant of embodiment 1 determine that the inclination (angle difference) of a biological part relative to a reference angle for acquiring a biometric image suitable for biometric authentication when acquiring a biometric image of a biological part used for biometric authentication is not within a predetermined range usable for biometric authentication, they can notify the user of the inclination of the biological part and provide assistance in acquiring a biometric image suitable for biometric authentication by generating a second feedback display screen that associates information regarding the calculated inclination of the biological part with the captured image Img in which the biological part is captured.
[0195] Furthermore, the information relating to the tilt of the body part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment includes information indicating the tilt direction of the body part, which allows the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment to visualize and notify the user of the tilt direction of the body part, thereby providing support for acquiring a biometric image suitable for biometric authentication.
[0196] Furthermore, the information relating to the tilt of the biological part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment includes information on the tilt direction and the tilt magnitude of the biological part. As a result, the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment can visualize and notify the user of the tilt direction and the tilt angle of the biological part, thereby providing assistance in obtaining a biometric image suitable for biometric authentication.
[0197] Furthermore, the information regarding the tilt of the body part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment includes information indicating the direction in which the user should tilt the body part. This allows the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment to visualize and notify the user of the direction in which the body part should be tilted, thereby providing support for acquiring a biometric image suitable for biometric authentication.
[0198] Furthermore, the information regarding the tilt of the biological part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment includes information indicating the direction and the amount by which the user should tilt the biological part. As a result, the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment can visualize and notify the user of the direction and the amount of the angle by which the biological part should be tilted, thereby providing support for obtaining a biometric image suitable for biometric authentication.
[0199] Furthermore, the processors 11 and 11A in the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment calculate the inclination of the biological part relative to the light receiving surface of the image sensor of the camera 13 that captured the captured image Img. This enables the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment to support the acquisition of a biometric image in which the angle difference between the biological part used for biometric authentication and the light receiving surface of the image sensor is within a predetermined range.
[0200] Furthermore, the communication unit 10A in the terminal device P1A according to the modification of the first embodiment acquires information regarding the inclination of a biometric part appearing in another captured image of the user (for example, a captured image of the user registered in the face image database DB2). The processor 11, 11A sets the inclination of the biometric part appearing in the other captured image as a reference angle, and calculates the inclination of the biometric part with respect to the reference angle. As a result, when a biometric part is captured in an inclined state and a biometric image that has already been registered exists, the terminal device P1A according to the modification of the first embodiment can support the acquisition of a biometric image that can improve the accuracy of biometric authentication with the biometric part appearing in the registered biometric image by setting the inclination of the biometric part appearing in the registered biometric image as the reference angle.
[0201] Furthermore, the predetermined range in the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment is between −5° and +5° with respect to the reference angle. As a result, the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment capture an image with the light receiving surface of the image sensor and the biological part substantially parallel to each other, which can assist in obtaining a biometric image that is more suitable for biometric authentication.
[0202] The terminal device P1, P1A according to the first embodiment and the modification of the first embodiment further includes a processor 11, 11A (an example of a detector) capable of detecting a plurality of joint points JNT of a body part of a user from a captured image Img acquired in the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment. The processor 11, 11A calculates the tilt of the body part with respect to a reference angle based on the positions of the detected plurality of joint points JNT. This allows the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment to calculate the tilt of the body part with higher accuracy.
[0203] Furthermore, the tilt of the body part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment is calculated based on the pitch axis PLp (an example of a first direction) and the roll axis PLr (an example of a second direction) that is substantially perpendicular to the pitch axis PLp. The processors 11 and 11A calculate the tilt of the body part based on the positions of any two of the multiple joint points along the pitch axis PLp and the positions of any two of the multiple joint points along the roll axis PLr. This allows the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment to calculate the tilt of the body part as the way the body part appears in a captured image.
[0204] Furthermore, when the biological part is one finger (e.g., a fingerprint), the processor 11, 11A in the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment calculates the tilt of the biological part based on the positions of two or more joint points of the one finger and the positions of the joint points of the fingertips of any two fingers. This allows the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment to determine whether the tilt of one of the fingers to be biometrically authenticated is an tilt that allows for the acquisition of a biometric image suitable for biometric authentication.
[0205] Furthermore, when the biological part is one of the fingers (for example, a vein), the processor 11, 11A in the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment calculates the tilt of the biological part based on the positions of two or more joint points of the one finger and the positions of the second joints of any two fingers. This allows the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment to determine whether the tilt of any of the fingers to be biometrically authenticated is an tilt that allows for the acquisition of a biometric image suitable for biometric authentication.
[0206] Furthermore, when the biological part is a palm (e.g., a palm print), the processors 11 and 11A in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment calculate the tilt of the biological part based on the positions of the wrist joint points, the positions of the joint points of either the index finger, the middle finger, or the ring finger, the position of the joint point at the base of the little finger, and the position of the joint point at the base of the index finger or the thumb. This allows the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment to determine whether the tilt of the palm that is the target of biometric authentication is an tilt that allows for the acquisition of a biometric image suitable for biometric authentication.
[0207] Furthermore, the processor 11, 11A in the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment calculates, for each finger, the distance between two adjacent joint points among multiple joint points of the finger in the biological part. The processor 11, 11A determines whether the finger is bent based on the calculated distance between the two joint points, and if it determines that the finger is bent, generates a second feedback display screen including instruction information instructing the user to spread the fingers, information about the tilt of the biological part, and the captured image Img, and outputs the second feedback display screen to the monitor 14, 14A. This allows the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment to detect not only the tilt of the biological part but also the bending of the finger in the biological part, thereby supporting the acquisition of a biometric image more suitable for biometric authentication.
[0208] Furthermore, when the processor 11, 11A in the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment determines that the tilt of the body part is within a predetermined range with respect to the reference angle, the processor 11, 11A generates a first feedback display screen (an example of a notification screen) notifying the user that the tilt of the body part is within the predetermined range, and outputs the first feedback display screen to the monitor 14, 14A. This allows the terminal device P1, P1A according to the first embodiment and the modification of the first embodiment to notify the user that the tilt of the body part is suitable for biometric authentication.
[0209] Furthermore, the processors 11 and 11A in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment acquire captured images Img in which the tilt of the biological parts falls within a predetermined range, and output and store the captured images Img in the fingerprint database DB1 (an example of a storage medium). This allows the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment to acquire only biometric images suitable for biometric authentication and register them as biometric images to be used for biometric authentication.
[0210] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that those skilled in the art can conceive of various modifications, alterations, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also fall within the technical scope of the present disclosure. Furthermore, the components of the various embodiments described above may be combined in any manner without departing from the spirit of the invention.
[0211] This application is based on a Japanese patent application (Patent Application No. 2023-088149) filed on May 29, 2023, the contents of which are incorporated herein by reference.
[0212] INDUSTRIAL APPLICABILITY The present disclosure is useful as providing a biometric image acquisition assistance device and a biometric image acquisition assistance method that assist in the acquisition of biometric data more suitable for biometric authentication in a non-contact state.
[0213] 10, 10A, 20 Communication unit 11, 11A, 21 Processor 12, 12A, 22 Memory 13, 13A Camera 14, 14A Monitor 15 Operation unit 100 Biometric authentication system DB1 Fingerprint database DB1A Biometric database DB2 Facial image database P1, P1A Terminal device S1 Server Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, Sc82 First feedback display screen Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, Sc81 Second feedback display screen
Claims
1. A biometric image acquisition support device comprising: an acquisition unit that acquires an image of a biological part of a subject; a calculation unit that calculates the inclination of the biological part with respect to a reference angle based on the acquired image; and a control unit that, when it is determined that the inclination of the biological part is not within a predetermined range with respect to the reference angle, generates a posture guidance screen including information regarding the inclination of the biological part and the captured image, and outputs the generated screen to a monitor.
2. The biometric image acquisition support device according to claim 1, wherein the information relating to the inclination of the biometric part includes information indicating the inclination direction of the biometric part.
3. The biometric image acquisition support device according to claim 2, wherein the information relating to the inclination of the biometric part includes information indicating the direction and magnitude of the inclination of the biometric part.
4. The biometric image acquisition support device according to claim 1, wherein the information regarding the inclination of the biometric part includes information indicating a direction in which the subject is to incline the biometric part.
5. The biometric image acquisition support device according to claim 4, wherein the information regarding the inclination of the biometric part includes information indicating a direction and a magnitude in which the subject is to incline the biometric part.
6. The biometric image acquisition support device according to claim 1, wherein the calculation unit calculates an inclination of the biometric part relative to a light receiving surface of an image sensor of a camera that captured the captured image.
7. The biometric image acquisition support device of claim 1, wherein the acquisition unit acquires information regarding the inclination of the biometric part appearing in other captured images of the subject, and the calculation unit sets the inclination of the biometric part appearing in the other captured images as a reference angle and calculates the inclination of the biometric part with respect to the reference angle.
8. The biometric image acquisition support device according to claim 1, wherein the predetermined range is equal to or greater than -5° and equal to or less than +5° with respect to the reference angle.
9. The biometric image acquisition assistance device of claim 1, further comprising a detection unit capable of detecting multiple joint points of the subject's biometric part from the acquired captured image, wherein the calculation unit calculates the inclination of the biometric part with respect to the reference angle based on the positions of the detected multiple joint points.
10. The biometric image acquisition support device of claim 9, wherein the inclination of the biological part is calculated based on a first direction and a second direction approximately perpendicular to the first direction, and the calculation unit calculates the inclination of the biological part based on the positions of any two joint points among the multiple joint points that are aligned along the first direction and the positions of any two joint points among the multiple joint points that are aligned along the second direction.
11. The biometric image acquisition support device of claim 9, wherein when the biometric part is one of the fingers, the calculation unit calculates the inclination of the biometric part based on the positions of two or more joint points of the one finger and the positions of the joint points of the fingertips of any two fingers.
12. The biometric image acquisition support device of claim 9, wherein when the biometric part is one of the fingers, the calculation unit calculates the inclination of the biometric part based on the positions of two or more joint points of the one finger and the positions of second joints of any two fingers.
13. The biometric image acquisition support device of claim 9, wherein the calculation unit calculates the inclination of the biometric part based on the position of the wrist joint point, the position of the joint point of either the index finger, middle finger or ring finger, the position of the joint point of the base of the little finger, and the position of either the joint point of the base of the index finger or the thumb when the biometric part is the palm.
14. The biometric image acquisition support device of claim 1, wherein the calculation unit calculates for each finger the distance between two adjacent joint points among a plurality of joint points of the fingers of the biological part, and the control unit determines whether the finger is bent or not based on the calculated distance between the two joint points, and if it determines that the finger is bent, generates the posture guidance screen including instruction information instructing the fingers to spread, information regarding the inclination of the biological part, and the captured image, and outputs it to the monitor.
15. The biometric image acquisition support device of claim 1, wherein when the control unit determines that the inclination of the body part is within the specified range with respect to the reference angle, it generates a notification screen notifying that the inclination of the body part is within the specified range and outputs the notification screen to the monitor.
16. The biometric image acquisition assistance device according to claim 1, wherein the control unit acquires captured images in which the inclination of the biometric part is within the predetermined range, and outputs and stores the captured images in a storage medium.
17. A biometric image acquisition support method performed by a computer, comprising: acquiring an image of a biological part of a subject; calculating an inclination of the biological part with respect to a reference angle based on the acquired image; and, if it is determined that the inclination of the biological part is not within a predetermined range with respect to the reference angle, generating a posture guidance screen including information regarding the inclination of the biological part and the acquired image, and outputting the screen to a monitor.
Citation Information
Patent Citations
Image registration device, image registration system, and image registration method
JP2019028959A
Information processing apparatus and information processing method
JP2023011333A
Cited By
System for suitability assessment of biometric input
US12775690B1
