Biological image acquisition assistance device and biological image acquisition assistance method

Abstract

To support acquisition of biometric data more suitable for biometric authentication in a contactless state.SOLUTION: A biological image acquisition assistance device includes: an acquisition unit configured to acquire a captured image in which a biological site of a subject is captured; a calculation unit configured to calculate, based on the acquired captured image, the inclination of the biological site with respect to a reference angle; and a control unit configured to determine whether the inclination of the biological site is within a predetermined range with respect to the reference angle, and acquire a captured image in which the inclination of the biological site is within the predetermined range. When it is determined that the inclination of the biological site is not within the predetermined range, the control unit generates an orientation guidance screen including the captured image and information pertaining to the inclination of the biological site and outputs the orientation guidance screen to a monitor.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a biometric image acquisition assistance device and a biometric image acquisition assistance method. [Background technology]

[0002] Patent Literature 1 discloses a method for performing fingerprint recognition using a mobile device, which extracts features of each of a plurality of fingers of a subject, generates a biometric identifier including the extracted features, and stores the generated biometric identifier in a memory. The method for performing fingerprint recognition includes capturing an image of a plurality of fingers of a subject by a camera, a storage medium, instructions stored in the storage medium, and a mobile device executing the instructions, detecting the plurality of fingers depicted in the image by a fingertip detection algorithm, identifying fingertip segments of each of the plurality of fingers, extracting features of the plurality of fingers from each identified fingertip segment, and generating a biometric identifier including the extracted features and storing the generated biometric identifier in a memory. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2020-501264 Summary of the Invention [Problem to be solved by the invention]

[0004] In the configuration of Patent Document 1, markings or guides are displayed on the display to instruct the user on how to position the fingers, and the user's fingers are guided to be imaged. However, since the size of the fingers varies from user to user depending on gender, age, etc., there is a possibility that the fingerprints are not in focus in the images of multiple fingers captured by the method of the conventional Patent Document 1.

[0005] The present disclosure has been devised in consideration of the above-described conventional circumstances, and aims to provide a biometric image acquisition assistance device and a biometric image acquisition assistance method that assist in the acquisition of biometric data more suitable for biometric authentication in a non-contact state. [Means for solving the problem]

[0006] The present disclosure provides a biometric image acquisition support device comprising: an acquisition unit that acquires an image of a subject's biometric part; a calculation unit that calculates the inclination of the biometric part relative to a reference angle based on the acquired image; and a control unit that, when it is determined that the inclination of the biometric part is not within a predetermined range relative to the reference angle, generates a posture guidance screen including information regarding the inclination of the biometric part and the acquired image, and outputs the generated screen to a monitor.

[0007] The present disclosure also provides a biometric image acquisition support method performed by a computer, which acquires an image of a biological part of a subject, calculates an inclination of the biological part with respect to a reference angle based on the acquired image, and if it is determined that the inclination of the biological part is not within a predetermined range with respect to the reference angle, generates a posture guidance screen including information regarding the inclination of the biological part and the acquired image, and outputs the image to a monitor. Effect of the Invention

[0008] According to the present disclosure, it is possible to assist in obtaining biometric data that is more suitable for biometric authentication in a non-contact state. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of an internal configuration of a terminal device according to a first embodiment; [Diagram 2] A flowchart showing an example of an operation procedure of a terminal device according to the first embodiment. [Diagram 3] FIG. 1 is a diagram for explaining an example of an operation procedure of a terminal device according to the first embodiment; [Figure 4] FIG. 11 is a diagram for explaining an example of a first feedback display screen in the first display example; [Diagram 5] FIG. 11 is a diagram for explaining an example of a second feedback display screen in the first display example; [Figure 6] FIG. 11 is a diagram illustrating a first feedback display screen example and a second feedback display screen example in the second display example. [Figure 7] FIG. 13 is a diagram illustrating a first feedback display screen example and a second feedback display screen example in the third display example. [Figure 8] FIG. 13 is a diagram illustrating a first feedback display screen example and a second feedback display screen example in the fourth display example. [Figure 9] FIG. 13 is a diagram for explaining a first feedback display screen example and a second feedback display screen example in the fifth and sixth display examples; [Figure 10] FIG. 13 is a diagram for explaining a first feedback display screen example and a second feedback display screen example in the seventh and eighth display examples; [Figure 11] FIG. 1 is a block diagram showing an example of an internal configuration of a biometric authentication system according to a modification of the first embodiment. [Figure 12] FIG. 11 is a sequence diagram showing an example of an operation procedure of a biometric authentication system according to a modification of the first embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] (Background to this disclosure) Conventionally, there is a fingerprint registration method in which a fingerprint is registered on a terminal device owned by a user. However, when a user captures a fingerprint by himself, the way in which the hand is held over the terminal device, that is, the posture of the hand held over the terminal device, changes at each capture timing. Therefore, the way in which the fingerprint is captured in the captured image during authentication differs from the way the fingerprint is captured during registration, and there is a possibility that the accuracy of biometric authentication using fingerprints will decrease.

[0011] In the past, a method for acquiring a fingerprint image was devised that allows a user to check whether or not the fingerprint on the fingertip is in focus by displaying an enlarged image of the fingertip captured in a captured image on the screen of a terminal device. However, it was difficult for a user to determine whether or not the current way of holding the hand is suitable for biometric authentication based on the enlarged image displayed on the screen of the terminal device, or to adjust the way the hand is held while viewing the enlarged image.

[0012] Therefore, in each of the embodiments described below, examples of a biometric image acquisition support device and a biometric image acquisition support method that can support the acquisition of biometric data more suitable for biometric authentication in a non-contact state will be described.

[0013] Hereinafter, with reference to the drawings as appropriate, each embodiment that specifically discloses the configuration and operation of the biometric image acquisition support device and biometric image acquisition support method according to the present disclosure will be described in detail. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of already well-known matters and duplicate explanation of substantially the same configuration may be omitted. This is to avoid the following explanation becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the attached drawings and the following explanation are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0014] (Embodiment 1) An example of the internal configuration of the terminal device P1 according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the internal configuration of the terminal device P1 according to the first embodiment.

[0015] A terminal device P1 as an example of a biometric image acquisition support device captures an image of a user's entire hand or a part used for biometric authentication (hereinafter referred to as a "biometric part") and detects joint points of the user's biometric part captured in the captured image. Based on the detected joint points of the user's biometric part, the terminal device P1 calculates the pitch angle and roll angle of the biometric part as the posture of the user's biometric part captured in the captured image (i.e., the inclination of the user's biometric part with respect to the light receiving surface of the image sensor (not shown) of the camera 13). Based on the calculated pitch angle and roll angle of the biometric part, the terminal device P1 generates and outputs a second feedback display screen (see FIGS. 5 to 10) for adjusting the posture (inclination) to one that allows biometric data suitable for biometric authentication to be acquired.

[0016] In the following description, an example of acquiring biometric data used for biometric authentication using a fingerprint will be described, but the present invention is not limited thereto. For example, biometric authentication may be authentication using veins or palm prints. In the case of biometric authentication using veins, the terminal device P1 captures an image of an area including at least one joint other than the fingertip as a biometric part, and in the case of biometric authentication using palm prints, the terminal device P1 captures an image of the palm as a biometric part and detects the posture (tilt) of the biometric part.

[0017] The terminal device P1 includes a communication unit 10, a processor 11, a memory 12, a camera 13, and a monitor 14. Note that the terminal device P1 shown in Fig. 1 is configured separately from the fingerprint database DB1, but is not limited to this. For example, the terminal device P1 may be configured integrally with the fingerprint database DB1.

[0018] The communication unit 10, which is an example of an acquisition unit, is configured using a communication interface circuit for transmitting and receiving data. The communication unit 10 is connected to the fingerprint database DB1 so as to be able to communicate wirelessly or via wire, and transmits and receives data.

[0019] The processor 11, which is an example of a calculation unit, a control unit, and a detection unit, is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU"), a Graphics Processing Unit (hereinafter referred to as "GPU"), or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and performs various processes and controls in cooperation with the memory 12. Specifically, the processor 11 refers to the programs and data stored in the memory 12, and executes the programs to realize various functions (for example, a function to detect the inclination of a biological part, a support function to allow the user to adjust the inclination of the biological part, etc.) executed to obtain biometric data suitable for biometric authentication.

[0020] The processor 11 associates the captured image (hereinafter referred to as "biometric image") or biometric data determined to be suitable for biometric authentication with user information (e.g., the user's name, gender, date of birth, etc.) and stores (registers) it in the fingerprint database DB1.

[0021] The biometric data referred to here is a pattern of a body part extracted from a biometric image (for example, a fingerprint pattern, a vein pattern, or a palm print pattern) or a feature amount indicating the individuality of a user extracted from a biometric image.

[0022] The memory 12 has, for example, a random access memory (hereinafter referred to as "RAM") as a work memory used when executing each process of the processor 11, and a read only memory (hereinafter referred to as "ROM") for storing programs and data that define the operation of the processor 11. The RAM temporarily stores data or information generated or acquired by the processor 11. The ROM has written therein programs that define the operation of the processor 11.

[0023] The camera 13 is configured to have at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a solid-state imaging element such as a Charged-Coupled Device (hereinafter referred to as "CCD") or a Complementary Metal Oxide Semiconductor (hereinafter referred to as "CMOS"), and converts an optical image formed on an imaging surface into an electrical signal.

[0024] The monitor 14 is configured using, for example, a Liquid Crystal Display (hereinafter referred to as "LCD") or an organic electroluminescence (hereinafter referred to as "EL"). The monitor 14 outputs and displays images captured by the camera 13 or various screens generated by the processor 11 (such as the first feedback display screen or the second feedback display screen in the first to eighth display examples shown in Figs. 4 to 10, respectively).

[0025] The fingerprint database DB1 is a so-called storage, and is configured using a storage medium such as a flash memory, a hard disk drive (hereinafter referred to as "HDD"), or a solid state drive (hereinafter referred to as "SSD"). The fingerprint database DB1 stores (registers) a biometric image generated by the processor 11 in association with user information. The fingerprint database DB1 may also store (register) biometric data extracted from a biometric image in association with user information. Furthermore, when storing (registering) each of a plurality of biometric images, the fingerprint database DB1 may store (register) each of the plurality of biometric images in association with information on a biometric part corresponding to each of the biometric images (for example, information on a middle finger and an index finger) and user information.

[0026] Next, an example of an operation procedure of the terminal device P1 will be described with reference to Fig. 2. Fig. 2 is a flowchart showing an example of an operation procedure of the terminal device P1 according to the first embodiment. In addition, in the example of the operation procedure of the terminal device P1 shown in Fig. 2, each of step St16 and step St19 is not essential and may be omitted.

[0027] The processor 11 acquires a captured image Img (see FIG. 3) in which the entire hand or a body part of the user is captured by the camera 13 (St11).

[0028] The processor 11 detects the three-dimensional coordinates of each joint point JNT (see FIG. 3) based on the acquired captured image Img (St12). The processor 11 estimates the skeleton SL (see FIG. 3) of the user's fingers based on each of the calculated joint points JNT. Note that the joint points detected here may be only the biological parts corresponding to at least the biological parts, or may be the entire hand. The skeletal parts to be detected in each biometric authentication (fingerprint authentication, vein authentication, palm print authentication, etc.) will be described later.

[0029] The processor 11 calculates the pitch angle θp and roll angle θr (see FIG. 3) of the user's fingers (biological parts) captured in the captured image Img based on the calculated three-dimensional coordinates of each joint point JNT (St13). The processor 11 calculates the angle difference between the pitch angle θp and a reference angle (e.g., 0 (zero)°, etc.) and the angle difference between the roll angle θr and the reference angle based on the calculated pitch angle θp and roll angle θr (St14).

[0030] The processor 11 determines whether or not the angle difference between each of the calculated pitch angle θp and roll angle θr and the reference angle is within a predetermined range (for example, −5° to +5°, etc.) (St15). That is, the processor 11 determines whether or not the inclination of the biological part calculated as the pitch angle θp and the roll angle θr is within a predetermined range with respect to the light receiving surface of the image sensor (not shown) of the camera 13, which is the reference angle. Note that the pitch angle θp is an angle centered on the pitch axis PLp (an example of the first direction, see FIG. 3). The roll angle θr is an angle centered on the roll axis PLr (an example of the second direction, see FIG. 3). Each of the pitch axis PLp and the roll axis PLr is a reference angle with respect to the biological part (fingers) of the user shown in the captured image Img, that is, two axes that can define a horizontal plane parallel to the light receiving surface of the image sensor (not shown) of the camera 13.

[0031] The predetermined range here refers to the range of the angle difference between the pitch angle θp and the roll angle θr, which indicate the inclination of a biometric part that can be used as biometric data for biometric authentication. The predetermined range may be set to a different range (value) for each biometric part used for biometric authentication.

[0032] When the processor 11 determines in the process of step St15 that the angle difference between each of the pitch angle θp and the roll angle θr and the reference angle is within a predetermined range (St15, YES), the processor 11 calculates the distance between two adjacent joint points among the joint points corresponding to each finger. For example, in the case of the middle finger, the processor 11 calculates, among each of the joint points "9" to "12", the distance between the joint point "12" and the joint point "11", the distance between the joint point "11" and the joint point "10", and the distance between the joint point "10" and the joint point "9". The processor 11 further determines whether or not the calculated distances between two joint points are all equal to or greater than a predetermined value (predetermined distance) (St16). The processor 11 may determine whether or not the ratios of the distances are all equal to or greater than a predetermined value in the process of step St16.

[0033] The predetermined value here is a value for determining whether or not a finger is bent, and indicates the minimum distance between joint points or a ratio to the distance between each joint point. The predetermined value is set to the same or different values ​​for each finger, such as the middle finger or index finger, and for each joint, such as the first joint or second joint.

[0034] Specifically, the predetermined value may be set for each finger and may be the minimum distance between two adjacent joint points, such as the minimum distance between the joint point "12" and the joint point "11" of the tip of the middle finger, the minimum distance between the joint point "11" and the joint point "10" of the middle finger, the minimum distance between the joint point "8" and the joint point "7" of the tip of the index finger, the minimum distance between the joint point "16" and the joint point "15" of the index finger, etc. Also, the predetermined value may be set for each finger and may be the ratio of the distance between two adjacent joint points (i.e., the length of the joint), such as the ratio of the length of the first joint of the middle finger (the distance between the joint point "12" and the joint point "11") and the second joint of the middle finger (the distance between the joint point "11" and the joint point "10").

[0035] On the other hand, when the processor 11 determines in the process of step St15 that the angle difference between each of the pitch angle θp and the roll angle θr and the reference angle is not within the predetermined range (St15, NO), the processor 11 generates second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 (examples of posture guidance screens, see Figs. 5 to 10) that visualize the current inclination of the body part based on the inclination of the body part indicated by each of the calculated pitch angle θp and roll angle θr and assist the user in adjusting the inclination of the body part. The processor 11 outputs the generated second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 to the monitor 14 for display (St17).

[0036] Furthermore, when the processor 11 determines in the process of step St16 that all the calculated distances between the two joint points are equal to or greater than a predetermined value (St16, YES), it determines that the user's fingers are not bent (i.e., not clenched), and generates first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, and Sc82 (examples of notification screens, see Figs. 4, 6 to 10) to notify the user that the current inclination of the body part is suitable for biometric authentication. The processor 11 outputs the generated first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, and Sc72 to the monitor 14 for display (St18). Note that the processor 11 may generate an audio signal to notify the user that the current inclination of the body part is suitable for biometric authentication, instead of the first feedback display screen, and output the audio signal from a speaker (not shown).

[0037] The processor 11 determines whether or not the state in which the pitch angle θp and the roll angle θr, which indicate the inclination of the user's body part, are within a predetermined range has continued for three seconds or more (St20).

[0038] On the other hand, if the processor 11 determines in the processing of step St16 that all of the calculated distances between the two joint points are not greater than or equal to a predetermined value (St16, NO), it determines that the user's fingers are bent (i.e., clenched), generates a second feedback display screen including a control command or message (e.g., "Please spread your hand and fingers") instructing the user to spread his / her fingers, and outputs it to the monitor 14 (St19).

[0039] If the processor 11 determines in the processing of step St20 that the angular difference between each of the pitch angle θp and roll angle θr and the reference angle remains within a predetermined range for three seconds or more (St20, YES), it acquires the captured image Img captured by the camera 13 as a biometric image of the user to be used for biometric authentication (St21).

[0040] The duration of "3 seconds" is merely an example and is not limited to this. The duration may be a time during which the processor 11 can determine whether the pitch angle θp and the roll angle θr are within a predetermined range, that is, whether the inclination of the biometric part is maintained, while notifying the user that the current inclination of the biometric part is suitable for biometric authentication.

[0041] On the other hand, if the processor 11 determines in the process of step St20 that the state in which the angular differences between the pitch angle θp and the roll angle θr are within the predetermined range does not continue for three seconds or more (St20, NO), the processor 11 returns to the process of step St11.

[0042] As described above, the terminal device P1 according to the first embodiment can determine whether or not the inclination (angle difference) of the biometric part of the user captured in the captured image allows acquisition of a biometric image suitable for biometric authentication by the process of step St15. When the terminal device P1 determines that the inclination (angle difference) of the biometric part is equal to or greater than a predetermined range and that a biometric image suitable for biometric authentication cannot be acquired, the terminal device P1 can notify the user of the inclination of the biometric part and can provide support for acquiring a biometric image suitable for biometric authentication by generating a second feedback display screen in which information regarding the calculated inclination of the biometric part is associated with the captured image Img in which the biometric part is captured. Thus, the terminal device P1 can suppress variation in how the biometric part appears in the captured image and more effectively support acquisition of a biometric image suitable for biometric authentication.

[0043] Furthermore, the terminal device P1 according to the first embodiment can determine whether each finger is clenched and bent by the process of step St16. This allows the terminal device P1 to eliminate the bent fingers and more effectively support the acquisition of a biometric image suitable for biometric authentication using a fingerprint or vein.

[0044] Next, a method for calculating the pitch angle θp and the roll angle θr will be described with reference to Fig. 3. Fig. 3 is a diagram for explaining an example of an operation procedure of the terminal device P1 according to the first embodiment. In Fig. 3, reference symbols are omitted for all joint points in order to make the drawing easier to see.

[0045] The processor 11 calculates the three-dimensional coordinates of a plurality of joint points JNT (each of the 21 joint points "0" to "20") of the user's hand H from the captured image Img in which the entire hand H of the user is captured.

[0046] The processor 11 calculates the pitch angle θp and the roll angle θr based on the calculated three-dimensional coordinates of each joint point JNT. Note that the pitch axis PLp shown in Fig. 3 is a pitch angle θp = 0 (zero) °, which indicates the reference angle of the pitch angle. The roll axis PLr is a roll angle θr = 0 (zero) °, which indicates the reference angle of the roll angle.

[0047] For example, when capturing an image of a fingertip suitable for fingerprint authentication as an example of a biological part, the processor 11 calculates a pitch angle θp of the user's hand H based on the three-dimensional coordinates of each of the joint points "9" at the base of the middle finger to "12" at the tip of the middle finger. Also, when capturing an image suitable for fingerprint authentication, the processor 11 calculates a roll angle θr of the user's hand H based on the three-dimensional coordinates of each of the joint points "8" at the tip of the index finger and "16" at the tip of the ring finger.

[0048] Here, other calculation examples of the pitch angle θp and the roll angle θr will be described. Note that the calculation methods of the pitch angle θp and the roll angle θr corresponding to each body part described below are merely examples, and are not limited to these.

[0049] When capturing an image of a finger suitable for fingerprint authentication as an example of a biological part, for example, the processor 11 calculates the pitch angle θp of the finger from which the fingerprint is to be obtained based on the three-dimensional coordinates of the joint point of the fingertip of the finger from which the fingerprint is to be obtained and the three-dimensional coordinates of any one of the joint points other than the fingertip of the finger. Specifically, when the finger from which the fingerprint is to be obtained is the middle finger, for example, the processor 11 calculates the pitch angle θp of the finger from which the fingerprint is to be obtained based on the three-dimensional coordinates of the joint point "12" of the middle fingertip and the three-dimensional coordinates of any one of the joint points "9" to "11" other than the tip of the middle finger.

[0050] Also, for example, processor 11 calculates the roll angle θr of the finger whose fingerprint is to be obtained based on the three-dimensional coordinates of joint point "8" at the tip of the index finger and the three-dimensional coordinates of joint point "16" at the tip of the ring finger or joint point "20" at the tip of the little finger.

[0051] Furthermore, when capturing an image of a finger suitable for vein authentication as an example of a biological part, for example, the processor 11 calculates a pitch angle θp of the finger from which veins are to be acquired based on the three-dimensional coordinates of a joint point near the tip of the finger from which veins are to be acquired and the three-dimensional coordinates of a joint point near the base of the finger. Specifically, when the finger from which veins are to be acquired is the middle finger, for example, the processor 11 calculates the pitch angle θp of the middle finger from which veins are to be acquired based on the three-dimensional coordinates of joint point "11" or joint point "12" near the tip of the middle finger and the three-dimensional coordinates of joint point "9" or joint point "10" near the base of the middle finger.

[0052] Also, for example, the processor 11 calculates the roll angle θr of the finger from which veins are to be acquired, based on the three-dimensional coordinates of the second joint point "6" of the fingertip of the index finger and the three-dimensional coordinates of the second joint point "14" of the ring finger.

[0053] Furthermore, when capturing an image of a palm suitable for palm-print authentication as an example of a biological part, for example, the processor 11 calculates the pitch angle θp of the finger from which the palm print is to be acquired based on the three-dimensional coordinates of the wrist joint point "0" and the three-dimensional coordinates of any one of the joint points "9" to "12" of the middle finger. The processor 11 may calculate the pitch angle θp based on the three-dimensional coordinates of the wrist joint point "0" and the three-dimensional coordinates of any one of the joint points "13" to "16" of the ring finger, or may calculate the pitch angle θp based on the three-dimensional coordinates of any two of the joint points "5" to "8" of the index finger.

[0054] Processor 11 also calculates the roll angle θr based on the three-dimensional coordinates of joint point "17" at the base of the little finger and the three-dimensional coordinates of joint point "5" at the base of the index finger. Processor 11 may also calculate the roll angle θr based on the three-dimensional coordinates of joint point "17" at the base of the little finger and the three-dimensional coordinates of any one of joint points "1" to "4" of the thumb.

[0055] As described above, the terminal device P1 can calculate the pitch angle θp and roll angle θr, which indicate the inclination of the user's biometric parts, based on the three-dimensional coordinates of the joint points corresponding to the biometric parts used in biometric authentication. This allows the terminal device P1 to more effectively suppress the variation in how the biometric parts are captured and obtain a biometric image that is more suitable for biometric authentication. Therefore, the terminal device P1 can realize more accurate biometric authentication.

[0056] <First display example> Next, the first display example will be described with reference to Fig. 4 and Fig. 5. Fig. 4 is a diagram for explaining an example of a first feedback display screen Sc11 in the first display example. Fig. 5 is a diagram for explaining examples of second feedback display screens Sc12 to Sc15.

[0057] A first feedback display screen example 1 and a second feedback display screen example 1 shown in Fig. 4 and Fig. 5, respectively, show examples in which the current inclination of the user's fingers is visualized by depicting a circular level CL. Each of the first feedback display screen Sc11 and the second feedback display screens Sc12 to Sc15 visualizes the current inclination of the user's fingers by generating a circular level CL in which an air bubble AB, which is smaller than the level HL and indicates the current inclination of the user's fingers, is superimposed on a level HL indicating a reference angle.

[0058] 4 and 5, an example in which the predetermined range is -5° to +5° will be described. Also, in each of the second feedback display screens Sc12 to Sc15 shown in Fig. 5, signs are omitted from the pitch reference line, roll reference line, pitch axis, and roll axis in order to make the positional relationship between the level HL and the air bubble AB easier to see. In this disclosure, the pitch angle θp and roll angle θr are defined as + (plus sign) in the direction of the tip of each arrow shown in Fig. 3 and - (negative sign) in the opposite direction to the direction of the tip of the arrow.

[0059] The processor 11 generates a first feedback display screen Sc11 including an image Img11 superimposed with the detected joint point JNT and a circular spirit level CL based on each of the pitch angle θp and roll angle θr, and a message Msg1 "Please stop holding your hand" requesting the user to maintain the current inclination of their fingers.

[0060] Specifically, the processor 11 generates a level HL whose center is the intersection of a pitch reference line LN11A corresponding to the pitch axis PLp and a roll reference line LN12A corresponding to the roll axis PLr. The processor 11 also generates an air bubble AB indicating the inclination of the user's fingers whose center is the intersection of a pitch axis LN11B corresponding to the pitch axis PLp and a roll axis LN12B corresponding to the roll axis PLr.

[0061] The processor 11 calculates the superimposition position of the air bubble AB to be superimposed on the level gauge HL based on each of the pitch angle θp and roll angle θr relative to the reference angle. The processor 11 superimposes the air bubble AB on the level gauge HL based on the calculated superimposition position to generate a circular level gauge CL. The processor 11 superimposes the detected joint points JNT (each joint point) and the circular level gauge CL based on each of the pitch angle θp and roll angle θr on the captured image Img1.

[0062] 4 shows an example of a message requesting the user to maintain the current inclination of the fingers, but may be a message notifying the user that the current inclination of the fingers is suitable for biometric authentication. Moreover, the message Msg1 may not be included in the first feedback display screen Sc11, but may be output as voice.

[0063] As a result, the terminal device P1 in the first display example can notify the user that the inclination of the user's fingers is suitable for biometric authentication and no adjustment is required. Also, the user can intuitively understand that the current inclination of the fingers should be maintained by the message Msg1 included in the first feedback display screen Sc11 or by the fact that the pitch reference line LN11A and the roll reference line LN12A of the level HL substantially coincide with the pitch axis LN11B and the roll axis LN12B of the bubble AB.

[0064] The processor 11 superimposes the detected joint points JNT (each joint point) and a circular level CL based on each of the pitch angle θp and roll angle θr on the captured images Img12, Img13, Img14, and Img15 to generate second feedback display screens Sc12 to Sc15, respectively.

[0065] The second feedback display screen Sc12 shown in FIG. 5 shows the positions of the level HL and the air bubble AB when the calculated pitch angle θp is greater than +5° and the roll angle θr is less than -5°. The second feedback display screen Sc13 shows the positions of the level HL and the air bubble AB when the calculated pitch angle θp and roll angle θr are both greater than +5°. The second feedback display screen Sc14 shows the positions of the level HL and the air bubble AB when the calculated pitch angle θp and roll angle θr are both less than -5°. The second feedback display screen Sc15 shows the positions of the level HL and the air bubble AB when the calculated pitch angle θp is less than -5° and the roll angle θr is greater than +5°.

[0066] Also, although not shown in Figure 5, if it is determined that the inclination of the user's fingers is not within a predetermined range, the processor 11 may generate a second feedback display screen Sc12-Sc15 including a message requesting the user to adjust the inclination of the fingers, such as "Tilt your hand so that the bubble is in the center of the circle."

[0067] As a result, the terminal device P1 in the first display example can support the adjustment of the inclination of the user's fingers by generating and outputting second feedback display screens Sc12-Sc15 in which a circular spirit level CL is superimposed on the user's fingers. Based on the second feedback display screens Sc12-Sc15, the user can intuitively adjust the inclination of the user's fingers so that the position of the center point of the bubble AB relative to the center point of the spirit level HL and the position of the pitch axis or roll axis of the bubble AB relative to the pitch reference line or roll reference line of the spirit level HL match, respectively.

[0068] In addition, in the first feedback display screen Sc11 and the second feedback display screens Sc12 to Sc15, the superimposition process of the skeleton SL and the joint points JNT (each joint point) is not essential and may be omitted.

[0069] <Second display example> Next, the second display example will be described with reference to Fig. 6. Fig. 6 is a diagram for explaining an example of a first feedback display screen Sc22 and an example of a second feedback display screen Sc21 in the second display example.

[0070] Also, although not shown in Figure 6, the processor 11 may generate a second feedback display screen Sc21 including a message requesting the user to adjust the inclination of their fingers, or generate a first feedback display screen Sc22 including a message requesting the user to maintain the current inclination of their fingers, based on the determination result of whether the inclination of the user's fingers is within a specified range.

[0071] In the second display example, spheres HS21 and HS22 with their centers at the intersection of the pitch axis PLp and roll axis PLr are superimposed on captured images Img21 and Img22 of the user's fingers. The second feedback display screen Sc21 in the second display example rotates each of the spheres HS21 based on the angle difference between the reference angle and each of the pitch angle θp and roll angle θr, and superimposes the rotated sphere HS21 on the captured image Img21 to visualize the inclination of the user's fingers.

[0072] The processor 11 detects joint points JNT of the user's fingers from the captured images Img21 and Img22, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.

[0073] The processor 11 generates spheres HS21, HS22 with the intersection of the pitch axis PLp and the roll axis PLr as the center point, and generates curves LN21A, LN22A, which are curves obtained by projecting the pitch axis PLp onto the surfaces of the spheres HS21, HS22 and pass through two intersections between the pitch axis PLp and the spheres, and curves LN21B, LN22B, which are curves along the surfaces of the spheres and pass through two intersections between the roll axis PLr and the spheres. The processor 11 calculates the angle difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angle difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.

[0074] When the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, the processor 11 rotates the sphere HS21 based on the calculated angular difference. The processor 11 superimposes the rotated sphere HS21, the curves LN21A and LN21B corresponding to the rotated sphere HS21, the skeleton SL, and the joint points JNT on the captured image Img21 to generate a second feedback display screen Sc21. The processor 11 outputs the generated second feedback display screen Sc21 to the monitor 14 to display it.

[0075] Thus, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by the second feedback display screen Sc21 that visualizes the angle differences of the pitch angle θp and the roll angle θr with respect to the reference angle, that is, the adjustment direction of the inclination of the user's fingers, by each of the two curves LN21A and LN21B on the second feedback display screen Sc21. Also, the user can intuitively grasp the inclination of the user's fingers by each of the two curves LN21A and LN21B on the second feedback display screen Sc21, and can intuitively grasp in which direction the fingers should be inclined.

[0076] On the other hand, when the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 omits the rotation of the ball HS22. The processor 11 superimposes the ball HS22, the curve LN22A along the pitch axis PLp, the curve LN22B along the roll axis PLr, the skeleton SL, and the joint point JNT on the captured image Img22 to generate a first feedback display screen Sc22. The processor 11 outputs the generated first feedback display screen Sc22 to the monitor 14 to display it.

[0077] The first feedback display screen Sc22 shows the sphere HS22 in a so-called front view. Therefore, each of the curves LN22A and LN22B displayed on the first feedback display screen Sc22 is displayed as a straight line that intersects at right angles with the cross at the center point of the sphere HS22.

[0078] Thus, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by displaying the two curves LN22A and LN22B approximately perpendicular to each other. Also, the user can intuitively understand that the current inclination of the user's fingers should be maintained by viewing the two curves LN22A and LN22B on the first feedback display screen Sc22. Therefore, the terminal device P1 can assist in obtaining a biometric image more suitable for biometric authentication.

[0079] The first feedback display screen Sc22 in the second display example may include a message notifying the user that the current inclination of the user's fingers is suitable for biometric authentication and that no adjustment is required. The processor 11 may change the line type, line thickness or line color of the sphere HS22 displayed on the first feedback display screen Sc22 and the line type, line thickness or line color of the sphere HS21 displayed on the second feedback display screen Sc21.

[0080] <Third display example> Next, a third display example will be described with reference to Fig. 7. Fig. 7 is a diagram for explaining an example of a first feedback display screen Sc32 and an example of a second feedback display screen Sc31 in the third display example.

[0081] Also, although not shown in Figure 7, the processor 11 may generate a second feedback display screen Sc31 including a message requesting the user to adjust the inclination of their fingers, or generate a first feedback display screen Sc32 including a message requesting the user to maintain the current inclination of their fingers, based on the determination result of whether the inclination of the user's fingers is within a specified range.

[0082] In the third display example, spheres HS31 and HS32 having a center point at the intersection of the pitch axis PLp and roll axis PLr are superimposed on captured images Img31 and Img32 of the user's fingers. The second feedback display screen Sc31 in the third display example rotates each of the spheres HS31 based on the angular difference between a reference angle and each of the pitch angle θp and roll angle θr, and superimposes each of vectors VT31A and VT31B corresponding to the angular difference between the rotated sphere HS31 and each of the pitch angle θp and roll angle θr on the captured image Img31, thereby visualizing the adjustment direction of the inclination of the user's fingers.

[0083] The processor 11 detects joint points JNT of the user's fingers from the captured images Img31 and Img32, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.

[0084] The processor 11 generates spheres HS31 and HS32 whose centers are the intersection points of the pitch axis PLp and the roll axis PLr, and pitch axes LN31A and LN32A and roll axes LN31B and LN32B superimposed on the spheres HS31 and HS32. The processor 11 calculates the angle difference between a reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angle difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.

[0085] When the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, the processor 11 generates a vector VT31A based on the calculated angular difference of the pitch angle θp. Specifically, the processor 11 determines the direction of the vector VT31A to the direction in which the user should adjust the inclination of the fingers (i.e., the direction in which the fingers should be inclined) based on the sign (positive or negative) of the angular difference of the calculated pitch angle θp, and determines the length of the vector VT31A based on the magnitude of the calculated angular difference.

[0086] Furthermore, processor 11 generates vector VT31B based on the calculated angle difference of roll angle θr. Specifically, processor 11 determines the direction of vector VT31B to the direction in which the user should adjust the inclination of the fingers (i.e., the fingers should be inclined) based on the sign of the calculated angle difference of roll angle θr, and determines the length of vector VT31B based on the magnitude of the calculated angle difference.

[0087] The processor 11 superimposes the sphere HS31, each of the pitch axis LN31A and the roll axis LN31B, the vectors VT31A and VT31B, the skeleton SL, and each of the joint points JNT on the captured image Img31 to generate a second feedback display screen Sc21. The processor 11 outputs the generated second feedback display screen Sc21 to the monitor 14 to display it.

[0088] As a result, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by the second feedback display screen Sc31 that visualizes the angle difference between the pitch angle θp and the roll angle θr relative to the reference angle, that is, the inclination of the user's fingers, in the direction along the pitch axis PLp (i.e., the horizontal direction) and the direction along the roll axis PLr (the vertical direction) by each of the two vectors VT31A and VT31B. Also, the user can intuitively grasp the inclination of the user's fingers from the direction and length of the vectors VT31A and VT31B displayed on the second feedback display screen Sc31, and can intuitively grasp in which direction the fingers should be inclined.

[0089] On the other hand, when the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 omits the generation of the vector. The processor 11 superimposes the sphere HS32, the pitch axis LN32A and the roll axis LN32B, the skeleton SL and the joint point JNT on the captured image Img32 to generate a first feedback display screen Sc32. The processor 11 outputs the generated first feedback display screen Sc32 to the monitor 14 to display it.

[0090] As a result, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by omitting the generation and display of vectors. Also, the user can intuitively understand that the current inclination of the user's fingers should be maintained by the pitch axis LN32A and the roll axis LN32B superimposed on the sphere HS32 of the first feedback display screen Sc32. Therefore, the terminal device P1 can support the acquisition of a biometric image more suitable for biometric authentication.

[0091] The first feedback display screen Sc32 in the third display example may include a message notifying the user that the current inclination of the user's fingers is suitable for biometric authentication and that no adjustment is required. The processor 11 may change the line type, line thickness, or line color of the sphere HS32 displayed on the first feedback display screen Sc32 and the line type, line thickness, or line color of the sphere HS31 displayed on the second feedback display screen Sc31.

[0092] <Fourth display example> Next, a fourth display example will be described with reference to Fig. 8. Fig. 8 is a diagram for explaining an example of a first feedback display screen Sc43 and examples of second feedback display screens Sc41 and Sc42 in the fourth display example.

[0093] Also, although not shown in Figure 6, the processor 11 may generate a second feedback display screen Sc41, Sc42 including a message requesting the user to adjust the inclination of their fingers, or generate a first feedback display screen Sc43 including a message requesting the user to maintain the current inclination of their fingers, based on the determination result of whether the inclination of the user's fingers is within a specified range.

[0094] In the fourth display example, spheres HS41, HS42, and HS43 having their centers at the intersection of the pitch axis PLp and roll axis PLr are superimposed on captured images Img41, Img42, and Img43 of the user's fingers. Each of the second feedback display screens Sc41 and Sc42 in the fourth display example visualizes the adjustment direction of the inclination of the user's fingers by superimposing, on the captured images Img41 and Img42, vectors corresponding to the angular differences between the reference angle and each of the pitch angle θp and roll angle θr.

[0095] The processor 11 detects joint points JNT of the user's fingers from the captured images Img41 to Img43, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.

[0096] The processor 11 generates a sphere with the intersection of the pitch axis PLp and the roll axis PLr as the center point, and generates pitch axes LN41A, LN42A, and LN43A which are curves obtained by projecting the pitch axis PLp onto the surfaces of the spheres HS41 to HS43, and roll axes LN41B, LN42B, and LN43B which are curves obtained by projecting the roll axis PLr onto the surfaces of the spheres HS41 to HS43. The processor 11 calculates the angle difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angle difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.

[0097] The processor 11 further calculates the angular difference between the calculated pitch angle θp and the pitch axis PLp (i.e., the reference angle), and generates a vector indicating the angular difference of the pitch angle θp based on the sign (positive or negative) and magnitude of the calculated angular difference.

[0098] Here, in the example shown on the second feedback display screen Sc41, the processor 11 determines the direction of the vector VT41 along the direction in which the user should tilt the fingers based on the calculated angular difference between the pitch angle θp and the roll angle θr and the signs of the pitch angle θp and the roll angle θr, and determines the length of the vector VT41 based on the magnitude of the calculated angular difference. Here, the vector VT41 is equal to a vector generated by combining a vector based on the angular difference and sign of the pitch angle θp and a vector based on the angular difference and sign of the roll angle θr.

[0099] The processor 11 superimposes the skeleton SL, the joint points JNT, the sphere HS41, the pitch axis LN41A and the roll axis LN41B, and the vector VT41 on the captured image Img41 to generate a second feedback display screen Sc41, and outputs and displays it on the monitor 14. Note that here, the processor 11 superimposes the starting point position of the vector VT41 on the zenith position of the sphere HS41, which is on the palm side of the user (i.e., the front side of the paper).

[0100] This allows the terminal device P1 to assist in the acquisition of a biometric image suitable for biometric authentication by the second feedback display screen Sc41 that visualizes the adjustment direction of the inclination of the user's fingers by the vector VT41. Also, the user can intuitively grasp the inclination of the user's fingers by the direction and length of the vector VT41 displayed on the second feedback display screen Sc41, and can intuitively grasp in which direction the user should incline the fingers.

[0101] In addition, in the example shown on the second feedback display screen Sc42, the processor 11 determines the direction of the vector VT42A along the direction in which the user should tilt his or her fingers (i.e., the opposite direction to the direction indicated by the sign of the pitch angle θp) based on the angle difference and sign of the calculated pitch angle θp, and determines the length of the vector VT42A based on the magnitude of the angle difference of the calculated pitch angle θp.

[0102] Based on the angle difference and sign of the calculated roll angle θr, processor 11 determines the direction of vector VT42B along the direction in which the user should tilt his or her fingers (i.e., the opposite direction to the direction indicated by the sign of the roll angle θr), and determines the length of vector VT42B based on the magnitude of the angle difference of the calculated roll angle θr.

[0103] The processor 11 superimposes the skeleton SL, the joint points JNT, the sphere HS42, the pitch axis LN42A and the roll axis LN42B, and the vectors VT42A and VT42B on the captured image Img42 to generate a second feedback display screen Sc42, which is output to and displayed on the monitor 14. Note that here, the processor 11 superimposes the starting positions of the vectors VT42A and VT42B on the zenith position of the sphere HS42, which is on the palm side of the user (the front side of the page).

[0104] As a result, the terminal device P1 can assist in obtaining a biometric image suitable for biometric authentication by the second feedback display screen Sc42 that visualizes the adjustment direction of the inclination of the user's fingers by each of the vectors VT42A and VT42B. Also, the user can intuitively grasp the inclination of the user's fingers by each of the directions and lengths of the vectors VT42A and VT42B displayed on the second feedback display screen Sc42, and can intuitively grasp in which direction the fingers should be inclined.

[0105] On the other hand, when the processor 11 determines that the angular differences between the calculated pitch angle θp and roll angle θr are within a predetermined range, the processor 11 omits the generation of vectors. The processor 11 superimposes the skeleton SL, the joint point JNT, the sphere HS43, and the pitch axis LN43A and the roll axis LN43B on the captured image Img43 to generate a first feedback display screen Sc43, and outputs and displays it on the monitor 14.

[0106] As a result, the terminal device P1 can notify the user that the current inclination of the user's fingers is suitable for biometric authentication by omitting the generation and display of vectors. Also, the user can intuitively understand that the current inclination of the user's fingers should be maintained by the pitch axis LN43A and the roll axis LN43B superimposed on the sphere HS43 of the first feedback display screen Sc43. Therefore, the terminal device P1 can support the acquisition of a biometric image more suitable for biometric authentication.

[0107] The first feedback display screen Sc43 in the fourth display example may include a message notifying the user that the current inclination of the user's fingers is suitable for biometric authentication and that no adjustment is required. The processor 11 may change the line type, line thickness or line color of the sphere HS43 displayed on the first feedback display screen Sc43 and the line type, line thickness or line color of the spheres HS41, HS42 displayed on the second feedback display screens Sc41, Sc42.

[0108] Next, a fifth display example and a sixth display example will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining examples of first feedback display screens Sc52 and Sc62 and examples of second feedback display screens Sc51 and Sc61 in the fifth and sixth display examples.

[0109] In addition, although not shown in FIG. 9, if the inclination of the user's fingers is not within a specified range, the processor 11 may generate a second feedback display screen Sc51, Sc61 including a message requesting the user to adjust the inclination of the fingers.

[0110] <Fifth display example> In the fifth display example, squares LN51 and LN52 are superimposed on the user's body parts shown in captured images Img51 and Img52 in which the user's fingers are captured. The second feedback display screen Sc51 in the fifth display example visualizes the adjustment direction of the inclination of the user's fingers by transforming the square LN51 based on the angular differences between the reference angle and each of the pitch angle θp and roll angle θr and superimposing it on the captured image Img51.

[0111] The processor 11 detects joint points JNT of the user's fingers from the captured images Img51 and Img52, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.

[0112] The processor 11 calculates the angle difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angle difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.

[0113] When the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, it deforms the shape of the square based on the angular difference between the calculated pitch angle θp and roll angle θr.

[0114] Specifically, the processor 11 detects an area including at least a part of a body part of the user (a palm in the example shown in FIG. 9) captured in the captured image Img51. The processor 11 generates a square in the detected area to be superimposed on a horizontal plane defined by the pitch axis PLp and the roll axis PLr. The processor 11 forms a horizontal plane defined by the calculated pitch angle θp and roll angle θr. The processor 11 calculates a transformation formula (transformation matrix) for superimposing a square (not shown) superimposed on the horizontal plane defined by the pitch axis PLp and the roll axis PLr on the horizontal plane defined by the pitch angle θp and the roll angle θr. The processor 11 transforms the square (not shown) superimposed on the horizontal plane defined by the pitch axis PLp and the roll axis PLr into a square LN51 that can be superimposed on the horizontal plane defined by the pitch angle θp and the roll angle θr based on the calculated transformation formula (transformation matrix).

[0115] The processor 11 superimposes the square LN51, which is a region including at least a part of the user's body part (the palm in the example shown in FIG. 9) shown in the captured image Img51 and is transformed into a horizontal plane defined by the pitch angle θp and the roll angle θr, on the captured image Img51 to generate a second feedback display screen Sc51. The processor 11 outputs the generated second feedback display screen Sc51 to the monitor 14 to display it.

[0116] This allows the terminal device P1 to generate a second feedback display screen Sc51 that visualizes the inclination of the user's fingers in the shape of a square LN51 superimposed on the user's biometric parts shown in the captured image Img51. The terminal device P1 can assist in the acquisition of a biometric image suitable for biometric authentication by generating the second feedback display screen Sc51. Furthermore, the user can intuitively grasp the inclination of the user's fingers from the shape of the square LN51 displayed on the second feedback display screen Sc51, and can intuitively grasp in which direction the user should incline the fingers to return the deformed square LN51 to the original square.

[0117] On the other hand, when the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within the predetermined range, it omits the shape deformation process of the square LN52.

[0118] The processor 11 superimposes the transformed square LN52 on the captured image Img52, which is an area including at least a part of the user's body part (the palm in the example shown in FIG. 9) shown in the captured image Img52, on a horizontal plane defined by the pitch axis PLp and the roll axis PLr. The processor 11 generates a first feedback display screen Sc52 including the captured image Img52 on which the square LN52 is superimposed, and a message Msg2 "OK!" notifying that the current inclination of the user's fingers is suitable for biometric authentication. The processor 11 outputs the generated first feedback display screen Sc52 to the monitor 14 to display it.

[0119] This allows the terminal device P1 to notify the user that the current inclination of the user's fingers is suitable for biometric authentication by the shape of the square LN52 superimposed on the user's biometric part in the captured image Img52 or the message Msg2. Also, the user can intuitively understand from the shape of the square LN52 or the message Msg2 that the current inclination of the user's fingers should be maintained. Therefore, the terminal device P1 can support the acquisition of a biometric image more suitable for biometric authentication.

[0120] In addition, in the fifth display example, the processor 11 may change the line type, line thickness or line color of the square LN52 displayed on the first feedback display screen Sc52 and the line type, line thickness or line color of the transformed square LN51 displayed on the second feedback display screen Sc51.

[0121] <Sixth display example> The sixth display example visualizes the tilt of the user's fingers by superimposing a sphere BL generated by Augmented Reality (AR) on the user's biological parts shown in captured images Img61 and Img62 in which the user's fingers are captured.

[0122] The processor 11 detects joint points JNT of the user's fingers from the captured images Img61 and Img62, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.

[0123] The processor 11 calculates the angle difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angle difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.

[0124] When the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, the processor 11 tilts the pitch axis PLp and roll axis PLr based on the angular difference between the calculated pitch angle θp and roll angle θr. The processor 11 also calculates the position (coordinates) of the ball BL when the ball BL is rolled from the intersection position of the pitch axis PLp and the roll axis PLr based on the angular difference between the calculated pitch angle θp and roll angle θr. The processor 11 superimposes the pitch axis LN61A and roll axis LN61B after the tilting process and the ball BL virtually rolled on the palm of the user on the captured image Img61 to generate a second feedback display screen Sc61. The processor 11 outputs the generated second feedback display screen Sc61 to the monitor 14 to display it.

[0125] Here, the processor 11 may change the size of the sphere BL superimposed on the captured image Img61 based on the distance between the intersection of the pitch axis PLp and the roll axis PLr and the position of the sphere BL, and the direction in which the sphere BL is located relative to the intersection of the pitch axis PLp and the roll axis PLr. This allows the processor 11 to express a sense of perspective by the size of the sphere BL.

[0126] This allows the terminal device P1 to generate a second feedback display screen Sc61 that visualizes the inclination of the user's fingers based on the rolling direction of the ball BL, the position or size of the ball BL, etc. The terminal device P1 can support the acquisition of a biometric image suitable for biometric authentication by generating the second feedback display screen Sc61. Furthermore, the user can intuitively grasp the inclination of the user's fingers displayed on the second feedback display screen Sc61, and can intuitively grasp in which direction the fingers should be inclined so that the position of the ball BL is the intersection position of the pitch axis PLp and the roll axis PLr.

[0127] On the other hand, when the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 superimposes a sphere BL at the intersection position (reference position) of the pitch axis PLp and the roll axis PLr in the captured image Img62. The processor 11 generates a first feedback display screen Sc62 including the captured image Img62 on which the sphere BL is superimposed and a message Msg3 "OK!" notifying that the current inclination of the user's fingers is suitable for biometric authentication. The processor 11 outputs the generated first feedback display screen Sc62 to the monitor 14 to display it.

[0128] This allows the terminal device P1 to notify the user that the current inclination of the user's fingers is suitable for biometric authentication by the position of the sphere BL in the captured image Img62 or the message Msg3. Also, the user can intuitively understand from the position of the sphere BL or the message Msg3 that the current inclination of the user's fingers should be maintained. Therefore, the terminal device P1 can support the acquisition of a biometric image more suitable for biometric authentication.

[0129] In the sixth display example, the processor 11 may change the color of the ball BL displayed on the first feedback display screen Sc62 and the color of the ball BL displayed on the second feedback display screen Sc61.

[0130] Next, the seventh and eighth display examples will be described with reference to Fig. 10. Fig. 10 is a diagram for explaining examples of first feedback display screens Sc72 and Sc82 and examples of second feedback display screens Sc71 and Sc81 in the seventh and eighth display examples.

[0131] In addition, although not shown in FIG. 10, if the inclination of the user's fingers is not within a specified range, the processor 11 may generate a second feedback display screen Sc71, Sc81 including a message requesting the user to adjust the inclination of the fingers.

[0132] <Seventh display example> In the seventh display example, a ring RG is superimposed on captured images Img71 and Img72 in which the user's fingers are captured. The second feedback display screen Sc71 in the seventh display example visualizes the adjustment direction of the inclination of the user's fingers by superimposing, on the ring RG, a ring bar RG71A based on the angle difference between the reference angle and each of the pitch angle θp and the roll angle θr.

[0133] The processor 11 detects joint points JNT of the user's fingers from the captured images Img71 and Img72, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.

[0134] The processor 11 calculates the angle difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angle difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.

[0135] If the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, it generates a ring RG to be superimposed on the captured image Img71, and generates a ring bar RG71A based on the angular difference between the pitch angle θp and the roll angle θr and superimposes it on the ring RG.

[0136] Specifically, the processor 11 superimposes the ring bar RG71A on the ring RG in a direction (angle) based on the angular difference between the pitch angle θp and the roll angle θr. The processor 11 also determines the circumferential length of the superimposed ring bar RG71A in the circumferential direction based on the magnitude of each of the angular differences between the pitch angle θp and the roll angle θr.

[0137] The processor 11 superimposes the ring RG with the ring bar RG71A superimposed thereon on the captured image Img71 to generate a second feedback display screen Sc71. The processor 11 outputs the generated second feedback display screen Sc71 to the monitor 14 to display it.

[0138] This allows the terminal device P1 to assist in the acquisition of a biometric image suitable for biometric authentication by generating a second feedback display screen Sc71 that visualizes the inclination of the user's fingers with the ring bar RG71A. Also, the user can intuitively grasp the inclination of the user's fingers from the position and circumferential length of the ring bar RG71A displayed on the second feedback display screen Sc71, and can intuitively grasp in which direction the user should incline the fingers to erase the ring bar RG71A.

[0139] On the other hand, when the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, it generates a ring RG to be superimposed on the captured image Img72 and superimposes it on the captured image Img72. The processor 11 generates a first feedback display screen Sc72 including the captured image Img72 on which the ring RG is superimposed and a message Msg4 "OK!" notifying that the current inclination of the user's fingers is suitable for biometric authentication. The processor 11 outputs the generated first feedback display screen Sc72 to the monitor 14 to display it.

[0140] This allows the terminal device P1 to notify the user that the current inclination of the user's fingers is suitable for biometric authentication by the ring RG or message Msg4 shown in the captured image Img72. Also, the user can intuitively understand by the ring RG or message Msg4 that it is sufficient to maintain the current inclination of the user's fingers superimposed on the first feedback display screen Sc72. Therefore, the terminal device P1 can support the acquisition of a biometric image more suitable for biometric authentication.

[0141] In addition, in the seventh display example, the processor 11 may change the line type, line thickness or line color of the ring RG displayed on the first feedback display screen Sc72 and the line type, line thickness or line color of the ring RG displayed on the second feedback display screen Sc71.

[0142] <8th display example> The eighth display example displays pitch sliders SL81A, SL82A and roll sliders SL81B, SL82B along two adjacent sides of captured images Img81, Img82 in which the user's fingers are captured. The second feedback display screen Sc81 in the eighth display example visualizes the adjustment direction of the inclination of the user's fingers by superimposing pitch pointers PT81A, PT82A based on the angle difference between a reference angle and the pitch angle θp and roll pointers PT81B, PT82B based on the angle difference between the reference angle and the roll angle θr on the pitch slider SL81A and the roll slider SL81B, respectively.

[0143] The processor 11 detects joint points JNT of the user's fingers from the captured images Img81 and Img82, and calculates the pitch angle θp and roll angle θr corresponding to the body parts based on the detected joint points JNT.

[0144] The processor 11 calculates the angle difference between the reference angle and each of the calculated pitch angle θp and roll angle θr, and determines whether the angle difference between the calculated pitch angle θp and roll angle θr is within a predetermined range.

[0145] When the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is not within a predetermined range, the processor 11 generates a pitch slider SL81A and a roll slider SL81B based on the calculated angular difference.

[0146] Specifically, the processor 11 generates a pitch slider SL81A including a pitch area RG81A indicating a tilt within a predetermined range suitable for biometric authentication and a pitch pointer PT81A indicating a current pitch angle θp based on the angle difference of the calculated pitch angle θp. The processor 11 generates a roll slider SL81B including a roll area RG81B indicating a tilt within a predetermined range suitable for biometric authentication and a roll pointer PT81B indicating a current roll angle θr based on the angle difference of the calculated roll angle θr.

[0147] The processor 11 generates a second feedback display screen Sc81 including the generated pitch slider SL81A, roll slider SL81B, and captured image Img81. The processor 11 outputs the generated second feedback display screen Sc81 to the monitor 14 to display it.

[0148] This allows the terminal device P1 to generate a second feedback display screen Sc81 that visualizes the inclination of the user's fingers based on the position of the pitch pointer PT81A relative to the pitch area RG81A and the position of the roll pointer PT81B relative to the roll area RG81B. The terminal device P1 can support the acquisition of a biometric image suitable for biometric authentication by generating the second feedback display screen Sc81. Furthermore, the user can intuitively grasp the inclination of the user's fingers based on the position of the pitch pointer PT81A relative to the pitch area RG81A and the position of the roll pointer PT81B relative to the roll area RG81B displayed on the second feedback display screen Sc81, and can intuitively grasp in which direction the user should incline the fingers so that the pitch pointer PT81A is included in the pitch area RG81A and the roll pointer PT81B is included in the roll area RG81B.

[0149] When the processor 11 determines that the angular difference between the calculated pitch angle θp and roll angle θr is within a predetermined range, the processor 11 generates a pitch slider SL82A and a roll slider SL82B based on the calculated angular difference.

[0150] Specifically, the processor 11 generates a pitch slider SL82A including a pitch area RG82A indicating a tilt within a predetermined range suitable for biometric authentication and a pitch pointer PT82A indicating a current pitch angle θp based on the angle difference of the calculated pitch angle θp. The processor 11 generates a roll slider SL82B including a roll area RG82B indicating a tilt within a predetermined range suitable for biometric authentication and a roll pointer PT82B indicating a current roll angle θr based on the angle difference of the calculated roll angle θr.

[0151] The processor 11 generates a message Msg5 "OK!" notifying the user that the current inclination of the user's fingers is suitable for biometric authentication. The processor 11 generates a first feedback display screen Sc82 including the generated message Msg5, a pitch slider SL82A, a roll slider SL82B, and a captured image Img82. The processor 11 outputs the generated first feedback display screen Sc82 to the monitor 14 for display.

[0152] As a result, the terminal device P1 can generate a first feedback display screen Sc82 that visualizes that the inclination of the user's fingers is suitable for biometric authentication based on the position of the pitch pointer PT82A relative to the pitch area RG82A and the position of the roll pointer PT82B relative to the roll area RG82B, or based on the message Msg5. The terminal device P1 can support the acquisition of a biometric image suitable for biometric authentication by generating the first feedback display screen Sc82. Furthermore, the user can intuitively understand that the current inclination of the user's fingers superimposed on the first feedback display screen Sc82 should be maintained based on the pitch pointer PT82A and the roll pointer PT82B, or the message Msg5. Therefore, the terminal device P1 can support the acquisition of a biometric image more suitable for biometric authentication.

[0153] In addition, in the eighth display example, the processor 11 may change the color of the pitch pointer PT82A and the roll pointer PT82B displayed on the first feedback display screen Sc82 and the color of the pitch pointer PT82A and the roll pointer PT82B displayed on the second feedback display screen Sc81.

[0154] (Modification of the first embodiment) The terminal device P1 according to the first embodiment sets the pitch angle θp=0 (zero)° and the roll angle θr=0 (zero)° as reference angles in determining the inclination of a biometric part suitable for biometric authentication, and acquires a captured image determined to have an inclination suitable for biometric authentication based on these reference angles as a biometric image. The biometric authentication system 100 according to the variation of the first embodiment sets the inclination of a biometric part when a pre-registered biometric image or biometric data is acquired as a reference angle, and acquires a captured image determined to have an inclination suitable for biometric authentication based on this reference angle as a biometric image.

[0155] Note that the terminal device P1A in the modified example of the embodiment 1 has a configuration similar to that of the terminal device P1 in the embodiment 1. Therefore, in the following description of the biometric authentication system 100 according to the modified example of the embodiment 1, the same reference numerals are given to the configurations similar to those of the terminal device P1 in the embodiment 1, and the description thereof will be omitted.

[0156] An example of the internal configuration of a biometric authentication system 100 according to a modification of the first embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of the internal configuration of a biometric authentication system 100 according to a modification of the first embodiment.

[0157] The biometric authentication system 100 described below performs user authentication by, for example, face authentication based on a face image of the user and multimodal biometric authentication based on a biometric image of the user's body part (e.g., fingerprint, vein, palm print, etc.), but is not limited thereto. The biometric authentication system 100 may perform two or more user authentication processes, such as multi-factor authentication.

[0158] The biometric authentication system 100 according to the modified example of the first embodiment identifies an individual user in advance through a first optional authentication, and then acquires a biometric image to be used in a second biometric authentication based on the inclination information of the biometric part at the time of enrollment associated with the biometric image or biometric data of the identified user.

[0159] The terminal device P1A, which is an example of a biometric image acquisition support device, captures a user's face image used in the first user authentication and transmits it to the server S1. The terminal device P1A also determines a reference angle for capturing an image of the user's body parts used in the second user authentication based on the face authentication result transmitted from the server S1, and performs capturing (capturing) the user's body parts. The terminal device P1A may extract user features from the face image and transmit the extracted user features to the server S1, or may similarly extract user features from the biometric image and transmit the extracted user features to the server S1.

[0160] The terminal device P1A includes a communication unit 10A, a processor 11A, a memory 12A, a camera 13A, and a monitor 14A. When multi-factor authentication is used, the terminal device P1A may have any configuration that enables the multi-factor authentication.

[0161] The communication unit 10A, which is an example of an acquisition unit, is configured using a communication interface circuit for transmitting and receiving data to and from the server S1 through wired or wireless communication. The communication unit 10A transmits a face image or a biometric image output from the processor 11A to the server S1. The communication unit 10A acquires information on the face authentication result transmitted from the server S1 and outputs it to the processor 11A. Note that the wireless communication referred to here is, for example, communication via a wireless Local Area Network (LAN) such as Wi-Fi (registered trademark).

[0162] The processor 11A, which is an example of a calculation unit, a control unit, and a detection unit, is configured using, for example, a CPU or an FPGA, and performs various processes and controls in cooperation with the memory 12A. Specifically, the processor 11A refers to the programs and data stored in the memory 12A and executes the programs to realize various functions related to imaging of a biological part.

[0163] The memory 12A has, for example, a RAM as a work memory used when executing each process of the processor 11A, and a ROM for storing programs and data that define the operation of the processor 11A. The RAM temporarily stores data or information generated or acquired by the processor 11A. The ROM stores programs that define the operation of the processor 11A.

[0164] The camera 13A is configured to have at least a lens (not shown) and an image sensor (not shown). The image sensor is, for example, a CCD or CMOS solid-state imaging element, and converts an optical image formed on an imaging surface into an electrical signal.

[0165] The monitor 14A is configured using, for example, an LCD or an organic EL. The monitor 14A outputs and displays images captured by the camera 13A, various screens generated by the processor 11A (such as the first feedback display screen and the second feedback display screen in the first to eighth display examples shown in FIGS. 4 to 10, respectively), or screens including face authentication results, biometric authentication results, etc. transmitted from the server S1.

[0166] The server S1 executes a first user authentication based on the face image transmitted from the terminal device P1A. The server S1 generates a face authentication result including tilt information of the identified biometric part of the user as a result of the user authentication, and transmits the result to the terminal device P1A. The server S1 executes a second user authentication based on the biometric image transmitted from the terminal device P1A. The server S1 includes a communication unit 20, a processor 21, a memory 22, a biometric database DB1A, and a face image database DB2. Each of the biometric database DB1A and the face image database DB2 may be configured integrally with the server S1.

[0167] The communication unit 20 is configured using a communication interface circuit for transmitting and receiving data to and from the terminal device P1A by wired communication or wireless communication. The communication unit 20 transmits to the terminal device P1A each of a face authentication result using a face image or a biometric authentication result using a biometric image output from the processor 21. The communication unit 20 acquires the face image or the biometric image transmitted from the terminal device P1A and outputs it to the processor 21.

[0168] The processor 21 is configured using, for example, a CPU or an FPGA, and performs various processes and controls in cooperation with the memory 22. Specifically, the processor 21 refers to the programs and data stored in the memory 22 and executes the programs to realize various functions related to imaging of a biological part.

[0169] The memory 22 has, for example, a RAM as a work memory used when executing each process of the processor 21, and a ROM for storing programs and data that define the operation of the processor 21. The RAM temporarily stores data or information generated or acquired by the processor 21. The ROM has written therein programs that define the operation of the processor 21.

[0170] The biometric database DB1A is a so-called storage, and is configured using a storage medium such as a flash memory, HDD, or SSD. The biometric database DB1A stores (registers) at least one pre-registered biometric image of a person, information on the inclination of a biometric part shown in the biometric image, and user information in association with each other. The biometric database DB1A may also store (register) biometric data extracted from the biometric image in association with user information. Furthermore, when storing (registering) each of a plurality of biometric images, the biometric database DB1A may also store (register) each of the plurality of biometric images in association with information on a biometric part corresponding to each of the biometric images, and user information.

[0171] The face image database DB2 is a so-called storage, and is configured using a storage medium such as a flash memory, HDD, SSD, etc. The face image database DB2 stores (registers) a face image of at least one person registered in advance or a feature amount extracted from a face image, in association with user information.

[0172] Next, an example of an operation procedure of the biometric authentication system 100 according to the modification of the first embodiment will be described with reference to Fig. 12. Fig. 12 is a sequence diagram showing an example of an operation procedure of the biometric authentication system 100 according to the modification of the first embodiment.

[0173] In the sequence diagram shown in FIG. 12, the processes of steps St32 to St33 and steps St39 to St40 indicated by dashed lines are not essential and may be omitted.

[0174] The processor 11A of the terminal device P1A accepts a user operation to start an application capable of capturing an image of a face and a body part (St31). Based on the user operation, the processor 11A generates an operation instruction screen (not shown) that requests the user to perform an image capturing operation of a face image, and outputs it to the monitor 14A (St33). Here, the operation instruction screen includes a message such as "Please face the camera" (St32). The user faces his / her face toward the camera 13A (St34).

[0175] The processor 11A outputs a control command to the camera 13A to capture an image of the user's face. The processor 11A acquires the face image output from the camera 13A (St35). The processor 11A outputs the acquired face image to the communication unit 10A and transmits it to the server S1 (St36). The processor 11A may also transmit the user's biometric data acquired from the face image to the server S1.

[0176] The processor 21 of the server S1 acquires the face image transmitted from the terminal device P1A, and performs face authentication by comparing the acquired face image with at least one face image registered in the face image database DB2 (St37).

[0177] The processor 21 acquires information on the inclination of the biometric part associated with the user information of the user authenticated (identified) by face authentication from the biometric database DB1A (for example, information on the reference angle corresponding to the registered biometric image, the pitch angle θp, the roll angle θr, etc.). The processor 21 associates the information on the inclination of the biometric part with the face authentication result, outputs it to the communication unit 20, and causes it to be transmitted to the terminal device P1A (St38).

[0178] The processor 11A outputs the face authentication result transmitted from the server S1 to the monitor 14A. If the acquired face authentication result indicates that the user authentication was successful, the processor 11A generates an operation instruction screen (not shown) for requesting the user to perform an image capturing operation of a body part, and outputs the operation instruction screen to the monitor 14A (St40). Here, the operation instruction screen includes a message such as "Please point your hand toward the camera" (St39). The user holds his / her hand (body part) over the camera 13A (St41).

[0179] The processor 11A outputs a control command to the camera 13A to capture an image of the user's body part. The processor 11A acquires a captured image Img (see FIG. 3) of the entire hand or a body part of the user output from the camera 13A (St42).

[0180] The processor 11A detects the three-dimensional coordinates of each joint point JNT (see FIG. 3) of the user's fingers based on the acquired captured image Img. The processor 11A calculates the pitch angle θp and roll angle θr (see FIG. 3) of the user's fingers (body parts) shown in the captured image based on the three-dimensional coordinates of each detected joint point JNT. Note that the joints for which the three-dimensional coordinates are calculated here may be at least the joints corresponding to the body parts.

[0181] The processor 11A sets the reference angle based on the information on the inclination of the body part transmitted from the server S1. The processor 11A calculates the angle difference between the calculated pitch angle θp and roll angle θr and the set reference angle (St43).

[0182] The processor 11A determines whether or not the angle difference between each of the calculated pitch angle θp and roll angle θr and the reference angle is within a predetermined range (for example, −5° to +5°, etc.) (St44). Note that the predetermined range here is the range of the pitch angle θp and roll angle θr that indicates the inclination of a biometric part suitable as biometric data used for biometric authentication. The predetermined range may be set to a different range (value) for each biometric part used for biometric authentication.

[0183] When the processor 11A determines in the process of step St44 that the angle difference between each of the pitch angle θp and the roll angle θr and the reference angle is within a predetermined range (for example, −5° to +5°, etc.) (St44, YES), the processor 11A generates first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, and Sc82 (see FIG. 4, FIG. 6 to FIG. 10) that notify the user that the current inclination of the body part is suitable for biometric authentication. The processor 11A outputs the generated first feedback display screens Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, and Sc82 to the monitor 14A for display (St45). Note that the processor 11A may generate an audio signal, instead of the first feedback display screen, that notifies the user that the current inclination of the body part is suitable for biometric authentication, and output the audio signal from a speaker (not shown).

[0184] On the other hand, when the processor 11A determines in the process of step St44 that the angle difference between each of the pitch angle θp and the roll angle θr and the reference angle is not within a predetermined range (for example, −5° to +5°, etc.) (St44, NO), the processor 11A generates second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 (see FIGS. 5 to 10) that support adjustment of the inclination of the current body part based on the angle difference corresponding to each of the pitch angle θp and the roll angle θr. The processor 11A outputs the generated second feedback display screens Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, and Sc81 to the monitor 14A for display (St46).

[0185] The processor 11A determines whether or not the pitch angle θp and roll angle θr, which are the inclinations of the user's body parts, remain within a predetermined range for three seconds or more, based on the captured image output from the camera 13A (St47).

[0186] The duration of "3 seconds" is an example and is not limited thereto. The processor 11A may determine whether each of the pitch angle θp and the roll angle θr is within a predetermined range, that is, whether the tilt of the biometric part is maintained, during the time when the processor 11A notifies the user that the current tilt of the biometric part is suitable for biometric authentication. In addition, the processor 11A may execute the process of determining whether each of the pitch angle θp and the roll angle θr is within the predetermined range at any interval (for example, every second, etc.) in the process of step St47.

[0187] If the processor 11A determines in the processing of step St47 that the angular difference between the pitch angle θp and the roll angle θr remains within a predetermined range for more than 3 seconds (St47, YES), it acquires a biometric image to be used for biometric authentication from among the images captured by the camera 13A (St48).

[0188] The captured image acquired as the biometric image may be an image captured in the processes of steps St42 to St47, or may be an image captured after the process of step St47. The captured image acquired as the biometric image may be an image in which the angle difference between the pitch angle θp and the reference angle and the angle difference between the roll angle θr and the reference angle are smallest among the captured images, an image in which a biometric part such as a fingerprint is most in focus, or the like.

[0189] On the other hand, if the processor 11A determines in the process of step St47 that the state in which the angular differences between the pitch angle θp and the roll angle θr are within the predetermined range does not continue for three seconds or more (St47: NO), the processor 11A returns to the process of step St41.

[0190] The processor 11A outputs the acquired biometric image to the communication unit 10A, and transmits it to the server S1 (St49). Note that the processor 11A may transmit biometric data extracted from the biometric image instead of the biometric image.

[0191] The processor 21 executes biometric authentication (user authentication) using the biometric image transmitted from the terminal device P1A (St50). The processor 21 executes biometric authentication by matching the acquired biometric image with at least one biometric image registered in the biometric database DB1A, generates a biometric authentication result, and transmits it to the terminal device P1A. The processor 21 may execute biometric authentication using the biometric data transmitted from the terminal device P1A and the biometric data registered in the biometric database DB1A.

[0192] As described above, even when a biometric image that is not acquired in a state where the pitch angle=0° and the roll angle=0° are not set as the reference angle, that is, when a biometric image that is not acquired in a state substantially parallel to the light receiving surface of the image sensor (not shown) of the camera 13 is registered in advance, the biometric authentication system 100 according to the modification of the first embodiment can support acquisition of a biometric image suitable for biometric authentication with the registered biometric image by setting the reference angle used for acquiring the biometric image to the reference angle corresponding to the registered biometric image. This enables the biometric authentication system 100 to more effectively suppress a decrease in the accuracy of biometric authentication.

[0193] As described above, the terminal device P1, P1A (an example of a biometric image acquisition support device, a computer) according to embodiment 1 and a modified example of embodiment 1 includes a communication unit 10, 10A (an example of an acquisition unit) that acquires an image Img in which a biometric part (e.g., a fingertip, a finger, a palm, etc.) of a user (an example of a subject) is imaged, a processor 11, 11A (an example of a calculation unit) that calculates the inclination of the biometric part relative to a reference angle based on the acquired image Img, and a processor 11, 11A (an example of a control unit) that generates a second feedback display screen (an example of a posture guidance screen) including information regarding the inclination of the biometric part and the captured image Img when it is determined that the inclination of the biometric part is not within a predetermined range relative to the reference angle, and outputs the second feedback display screen to the monitor 14, 14A.

[0194] As a result, when the terminal devices P1, P1A according to embodiment 1 and the modified example of embodiment 1 determine that the inclination (angle difference) of a biometric part relative to a reference angle for acquiring a biometric image suitable for biometric authentication in acquiring a biometric image of a biometric part used for biometric authentication is not within a predetermined range usable for biometric authentication, they can notify the user of the inclination of the biometric part and provide assistance for acquiring a biometric image suitable for biometric authentication by generating a second feedback display screen that associates information regarding the calculated inclination of the biometric part with the captured image Img in which the biometric part is captured.

[0195] Furthermore, the information on the inclination of the body part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment includes information indicating the inclination direction of the body part. As a result, the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment can visualize and notify the user of the inclination direction of the body part, and can provide support for obtaining a biometric image suitable for biometric authentication.

[0196] Furthermore, the information on the inclination of the body part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment includes information on the inclination direction and the inclination magnitude of the body part. As a result, the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment can visualize and notify the user of the inclination direction and the inclination angle of the body part, and can provide support for obtaining a biometric image suitable for biometric authentication.

[0197] Furthermore, the information regarding the inclination of the biometric part in the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment includes information indicating the direction in which the user should incline the biometric part. As a result, the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment can visualize and notify the user of the direction in which the biometric part should be inclined, thereby providing support for obtaining a biometric image suitable for biometric authentication.

[0198] Furthermore, the information regarding the inclination of the biological part in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment includes information indicating the direction and the amount by which the user should incline the biological part. As a result, the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment can visualize and notify the user of the direction and the amount of the angle by which the biological part should be inclined, and can provide support for obtaining a biometric image suitable for biometric authentication.

[0199] Furthermore, the processors 11, 11A in the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment calculate the inclination of the biological part relative to the light receiving surface of the image sensor of the camera 13 where the captured image Img was captured. This allows the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment to support the acquisition of a biological image in which the angle difference between the biological part used for biometric authentication and the light receiving surface of the image sensor is within a predetermined range.

[0200] Furthermore, the communication unit 10A in the terminal device P1A according to the modification of the first embodiment acquires information regarding the inclination of a biometric part appearing in another captured image of the user (for example, a captured image of a user registered in the face image database DB2). The processors 11, 11A set the inclination of the biometric part appearing in the other captured image as a reference angle, and calculate the inclination of the biometric part relative to the reference angle. As a result, when a biometric part is captured in an inclined state and there is an already registered biometric image, the terminal device P1A according to the modification of the first embodiment can support acquisition of a biometric image that can improve the accuracy of biometric authentication with the biometric part appearing in the registered biometric image by setting the inclination of the biometric part appearing in the registered biometric image as a reference angle.

[0201] Moreover, the predetermined range in the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment is -5° or more and +5° or less with respect to the reference angle. As a result, the terminal devices P1 and P1A according to the first embodiment and the modified example of the first embodiment capture an image with the light receiving surface of the image sensor and the biological part substantially parallel to each other, which can assist in obtaining a biological image more suitable for biometric authentication.

[0202] The terminal device P1, P1A according to the first embodiment and the modified example of the first embodiment further includes a processor 11, 11A (an example of a detector) capable of detecting a plurality of joint points JNT of a body part of a user from a captured image Img acquired in the terminal device P1, P1A according to the first embodiment and the modified example of the first embodiment. The processor 11, 11A calculates the inclination of the body part with respect to a reference angle based on the positions of the detected plurality of joint points JNT. This allows the terminal device P1, P1A according to the first embodiment and the modified example of the first embodiment to calculate the inclination of the body part with higher accuracy.

[0203] Furthermore, the inclination of the body part in the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment is calculated based on the pitch axis PLp (an example of a first direction) and the roll axis PLr (an example of a second direction) that is substantially perpendicular to the pitch axis PLp. The processors 11, 11A calculate the inclination of the body part based on the positions of any two joint points along the pitch axis PLp among the multiple joint points and the positions of any two joint points along the roll axis PLr among the multiple joint points. This allows the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment to calculate the inclination of the body part as the way the body part appears in the captured image.

[0204] Furthermore, when the biological part is one finger (e.g., a fingerprint), the processors 11 and 11A in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment calculate the inclination of the biological part based on the positions of two or more joint points of the one finger and the positions of the joint points of the fingertips of the two fingers. This allows the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment to determine whether the inclination of any finger that is the subject of biometric authentication is an inclination that allows acquisition of a biometric image suitable for biometric authentication.

[0205] Furthermore, when the biological part is one of the fingers (for example, veins), the processors 11 and 11A in the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment calculate the inclination of the biological part based on the positions of two or more joint points of the one finger and the positions of the second joints of the two fingers. This allows the terminal devices P1 and P1A according to the first embodiment and the modification of the first embodiment to determine whether the inclination of any of the fingers to be subjected to biometric authentication is an inclination that allows acquisition of a biometric image suitable for biometric authentication.

[0206] Furthermore, when the biometric part is a palm (e.g., a palm print), the processors 11, 11A in the terminal devices P1, P1A according to the first embodiment and the modified version of the first embodiment calculate the inclination of the biometric part based on the positions of the wrist joint points, the positions of the joint points of the index finger, the middle finger, or the ring finger, the position of the joint point of the base of the little finger, and the position of the joint point of the base of the index finger or the thumb. This allows the terminal devices P1, P1A according to the first embodiment and the modified version of the first embodiment to determine whether the inclination of the palm, which is the subject of biometric authentication, is an inclination that allows acquisition of a biometric image suitable for biometric authentication.

[0207] Moreover, the processors 11, 11A in the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment calculate the distance between two adjacent joint points among a plurality of joint points of a finger among the biological parts for each finger. The processors 11, 11A determine whether the finger is bent based on the calculated distance between the two joint points, and if it is determined that the finger is bent, generate a second feedback display screen including instruction information for instructing the user to open the finger, information regarding the inclination of the biological part, and the captured image Img, and output the second feedback display screen to the monitors 14, 14A. As a result, the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment can detect not only the inclination of the biological part but also the bending of the finger of the biological part, and can support the acquisition of a biological image more suitable for biometric authentication.

[0208] Furthermore, when the processors 11, 11A in the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment determine that the inclination of the body part is within a predetermined range with respect to the reference angle, they generate a first feedback display screen (an example of a notification screen) notifying the user that the inclination of the body part is within the predetermined range, and output it to the monitors 14, 14A. This allows the terminal devices P1, P1A according to the first embodiment and the modified example of the first embodiment to notify the user that the inclination of the body part is suitable for biometric authentication.

[0209] Moreover, the processors 11, 11A in the terminal devices P1, P1A according to the first embodiment and the modification of the first embodiment acquire captured images Img in which the inclination of the body part is within a predetermined range, and output and store them in the fingerprint database DB1 (an example of a storage medium). This allows the terminal devices P1, P1A according to the first embodiment and the modification of the first embodiment to acquire only biometric images suitable for biometric authentication, and to register them as biometric images to be used for biometric authentication.

[0210] Although various embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also belong to the technical scope of the present disclosure. In addition, the components in the various embodiments described above may be arbitrarily combined within the scope of the invention. [Industrial Applicability]

[0211] INDUSTRIAL APPLICABILITY The present disclosure is useful as providing a biometric image acquisition assistance device and a biometric image acquisition assistance method that assist in the acquisition of biometric data more suitable for biometric authentication in a non-contact state. [Explanation of symbols]

[0212] 10, 10A, 20 Communication section 11,11A,21 Processor 12,12A,22 Memory 13,13A Camera 14,14A Monitor 15 Control section 100 Biometric authentication system DB1 Fingerprint Database DB1A Biological Database DB2 Face Image Database P1, P1A terminal equipment S1 Server Sc11, Sc22, Sc32, Sc43, Sc52, Sc62, Sc72, Sc82 1st feedback display screen Sc12, Sc13, Sc14, Sc15, Sc21, Sc31, Sc41, Sc42, Sc51, Sc61, Sc71, Sc81 Second feedback display screen

Claims

1. an acquisition unit for acquiring an image of a biological part of a subject; a calculation unit that calculates an inclination of the body part with respect to a reference angle based on the acquired captured image; and a control unit that generates a posture guidance screen including information regarding the inclination of the body part and the captured image when it is determined that the inclination of the body part is not within a predetermined range with respect to the reference angle, and outputs the generated posture guidance screen to a monitor. Biometric image acquisition support device.

2. The information regarding the inclination of the body part includes information indicating a direction of inclination of the body part. The biometric image acquisition support device according to claim 1 .

3. The information regarding the inclination of the body part includes information indicating a direction and a magnitude of the inclination of the body part. The biometric image acquisition support device according to claim 2 .

4. The information regarding the inclination of the body part includes information indicating a direction in which the subject is to incline the body part. The biometric image acquisition support device according to claim 1 .

5. The information regarding the inclination of the body part includes information indicating a direction and a magnitude in which the subject is to incline the body part.

5. The biometric image acquisition support device according to claim 4.

6. The calculation unit calculates an inclination of the biological part with respect to a light receiving surface of an image sensor of a camera that captured the captured image. The biometric image acquisition support device according to claim 1 .

7. The acquisition unit acquires information regarding an inclination of a body part appearing in another captured image of the subject, the calculation unit sets an inclination of the body part shown in the other captured image as a reference angle, and calculates an inclination of the body part with respect to the reference angle; The biometric image acquisition support device according to claim 1 .

8. The predetermined range is −5° or more and +5° or less with respect to the reference angle. The biometric image acquisition support device according to claim 1 .

9. A detection unit capable of detecting a plurality of joint points of a body part of the subject from the acquired captured image, the calculation unit calculates an inclination of the body part with respect to the reference angle based on positions of a plurality of detected joint points; The biometric image acquisition support device according to claim 1 .

10. The inclination of the body part is calculated based on a first direction and a second direction substantially perpendicular to the first direction; the calculation unit calculates a tilt of the body part based on positions of any two joint points among the plurality of joint points along the first direction and positions of any two joint points among the plurality of joint points along the second direction; The biometric image acquisition support device according to claim 9.

11. When the body part is any one finger, the calculation unit calculates a tilt of the body part based on positions of two or more joint points of the one finger and positions of joint points of fingertips of any two fingers. The biometric image acquisition support device according to claim 9.

12. When the body part is any one finger, the calculation unit calculates a tilt of the body part based on positions of two or more joint points of the one finger and positions of second joints of any two fingers. The biometric image acquisition support device according to claim 9.

13. When the body part is a palm, the calculation unit calculates the inclination of the body part based on a position of a joint point of the wrist, a position of a joint point of any one of the index finger, the middle finger, or the ring finger, a position of a joint point of a base of the little finger, and a position of a joint point of either the base of the index finger or the thumb. The biometric image acquisition support device according to claim 9.

14. the calculation unit calculates a distance between two adjacent joint points of a plurality of joint points of a finger of the biological part for each finger; The control unit determines whether the finger is bent based on the calculated distance between the two joint points, and when it is determined that the finger is bent, generates the posture guidance screen including instruction information to open the fingers, information regarding the inclination of the body part, and the captured image, and outputs the posture guidance screen to the monitor. The biometric image acquisition support device according to claim 1 .

15. When the control unit determines that the inclination of the body part is within the predetermined range with respect to the reference angle, the control unit generates a notification screen notifying that the inclination of the body part is within the predetermined range, and outputs the notification screen to the monitor. The biometric image acquisition support device according to claim 1 .

16. The control unit acquires a captured image in which the inclination of the body part is within the predetermined range, and outputs and stores the captured image in a storage medium. The biometric image acquisition support device according to claim 1 .

17. A biometric image acquisition assistance method implemented by a computer, comprising: Acquire a captured image of a biological part of the subject; Calculating an inclination of the body part with respect to a reference angle based on the acquired captured image; if it is determined that the inclination of the body part is not within a predetermined range with respect to the reference angle, a posture guidance screen including information regarding the inclination of the body part and the captured image is generated and output to a monitor. A method for assisting in biometric image acquisition.

Images