Contactless input device comprising biometric authentication system

Abstract

[Problem] To provide a contactless input device that can only be accessed by a properly registered user at all times.　The contactless input device, on which an input operation can be performed by movement of a hand in the air, comprises: an imaging device that captures a hand of a user; a detection device that detects motion of the hand of the user on the basis of a video captured by the imaging device, and detects an input operation corresponding to the detected motion of the hand; and an authentication device that performs an authentication process using biometric information of the hand of the user on the basis of the video captured by the imaging device. In addition, simultaneously with the detection of the input operation by the detection device, the authentication process by the authentication device is continuously and repeatedly performed. If the authentication process by the authentication device fails, the repetition of the authentication process is continued while invalidating the input operation, and thereafter the input operation is considered valid once the authentication process succeeds.

Description

[Technical Field]

[0001] The present invention relates to a non-contact input device that allows input operations without contact, and more particularly to a non-contact input device equipped with a biometric authentication system. [Background technology]

[0002] The current COVID-19 pandemic has reminded us of how vulnerable our 21st century society is to infectious diseases, a time of increasing population density and high-speed mass transportation. In such circumstances, everything around us seems like a potential source of infection, making life extremely difficult. In fact, as the infection spreads, we need to be suspicious of everything.

[0003] It is important to take measures such as wearing a mask, maintaining a sufficient distance from others, and avoiding crowded places, but in order to lead a necessary social life, it is unavoidable to minimize contact with infection routes. Minimizing this risk as much as possible is essential to preventing infection.

[0004] For example, at bank ATMs, supermarket self-checkouts, corporate reception areas, and hotel check-ins, an unspecified number of users operate the same touch panel. Such touch panels can be potential infection routes. To avoid this risk, it is necessary to disinfect the touch panel every time a user changes, but such a procedure is quite time-consuming and difficult to carry out adequately.

[0005] In recent years, a technology that uses special optical elements to project mid-air images without contact has been gaining attention. This technology is called aerial display. Using this technology, a completely contactless interface can be realized with the same ease of use as a conventional touch panel.

[0006] For example, in Patent Document 1, because of hygiene concerns about the surgeon touching a pointing device such as a computer mouse during surgery, the surgeon operates a mouse on a non-contact remote pointer control device displayed using aerial imaging technology. Similar technology is also effective outside of medical settings, such as in restaurant kitchens, where hygiene considerations are required.

[0007] An example of a special optical element used in such aerial imaging technology is the optical imaging device described in Patent Document 2.

[0008] The contactless interface used in an aerial display is usually implemented using a motion capture system such as that described in Patent Document 3. This motion capture system detects the movement of fingers in the air, but does not necessarily need to be combined with an aerial display. Input can also be performed by placing a motion capture system in front of a general display device such as an LCD display, and moving fingers in the air while relying on the display image. For example, a contactless interface can be implemented by displaying buttons on the display and having the motion capture system detect clicks or touch actions (gestures) made at a short distance away without directly touching the display screen. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 2018-147054 [Patent Document 2] International Publication No. 2009 / 131128 [Patent Document 3] Special Publication No. 2016-501426 Summary of the Invention [Problem to be solved by the invention]

[0010] Although such contactless interface technology is useful, it can be inconvenient if anyone can access it freely. For example, access to personal information must be limited to certain users. To achieve this, it is possible to equip the device with an authentication device, but even if the operation itself is contactless, it is not enough to prevent infection unless authentication can be done contactless.

[0011] In light of this situation, the present applicant proposed an aerial image display device equipped with a biometric authentication system that enables contactless authentication (Patent Application No. 2022-183319). However, the contactless interface poses a problem: after authentication, anyone can easily access the device without worrying about infection. In other words, once contactless authentication is performed, operation remains possible even if a different person takes over. For example, with technology such as that described in Patent Document 1, after a user (surgeon) authenticates and operates the system using a contactless remote pointer during surgery, another doctor or nurse may inadvertently operate the same system. This can result in the user (surgeon) being unable to operate the system as smoothly as intended.

[0012] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an authentication system for a contactless interface that can always be accessed only by authorized registered users. [Means for solving the problem]

[0013] In order to solve the above-mentioned problems, a non-contact input device according to one aspect of the present invention is a non-contact input device that allows input operations to be performed by hand movements in the air, and includes a photographing device that photographs a user's hand, a detection device that detects the user's hand movements based on the image photographed by the photographing device and detects an input operation corresponding to the detected hand movements, and a recognition device that uses biometric information of the user's hand based on the image photographed by the photographing device. Proof equipped with an authentication device that performs If the authentication is successful, the input operation is validated, and even after the authentication is successful and the input operation is validated, the authentication is continuously repeated, and the user can continuously perform input operations only while the authentication is repeatedly successful. It is characterized by:

[0014] In one preferred embodiment, If the authentication fails after the input operation has been validated due to the success of the authentication, the input operation is invalidated, but the authentication is continued and repeated thereafter.It is characterized by: [Effects of the Invention]

[0015] The aerial image display device of the present invention prevents a situation in which another person inadvertently performs an input operation while an authenticated user is performing an input operation, and allows the user to carry out a series of operations as desired. [Brief explanation of the drawings]

[0016] [Figure 1] FIG. 1 is a perspective view showing an aerial image display device 1 as a non-contact input device according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a cross-sectional view showing the internal configuration of the aerial image display device 1 shown in FIG. [Figure 3] FIG. 3 is a flowchart showing the process of input operations to the aerial image display device 1 shown in FIG. [Figure 4] FIG. 4 is a flowchart illustrating the operation related to vein authentication of aerial image display device 1 shown in FIG. [Figure 5] FIG. 5 is a perspective view showing a computer system equipped with a non-contact input device according to a second embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart illustrating operations related to vein authentication executed by the computer system shown in FIG. [Figure 7] FIG. 7 is a flowchart showing another example of the process of input operations to the aerial image display device 1 shown in FIG. [Figure 8] FIG. 8 is a flowchart illustrating another example of the operation related to vein authentication of the aerial image display device 1 shown in FIG. DETAILED DESCRIPTION OF THE INVENTION [Example]

[0017] An aerial image display device equipped with a biometric authentication system will now be described as an embodiment of a non-contact input device according to the present invention with reference to the accompanying drawings. Fig. 1 is a perspective view showing an aerial image display device 1 as a non-contact input device according to embodiment 1 of the present invention. Fig. 2 is a cross-sectional view showing the internal configuration of the aerial image display device 1 as a non-contact input device according to embodiment 1 of the present invention. In this embodiment 1, palm vein authentication is performed as biometric authentication.

[0018] As shown in these figures, the aerial image display device 1 comprises a box-shaped housing 10, an LCD display 20 installed inside the housing at an angle of approximately 45 degrees to the horizontal, an optical plate 30 placed horizontally on the top surface of the housing 10, a pair of left and right infrared LEDs 32 and a pair of left and right infrared cameras 34 installed in front of the optical plate 30, a speaker 39, and a control device 40 that processes the input and output signals of each of these elements.

[0019] The control device 40 is essentially a small computer and is composed of a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), a storage device for storing various programs and data, an input / output interface, a timer, etc. The input / output interface may be, for example, a USB port or a wireless LAN such as WIFI.

[0020] Optical plate 30 has incident surface 31 facing downward, facing display surface 24 of LCD display 20, and exit surface 33 facing directly upward. The image on display surface 24 of LCD display 20 is then re-focused via optical plate 30 at a position the same distance on the opposite side as the original, forming aerial image 4. By displaying non-contact interface I on this aerial image 4, the aerial image display device functions as a non-contact input device.

[0021] For example, the optical imaging element described in Japanese Patent Laid-Open No. 2011-175297 can be used as this optical plate 30. This optical imaging element is realized by arranging a large number of mutually orthogonal planar light reflecting portions at a fixed pitch. Alternatively, a structure such as a two-sided corner reflector, in which reflective surfaces are formed on the side surfaces of a square hole, as described in Japanese Patent No. 4900618, can also be used.

[0022] Furthermore, a pair of infrared cameras 34 continuously capture images of the user's hands, illuminated by a pair of infrared LEDs 32, from the left and right near the aerial image 4, thereby obtaining time-series data for a pair of images. By performing image analysis on this pair of images, the control device 40 can identify the user's hand movements in three dimensions. If the user's hand movements are a predetermined action, it is determined that an input operation has been performed. Therefore, the control device 40 functions as a detection device that detects the user's hand movements as input operation information based on the images captured by the imaging device.

[0023] For example, if controls such as buttons, checkboxes, or drop-down menus are displayed on the aerial image 4 and the user makes a gesture to operate one of them with their finger, the detection device comprising the control device 40 will detect the action via the imaging device comprising the infrared LED 32 and infrared camera 34, and various processes will be performed accordingly, such as changing the display of the aerial image 4. Therefore, the aerial image 4 will function as a non-contact interface I.

[0024] The control device 40 also performs vein authentication using images of the fingers captured by an image capture device consisting of the infrared LED 32 and the infrared camera 34. Specifically, the control device 40 detects the vein patterns of the user's palm and fingers. That is, near-infrared rays are irradiated onto the palm and fingers from the infrared LED 32, and when the near-infrared rays penetrate the inside of the palm, the reduced hemoglobin in the blood absorbs the near-infrared rays.

[0025] As a result, the veins in the palm of the hand become shadows, which are captured by the infrared camera 34, and the vein pattern is acquired as an image signal. The acquired image signal is processed by the control device 40, where general vein authentication processing such as image registration and matching is performed. Therefore, the control device 40, together with the infrared LED 32 and the infrared camera 34, also functions as a biometric authentication device.

[0026] Conventionally, to perform vein authentication, the user holds their palm over an infrared LED 32. Once authentication is complete, there is no need for further authentication, and the user can operate the system freely.

[0027] However, in the present invention, in order to ensure that only the authenticated person can operate the device, vein authentication is performed repeatedly, even while the user is moving their hand to perform input operations. For example, vein authentication is performed continuously at one-second or two-second intervals. If authentication fails, the input operation is temporarily invalidated.

[0028] Therefore, the registration of a user's vein pattern used for vein authentication matching is acquired from a wide range of parts, such as each finger or palm, but as long as the veins are photographed, it should be possible to use it regardless of the posture, such as the orientation or tilt of the hand. For this purpose, the vein pattern is photographed and registered while the user is freely performing input operations. Furthermore, when matching patterns for authentication, the registered pattern is subjected to the necessary coordinate transformation according to the orientation and tilt of the hand, making it possible to match even while the fingers are moving freely.

[0029] As mentioned above, the veins used for authentication are acquired from a wide range of areas, such as each finger or palm, so there are few situations in which the image required for authentication cannot be obtained, even when the user is moving their hand to perform an operation. If authentication is still not successful after a certain period of time, the system temporarily disables the user's input operation and waits until the necessary image is obtained. If the authentication process continues and is successful, the user's input operation is validated. Therefore, if the user feels unable to perform input operations, they can immediately resume input operations by consciously moving their hand to make authentication easier.

[0030] However, if another person accesses the contactless interface I while the user is not operating the system, authentication will fail and the other person will not be able to operate the system. Therefore, the user can smoothly carry out operations as they wish without being disturbed by anyone.

[0031] Next, a specific control method for the aerial image display device 1 as a non-contact input device according to an embodiment of the present invention will be described. Fig. 3 is a flowchart showing the process of input operations to the aerial image display device 1 shown in Fig. 1. Fig. 4 is a flowchart explaining the operations related to vein authentication of the aerial image display device 1. The process of input operations to the aerial image display device 1 and the process of performing operations related to vein authentication are executed simultaneously in parallel.

[0032] As shown in FIG. 3, when the system is started, continuous shooting begins with the pair of infrared cameras 34 (step S1). For example, the frame rate is set to 120. In other words, 120 images are taken per second. One frame consists of a pair of infrared images, one on each side. The series of captured images is stored in a buffer for a certain number of seconds (for example, several seconds) so that they can be used for analyzing the user's finger movements and for authentication processing.

[0033] Next, it is determined whether flag F indicating the authentication result is 1 (step S2). That is, it is determined whether authentication is successful or not. If authentication is not successful in step S2 (NO in step S2), the same process is repeated until authentication is successful. During this time, the user's finger movement analysis is not performed, and no processing based on the analysis result is performed. Therefore, the user's input operation is invalid.

[0034] The flag F used here is a shared variable used for inter-process communication between a process that performs operations related to the contactless interface I and a process that performs operations related to vein authentication, and is continuously updated by the process that performs operations related to vein authentication, as described below.

[0035] If authentication is successful in step S2, the series of images captured by infrared camera 34 are image processed to analyze the user's finger movements (step S3). Then, if there is input operation information corresponding to the movements, necessary processing corresponding to the input operation, such as changing the display of aerial image 4, is performed accordingly (step S4). Once the processing corresponding to the input operation is complete, the system returns to step S2 where the authentication result is confirmed, and the same processing is repeated.

[0036] 3, operations related to vein authentication shown in Fig. 4 are also performed in parallel. First, a series of images of the user's fingers are captured by the infrared camera 34 (step S11).

[0037] Next, the acquired series of images are analyzed to obtain information on the vein pattern of the user's fingers and perform authentication processing (step S12). If authentication fails (NO in step S13), a flag F indicating the authentication result is set to 0 (step S14), and a prompt urging the user to perform authentication is displayed on the aerial image 4 (step S15).

[0038] Then, the device sleeps for a certain period of time (step S16), and returns to step S1 to repeat the authentication process. This certain period of time may be, for example, 0.1 seconds. The reason for providing such a sleep period is to wait for the user's finger movement.

[0039] If authentication is successful in step S13, it is determined whether flag F is set to 1 (step S17). If flag F is set to 1 (YES in step S17), the system sleeps for a fixed period of time (step S18) and returns to step S1 to repeat the authentication process. This fixed period of time may be, for example, one second. The reason for providing such a sleep period is that, unlike analysis of the user's finger movements, personal identification after successful authentication does not need to be repeated in such detail. This sleep period can be said to be the valid time for successful authentication.

[0040] If flag F is 0 in step S17, the prompt is cleared if it is displayed (step S19), and flag F is set to 1 (step S20). Then, the system goes into sleep mode for a certain period of time (step S18), and returns to step S1 to repeat the authentication process.

[0041] By implementing this type of control, users of a system using a non-contact input device can use the system as their own personal system through authentication processing, preventing other people from accidentally using the system in a way that goes against the user's intentions, and enabling smooth operation as desired. [Example]

[0042] Next, a second embodiment of the contactless input device according to the present invention will be described. This contactless input device is not the aerial image display device used in the first embodiment, but rather has a motion sensor installed in front of a normal physical display, and detects gestures and the like made above the motion sensor (in front of the display) as input operations. Then, biometric authentication processing is performed using images of the user's fingers acquired by the motion sensor.

[0043] 5 is a perspective view showing a second embodiment of a non-contact input device according to the present invention. The non-contact input device 2 here comprises a normal liquid crystal display 21, a computer 23, and a motion sensor 25. The computer 23 is connected to the liquid crystal display 21 and the motion sensor 25 to send and receive signals.

[0044] Similar to the imaging device of Example 1, the motion sensor 25 includes a pair of left and right infrared cameras 26 and a pair of left and right infrared LEDs 27. The pair of infrared cameras 26 continuously capture images of the user's hands illuminated by the infrared LEDs 27, thereby obtaining time-series data of a pair of images. By performing image analysis of this pair of images, the movement of the user's hands can be identified in three dimensions.

[0045] The user's hand movements identified by the motion sensor 25 are input into an application running on the computer 23 and used as input operation information according to the user's presumed intention. The simplest example of such input operation information is the operation of moving a cursor by moving the tip of the index finger.

[0046] Other examples of input operation information include gestures defined as follows: "circle" as the action of drawing a circle with a finger, "key tap" as the action of tapping downward with a finger (as if pressing a key), "screen tap" as the action of tapping "forward" with a finger (as if pressing the screen), and "swipe" as the action of drawing a straight line with an outstretched hand.

[0047] Time-series data of the video showing the user's hand movements is acquired by the motion sensor 25, and at least a part of the analysis processing of this time-series data can be performed by the computer 23. Therefore, the motion sensor 25 and the computer 23 function as a detection device that detects the user's hand movements as input operation information.

[0048] Here, as in the first embodiment, vein authentication is performed simultaneously with contactless input operation by the motion sensor 25. That is, vein authentication is performed by the computer 23 using video data showing the user's hand movements acquired by a pair of infrared cameras 26. Therefore, the motion sensor 25 and the computer 23 also function as a biometric authentication device.

[0049] The process of non-contact input operation using the motion sensor 25 is similar to the process of the embodiment shown in Fig. 3. That is, referring again to Fig. 3, first, continuous shooting is started by the pair of infrared cameras 26 (step S1). For example, the frame rate is set to 120. The series of captured images are stored in a buffer for a certain number of seconds (for example, about several seconds) so that they can be used for analysis of the user's finger movements and authentication processing.

[0050] Next, it is determined whether flag F indicating the authentication result is 1 (step S2). In other words, it is determined whether authentication is successful or not. If authentication is not successful in step S2 (NO in step S2), the same process is repeated until authentication is successful. During this time, analysis of the user's finger movements is not performed, and input operations are invalid. During this time, analysis of the user's finger movements is not performed, and no processing based on the analysis results is performed. Therefore, the user's input operations are invalid.

[0051] If authentication is successful in step S2, the system processes the series of images captured by infrared camera 26 and analyzes the user's finger movements (step S3). If there is input operation information corresponding to the movements, the system performs the necessary processing corresponding to the input operation, such as changing the display on liquid crystal display 21 (step S4). Once the processing corresponding to the input operation is complete, the system returns to step S2 where the authentication result is confirmed, and the same processing is repeated.

[0052] 3, operations related to vein authentication shown in Fig. 6 are also performed in parallel. First, a series of images of the user's fingers are captured by the infrared camera 26 (step S21).

[0053] Next, the acquired series of images are analyzed to obtain information on the vein pattern of the user's fingers and perform authentication processing (step S22). If authentication is successful (YES in step S23), a flag F indicating the authentication result is set to 1 (step S24). Then, the system sleeps for a fixed period of time (step S25), and returns to step S21 to repeat the authentication processing. This fixed period of time may be, for example, one second. The reason for providing such a sleep period is that, unlike the analysis of the user's finger movements, authentication does not need to be repeated so frequently.

[0054] If the authentication fails (NO in step S23), a flag F indicating the authentication result is set to 0 (step S26). Then, the device sleeps for a fixed period of time (step S27), and returns to step S2 to repeat the authentication process. This fixed period of time may be, for example, 0.1 seconds. The reason for providing such a sleep period is to wait for the user's finger movement.

[0055] In the process shown in FIG. 6 used in the second embodiment, the step of displaying a prompt is omitted from the process shown in FIG. 4 used in the first embodiment. Vein authentication according to the present invention tracks and acquires vein patterns from images of moving fingers, allowing the user to smoothly perform input operations within a normal range of movement. Even if authentication fails, it is considered that the user can immediately recover (authentication is successful) and continue input operations. Therefore, there is usually no problem even if a bothersome prompt is not displayed. If another person attempts input operations, it simply does not work.

[0056] In the above embodiment, if the user fails authentication even once, the user's input operation is immediately invalidated. However, this may hinder smooth input operations. In other words, even if the user is registered, the authentication process may fail by chance.

[0057] In such a case, an algorithm can be used that does not immediately invalidate the user's input operation even if authentication fails once, as in the alternative examples shown in Figures 7 and 8. Figure 7 is a flowchart showing the process of contactless input operation according to this alternative example, and Figure 8 is a flowchart showing the process of vein authentication according to this alternative example.

[0058] The flowchart shown in Fig. 7 differs from the flowchart in Fig. 3 only in the step of determining flag F. That is, while the flowchart in Fig. 3 determines whether flag F is 1, the flowchart in Fig. 7 determines whether flag F is positive in step S2'. This is done in combination with the vein authentication process shown in Fig. 8 so that if authentication is successful twice, the input operation is not invalidated even if authentication fails once. The other steps in the flowchart shown in Fig. 7 are the same as those in the flowchart in Fig. 3, and detailed description thereof will not be repeated.

[0059] In the vein authentication process shown in FIG. 8, a series of images of the user's fingers are captured by the infrared camera 26, as in the above embodiment (step S31).

[0060] Next, the acquired series of images are analyzed to obtain information on the vein pattern of the user's fingers and perform authentication processing (step S32). If authentication is successful (YES in step S33), flag F is incremented by 1 (step S34). However, if the value of flag F is 2, the value of flag F remains 2. Then, the system sleeps for a fixed period of time (step S35), and returns to step S31 to repeat the authentication processing. This fixed period of time may be, for example, 0.5 seconds.

[0061] If the authentication fails (NO in step S33), 1 is subtracted from flag F, which indicates the authentication result (step S36). However, if the value of flag F is 0, the value of flag F remains 0. Then, the system sleeps for a fixed period of time (step S37), and returns to step S3 to repeat the authentication process. This fixed period of time may be, for example, 0.1 seconds.

[0062] In the above algorithm, even if there are successive authentication failures after successive authentication successes (successful authentication > successful authentication > failed authentication), the input operation is not invalidated. On the other hand, if there are successive authentication failures, the input operation is invalidated. This allows a registered user to smoothly perform input operations even if the authentication process happens to fail.

[0063] 7 and 8, in step S34, the value of flag F is incremented by 1 if authentication is successful, but is not set to exceed the upper limit of 2. In step S34, the upper limit may be set to 3 or more to adjust the authentication tolerance. For example, if the fixed number is set to 3, after three consecutive successful authentication attempts, the input operation will not be invalidated even if thereafter two consecutive unsuccessful authentication attempts.

[0064] Furthermore, if the upper limit is set to 3, the decrement of the value of flag F in step S36 upon authentication failure may be changed from 1 to 2. This makes it possible to invalidate input operations when authentication success and authentication failure occur alternately, such as authentication failure > authentication success > authentication failure. By appropriately changing these upper limit values ​​and decrements according to the probability of authentication malfunction, adjustments can be made to allow registered users to perform input operations safely and smoothly. [Industrial Applicability]

[0065] According to the present invention, it is possible to prevent a situation in which another person inadvertently performs an input operation while an authenticated user is performing an input operation, and the user can carry out a series of operations as he or she wishes.

[0066] The above describes the non-contact input device according to the present invention based on an embodiment, but the present invention is not limited to this, and modifications may be made within the scope of the spirit of the present invention, and if possible, the techniques described in each embodiment may be combined, or publicly known techniques may be combined, etc.

[0067] For example, while the above embodiment employs vein authentication as biometric authentication, the present invention is not limited to this. That is, other authentication methods using biometric information other than vein information can also be employed. Specifically, general fingerprint authentication using the print pattern of each finger, palm print authentication that authenticates an individual based on the pattern created by the ridges of the skin on the palm, and hand geometry authentication that authenticates an individual based on the shape of the palm can also be employed in the same way as the above vein authentication. Furthermore, it is also possible to improve authentication accuracy by combining multiple of these authentication methods.

[0068] In addition, the first embodiment uses the authentication process shown in the flowchart of Fig. 4, and the second embodiment uses the authentication process shown in the flowchart of Fig. 6 or the flowcharts of Fig. 7 and 8, but the combination is not limited to this. For example, the first embodiment may use the authentication process shown in the flowchart of Fig. 6 or the flowcharts of Fig. 7 and 8, and the second embodiment may use the authentication process shown in the flowchart of Fig. 4. [Explanation of symbols]

[0069] 1. Aerial image display device 4. Aerial footage 10. Cabinet 20, 21 LCD display 23 Computer 24 Display surface 25 Motion Sensor 26, 34 Infrared camera 27, 32 Infrared LED 30 Optical Plate 31 Incidence plane 33 Exit surface 39 speakers 40 Control device

Claims

1. A non-contact input device that allows input operations to be performed by moving a hand in the air, An imaging device that captures an image of a user's hand; a detection device that detects a hand movement of a user based on the image captured by the imaging device and detects an input operation corresponding to the detected hand movement; an authentication device that performs authentication using biometric information of the user's hand based on the image captured by the imaging device; A non-contact input device characterized in that if the authentication is successful, the input operation is validated, and even after the authentication is successful and the input operation is validated, the authentication is continuously repeated, and the user can continuously perform input operations only while the authentication is repeatedly successful.

2. A non-contact input device as described in Claim 1, characterized in that if the authentication fails after the input operation becomes valid due to successful authentication, the input operation is invalidated, but the authentication continues to be performed repeatedly thereafter.