Authentication system, authentication method, registration device, registration method, and program

Abstract

To provide an authentication system, an authentication method, a registration device, a registration method, and a program capable of appropriately performing user authentication and gesture authentication.SOLUTION: In a method for registering one or more gesture images in a database in an authentication system, one or more gesture images showing a hand of a user performing a gesture arbitrarily decided by the user are acquired by a user device or a registration device, reference data including one or more gesture images are generated by associating a gesture appearing in each gesture image with one instruction selected by a user among a plurality of instructions, and the reference data are stored in a database by a registration server.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present disclosure relates to an authentication system, an authentication method, a registration device, a registration method, and a program. [Background technology]

[0002] Patent Document 1 discloses a method for instructing operations using pre-registered gestures to make contactless payments at self-checkouts. In this method, a peace sign gesture is used to make a payment, and a gesture of turning the palm forward is used to cancel the payment. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-030249 Summary of the Invention [Problem to be solved by the invention]

[0004] When performing a payment by gesture instruction, it is necessary to reliably verify the identity of the user prior to the payment in order to prevent unauthorized operations by a third party. For example, Patent Document 1 discloses that user authentication is performed by contactless biometric authentication such as face authentication, iris authentication, or vein authentication.

[0005] When issuing instructions for any process requiring user authentication, not limited to payments, using gestures, it is necessary to properly authenticate the user and the gesture. Furthermore, biometric authentication requires that biometric data, such as face, iris, or hand veins, used for authentication be registered in advance.

[0006] An object of the present disclosure is to provide an authentication system, an authentication method, a registration device, a registration method, and a program that can appropriately perform user authentication and gesture authentication. [Means for solving the problem]

[0007] An authentication system according to one embodiment of the present disclosure includes: a first image acquisition unit configured to acquire one or more gesture images, each of which shows a user's hand making a gesture arbitrarily determined by the user; a data generation unit configured to generate reference data for the user by linking a gesture captured in each of the one or more gesture images with one instruction selected by the user from a plurality of instructions, the reference data including the one or more gesture images; a second image acquisition unit configured to acquire an authentication image, each of which shows the user's hand making a gesture; an authentication processing unit configured to perform biometric authentication and authentication of the gesture captured in the authentication image by comparing the authentication image with the one or more gesture images; and an instruction output unit configured to output the one instruction linked to the authenticated gesture.

[0008] An authentication method according to one embodiment of the present disclosure includes acquiring one or more gesture images, each of which depicts a user's hand making a gesture arbitrarily determined by the user; linking a gesture depicted in each of the one or more gesture images to one instruction selected by the user from a plurality of instructions; acquiring an authentication image, each of which depicts the user's hand making a gesture; comparing the authentication image with the one or more gesture images to perform biometric authentication and authenticate the gesture depicted in the authentication image; and outputting the one instruction linked to the authenticated gesture.

[0009] A registration device according to one embodiment of the present disclosure is a registration device for registering one or more gesture images, the registration device comprising one or more processors and one or more memories storing a registration program, the registration program being configured to be executed by the one or more processors, the registration program including instructions to acquire one or more gesture images, each of which depicts a user's hand making a gesture arbitrarily determined by the user, and to generate reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among multiple instructions, the reference data including the one or more gesture images.

[0010] A registration method according to one embodiment of the present disclosure is a registration method for registering one or more gesture images in a database, the registration method including: acquiring one or more gesture images, each of which depicts a user's hand making a gesture arbitrarily determined by the user; generating reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from a plurality of instructions, the reference data including the one or more gesture images; and storing the reference data in the database.

[0011] A program according to one embodiment of the present disclosure is a program including a plurality of instructions, the plurality of instructions including instructions to acquire one or more gesture images, each of which depicts a user's hand making a gesture arbitrarily determined by the user, and to generate reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of an authentication system according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram showing an example of a registration screen displayed by the registration program according to the embodiment. [Figure 3] FIG. 3 is a schematic diagram showing another example of the registration screen displayed by the registration program according to the embodiment. [Figure 4] FIG. 4 is a flow diagram showing a registration method according to an embodiment. [Figure 5] FIG. 5 is a flow diagram showing an authentication method according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] Examples of an authentication system, authentication method, registration device, registration method, and program will be described below with reference to the drawings. The present invention is not limited to these examples, but is defined by the claims, and is intended to include all modifications within the meaning and scope of the claims.

[0014] [Terminology] In the present disclosure, a "gesture" primarily involves a hand signal using the fingers. A gesture includes a motionless hand signal, but may also involve a hand or arm movement. One gesture is associated with one instruction. According to the system, device, method, and program of the present disclosure, each of multiple users can associate one gesture of their own choosing with one instruction selected by them from multiple instructions.

[0015] The multiple instructions are, for example, instructions that would conventionally be input manually to a device or system, for example, by operating a button. The multiple instructions may also be, for example, a series of instructions required in a payment procedure at a cash register. For example, when making a payment at a store, the multiple instructions that can be given by gestures may be predetermined by the store operator. The series of instructions may include, for example, a primary instruction such as specifying a payment method and a secondary instruction such as presenting a point card. Payment performed by a gesture instruction is called gesture payment.

[0016] The technology disclosed herein provides an authentication service that enables such operation instructions to be replaced with gestures associated with the instructions. The technology disclosed herein can be applied to instructions for procedures that require user authentication, and is not limited to instructions for operations involving payment. The authentication service performs user authentication and gesture authentication before executing an instruction. Hereinafter, gesture authentication will be simply referred to as "gesture authentication."

[0017] [Authentication system overview] The authentication system 11 in Fig. 1 is configured to provide an authentication service. The authentication system 11 also provides a registration service for one or more users who use the authentication service, allowing each user to register user data. The authentication system 11 is configured to perform user authentication and gesture authentication based on a gesture image showing the user's hand making a gesture. The user data includes one or more gesture images registered for user authentication and gesture authentication. User authentication based on gesture images is biometric authentication.

[0018] The authentication system 11 may include one or more servers 12 and databases 13. Each server 12 may include one or more processors 14 and one or more memories 15. Various programs are stored in the memories 15. The one or more processors 14 execute one or more instructions included in the programs, thereby causing the server 12 to realize various functions.

[0019] In the following, an example will be described in which one or more servers 12 include a registration server 12A and an authentication server 12B. In another example, one server 12 may perform the functions of both the registration server 12A and the authentication server 12B. In yet another example, at least one of the servers 12A and 12B may include a database 13. If the server 12 does not include the database 13, each server 12 accesses the database 13 to refer to or update the database stored in the database 13.

[0020] The database 13 may include a store database and a user database. The store database includes store data for one or more stores that employ the authentication system 11. The store data includes, for example, but is not limited to, the store name, store ID, location, and available payment methods. Examples of payment methods include, but are not limited to, electronic money payment, cash payment, credit card payment, and point payment. The user can select any payment method from one or more payment methods supported by each store.

[0021] The electronic money payment may be an electronic payment using pre-registered credit card information, a code payment using a two-dimensional code such as a QR code (registered trademark), or a prepaid electronic money payment where cash is charged in advance, but is not limited to these.

[0022] The user database contains user data for each of multiple users who use the authentication system 11. User data includes, but is not limited to, name, user ID, address, payment information, and various registration information related to authentication. Examples of payment information include credit card number, debit card number, or debit account number.

[0023] The authentication system 11 may include one or more authentication devices 40. The authentication devices 40 may be installed in one or more stores that employ the authentication system 11. The authentication device 40 may have a guidance function for providing operation instructions using text information, illustrations, or audio. The authentication device 40 may be integrated with a payment terminal such as a self-checkout device, as shown in FIG. 1.

[0024] The authentication device 40 may include one or more processors 41, one or more memories 42, a display 43, a camera 44, and one or more operation units 45. The memory 42 stores an authentication program for the authentication process, which is executed by the one or more processors 41. The memory 42 stores an authentication program for executing the authentication process.

[0025] The camera 44 may be separate from the authentication device 40, or may be externally attached to the authentication device 40. The one or more operation units 45 may include buttons, or may include a touch panel integrated with the display 43. The one or more operation units 45 may include buttons, or may include a touch panel integrated with the display 43.

[0026] The authentication system 11 may include one or more registration devices 30. The registration devices 30 may be placed in one or more stores that employ the authentication system 11, but the installation location is not limited to a store. The registration devices 30 may include a guidance function for providing operation instructions using text information, illustrations, or audio.

[0027] The registration device 30 may include one or more processors 31, one or more memories 32, a display 33, a camera 34, and one or more operation units 35. The memory 32 stores a registration program for registering user data, which is executed by the one or more processors 31. The camera 34 may be separate from the registration device 30, or may be externally attached to the registration device 30. The one or more operation units 35 may include buttons, or may include a touch panel integrated with the display 33.

[0028] The registration program includes instructions for displaying a photographing screen 36 on the display 33. The photographing screen 36 may include a photographing area 37 in which an image captured by the camera 34 is displayed, and one or more operation buttons 38. The photographing area 37 defines the range of an image to be acquired. The user can photograph one or more gesture images for registration via the photographing screen 36.

[0029] The captured gesture image is stored in memory 32. As a result, registration device 30 acquires the gesture image. If the user desires to give multiple instructions with gestures, the user captures at least as many gesture images as the desired instructions. If camera 34 is a 3D camera, at least some of the one or more gesture images may be three-dimensional images.

[0030] After capturing the gesture images, the user associates each gesture with one of a plurality of instructions using the registration device 30. Based on this, the registration device 30 generates reference data for the user. The reference data includes data for identifying the user, one or more gesture images, and data indicating the instructions associated with each gesture.

[0031] The registration device 30 is configured to transmit the generated reference data to the registration server 12A via a network, which may include, but is not limited to, the Internet, a wide area network (WAN), a local area network (LAN), a provider terminal, a wireless communication network, a wireless base station, a dedicated line, etc.

[0032] The registration server 12A acquires one or more gesture images by receiving reference data from the registration device 30. The registration server 12A is configured to register the reference data received from the registration device 30 in the database 13. The gesture images included in the reference data and registered in the database 13 are also referred to as reference images.

[0033] The authentication device 40 may be integrated with the registration device 30. In this case, the processor 41, the memory 42, the display 43, the camera 44, and the operation unit 45 may be shared with the processor 31, the memory 32, the display 33, the camera 34, and the operation unit 35, respectively.

[0034] The registration device 30 may be replaced by a user terminal 20 equipped with a camera used by a user. The user terminal 20 may be a computer with an external camera, or a computer or mobile terminal with a built-in camera. Examples of mobile terminals include, but are not limited to, a tablet or a smartphone.

[0035] The user terminal 20 includes, for example, one or more processors 21, one or more memories 22, a display 23, a camera 24, and one or more operation units 25. The one or more operation units 25 may include buttons or a touch panel integrated with the display 23.

[0036] A registration application (hereinafter abbreviated as "registration application") may be installed in the user terminal 20. The program of the registration application is stored in the memory 22 upon installation. When the processor 21 executes the program of the registration application, the user terminal 20 operates as a registration device.

[0037] The program of the registration app may include instructions similar to those of the registration program stored in the memory 32 of the registration device 30, or may include instructions that realize more limited functions than those of the registration device 30. For example, there may be a difference in function between the registration program included in the registration app and the registration program of the registration device 30, such that the registration device 30 can register a three-dimensional image as a gesture image, but the user terminal 20 cannot.

[0038] When the registration app is started by the user, it displays a shooting screen 26 on the display 23. The shooting screen 26 may include a shooting area 27 that displays an image captured by the camera 24, and one or more operation buttons 28. The shooting area 27 defines the range of the image to be captured.

[0039] When the user operates operation button 28 (photograph button) with the hand making the hand sign photographed in photographing area 27, the registration application stores the acquired gesture image in memory 22. As a result, user terminal 20 acquires the gesture image. Thereafter, the registration application generates reference data according to the user's operation.

[0040] The registration application transmits the generated reference data to the registration server 12A via the network. The registration server 12A receives the reference data to acquire a gesture image and registers the reference data in the database 13.

[0041] When a user makes a gesture payment, authentication device 40 captures an image of the gesture made by the user for authentication. The image of the gesture captured by authentication device 40 for authentication is called an authentication image. Authentication device 40 is configured to transmit the captured authentication image to authentication server 12B via a network.

[0042] The authentication image of one user may be the same type as the reference image registered by that user, or may be a different type. For example, if the reference image is a video, the authentication image may also be a video, or if the reference image is a three-dimensional image, the authentication image may also be a three-dimensional image. Alternatively, one of the authentication image and the reference image may be a still image, and the other may be a video.

[0043] For example, if the reference image is a still image, the corresponding authentication image may be a video. In this case, multiple still images corresponding to multiple time points in the video may be generated, and these multiple still images may be compared with the reference image. Alternatively, if the reference image is a three-dimensional image, the corresponding authentication image may be a two-dimensional image. In this case, multiple two-dimensional images with different viewpoints may be generated from the three-dimensional image, and the authentication image may be compared with the multiple two-dimensional images.

[0044] The authentication server 12B performs user authentication and gesture authentication by comparing the authentication image with one or more reference images. Once the user authentication and gesture authentication are completed, the payment terminal performs payment processing for the authenticated user according to instructions associated with the authenticated gesture.

[0045] [Register gesture image] Each reference image registered for a user must include at least gesture data based on the shape of the hand that indicates the gesture. The gesture data is used to identify the gestures shown in the image. The gesture data may be, for example, data extracted from features such as the outline of the hand, but is not limited to this.

[0046] At least one of the one or more reference images registered for one user must include biometric data based on the characteristics of the hand while making a gesture. The hand characteristics reflected in the biometric data may be, for example, the veins of the palm, the back of the hand, or the fingers, or the unevenness of the palm (so-called palm lines). The biometric authentication may be palm authentication. The biometric authentication may also be vein authentication based on the vein pattern of the veins of the palm, the back of the hand, or the fingers.

[0047] A gesture image that includes both biometric data and gesture data is called a standard image. A gesture image that includes gesture data but not biometric data is called an auxiliary image. The collective term for biometric data and gesture data is authentication data.

[0048] When enrolling multiple gestures, it may be difficult to devise and execute enough gesture variations that show the palm or back of the hand to include biometric data. Therefore, if multiple gesture images are required, some of them may be auxiliary images. The enrollment program may include instructions for displaying easy-to-use gesture examples in the standard image or auxiliary image as part of the instructions.

[0049] The registration program may include instructions for determining whether biometric data can be acquired from the hand captured in the gesture image, and may further include instructions for notifying the user of the determination result if it is determined that biometric data cannot be acquired.

[0050] For example, the registration program may change the display when both biometric data and gesture data can be acquired from the hand captured in the image capture area 27. For example, the display may be changed so that the highlight line surrounding the outline of the hand changes from blinking (shown by a dashed line in FIG. 1) to being constantly lit, but this is not limiting. Additionally or alternatively, when biometric data and gesture data can be acquired from the hand captured in the image capture area 27, the operation button 28 may be displayed or the operation button 28 may become operable. Such a change in the display on the screen is also an example of a notification.

[0051] Furthermore, the registration program may include an instruction for determining whether gesture data can be acquired when biometric data cannot be acquired from the gesture image captured in the image capturing area 27. The registration program may include an instruction for notifying the user of the determination result when it is determined that gesture data can be acquired. The notification may be a text or audio message, or a change in display on the screen.

[0052] At least one of the biometric data and the gesture data may be acquired by the registration server 12A. The registration server 12A may also determine whether or not the authentication data can be acquired. In this case, the registration program may include an instruction to transmit the acquired gesture image to the registration server 12A and an instruction to subsequently acquire the determination result from the registration server 12A.

[0053] At least some of the one or more gesture images may be gesture videos that capture gestures in motion. The gesture videos may be the motions themselves that serve as signs. Alternatively, the gesture videos may capture images of a hand making a hand sign while rotating the hand around the forearm as an axis of rotation, with the palm or back of the hand photographed at different angles relative to the camera. In this case, hand features for biometric authentication can be captured from multiple angles.

[0054] The registration program may include instructions for acquiring a plurality of still images from the gesture video and storing them as a series of gesture images in the memory 22. Such a series of gesture images includes a plurality of images showing a single gesture from a plurality of different directions.

[0055] 2 and 3 show examples of a registration screen 70 (70A, 70B) that is displayed on the display 33 by executing the registration program. The registration screen 70 is used to allow the user to associate a gesture with an instruction.

[0056] The registration program may include an instruction for displaying, each time a gesture image is captured, a registration screen 70A including the captured gesture image 71, as shown in Fig. 2. Alternatively or in addition to this, the registration program may include an instruction for displaying, after capturing a plurality of gesture images, a registration screen 70B displaying a list of the plurality of gesture images 71, as shown in Fig. 3. The user can associate a plurality of instructions with each of the plurality of gesture images 71 via the registration screen 70B.

[0057] The registration screen 70 may include a display area 72 for gesture images 71, one or more selection buttons 73, one or more decision buttons 74, and one or more deletion buttons 75. One or more gesture images 71 captured by the camera 34 are displayed in the display area 72. One or more selection buttons 73 and one deletion button 75 may be arranged corresponding to each gesture image. When the deletion button 75 is operated, the corresponding gesture image is deleted from the memory 32.

[0058] When multiple gesture images 71 include auxiliary images 71b, the standard image 71a and the auxiliary image 71b may be displayed differently on the registration screen 70. In Figures 2 and 3, the standard image 71a is displayed with a double frame, and the auxiliary image 71b is displayed with a single frame. Alternatively, or in addition, different marks may be added to the standard image 71a and the auxiliary image 71b.

[0059] The registration program may include instructions for displaying an edit screen (similar to registration screen 70B) for editing registered reference data. On the edit screen, the user can delete registered gesture images or change the linking instructions.

[0060] The one or more instructions may include, for example, one or more payment instructions corresponding to one or more payment methods. The one or more payment methods may include, but are not limited to, at least one of one or more of electronic money payment, cash payment, credit card payment, and point payment. The one or more electronic money payment may include electronic payment services provided by multiple different businesses. The electronic money payment includes code payment using a two-dimensional code.

[0061] The one or more instructions may include at least one of presenting a point card related to a point program, using points, using a coupon, and charging electronic money. A point program is an example of a loyalty program for giving benefits to users, and uses "points" as a medium for exchanging for the benefits. The unit of the medium for exchanging for the benefits is not limited to "points," and can be replaced with any other name, such as "miles," "money," "coins," "dollars," "stamps," or "cash."

[0062] When the multiple gesture images 71 include an auxiliary image 71b, the instructions that can be linked between the standard image 71a and the auxiliary image 71b may be different. For example, the auxiliary image 71b may be able to be linked to an auxiliary instruction among the multiple instructions. The auxiliary instruction may be an instruction that does not involve payment. The auxiliary instruction may include, for example, instructions such as "cancel" a previous instruction, "check the charge balance," or "issue a receipt" after payment, but is not limited to these.

[0063] As shown in Fig. 3, the multiple options indicated by the selection button 73 may be grouped into categories such as "cashless payment," "points related," or "other operations." The "other operations" may include auxiliary instructions. After the user specifies an instruction to be linked to each gesture image 71 using the selection button 73, the user operates the decision button 74, and reference data is generated.

[0064] The instruction associated with the gesture captured in the standard image may be an initial instruction that should be issued at the beginning of a series of processes. The instruction associated with the gesture captured in the auxiliary image may be an instruction that should be issued after the initial instruction. The initial instruction may be, for example, the first or second instruction in a series of processes. In this case, if biometric authentication is completed based on the standard image in the initial instruction, subsequent instructions can be considered to be executed by the same user, and user authentication can be omitted each time.

[0065] For example, in a series of processes related to payment, if the first instruction is an instruction to present a point card (including an instruction to "not present a point card"), the second instruction may be an instruction to specify a payment method. Subsequent instructions may be supplementary instructions such as an instruction to print a receipt or an instruction to display the charge balance, but are not limited to these.

[0066] [How to register] Registration of a gesture image that serves as a reference image will be described with reference to Fig. 4. Here, an example in which registration is performed using a registration application of the user terminal 20 will be shown, but similar processing is performed when registration is performed using the registration device 30.

[0067] When the user operates the operation button 28 with the hand making a gesture captured in the image capturing area 27, a gesture image is acquired (step S11). Next, the registration application acquires authentication data from the acquired gesture image (step S12). The authentication data includes biometric data for biometric authentication and gesture data for identifying the gesture.

[0068] The registration application associates the gesture depicted in each of the one or more gesture images with one instruction selected by the user from among the multiple instructions (step S13) in accordance with the user's operation via the registration screen 70. As a result, the registration application generates reference data including the gesture image (step S14).

[0069] Thereafter, the registration application transmits the generated reference data to the registration server 12A based on the user's operation (step S15). The registration server 12A receives the reference data and acquires a gesture image (step S16). The registration server 12A registers the received reference data in the user database as user data of the corresponding user (step S17).

[0070] The registration server 12A may transmit registration completion information to the user terminal 20 (or the registration device 30) that transmitted the reference data (step S18). The user terminal 20 (or the registration device 30) that received the registration completion information may display the registration completion information on the display 23 (step S19).

[0071] [Store visit detection system] As shown in FIG. 1, the authentication system 11 may include a store visit detection system 50. The store visit detection system 50 is used to assist user authentication by the authentication server 12B. The store visit detection system 50 is configured to detect a user's visit to a store via a mobile terminal 60 carried by the user. The store is a store that employs the authentication system 11 and is registered in a store database.

[0072] The mobile terminal 60 may be, but is not limited to, a tablet, a smartphone, or a smartwatch. For example, the mobile terminal 60 may be a signal transmitting / receiving device or a user terminal 20.

[0073] The store entrance detection system 50 may include one or more signal emitting devices 51. The signal emitting devices 51 are configured to emit signals within the store. The signal emitting devices 51 may be installed in one or more locations within the store, such as at the store entrance, on the ceiling, or near a cash register.

[0074] The signal transmitting device 51 may be, for example, a beacon, an ultrasonic transmitting device that emits ultrasonic waves, or a device that transmits short-range wireless communication signals such as Bluetooth (registered trademark) or UWB (Ultra-Wide Band), but is not limited to these.

[0075] The mobile terminal 60 has a receiving function for receiving a signal transmitted from the signal transmission device 51. The receiving function may be, for example, a microphone mounted on the mobile terminal 60, or a receiver for short-range wireless signals.

[0076] The store visit detection system 50 may include a management computer 52 located in each store. The management computer 52 may include one or more processors 53 and one or more memories 54. The memories 54 store programs for implementing various functions. The management computer 52 may be configured to control one or more signal emitting devices 51. The management computer 52 may be capable of communicating with the authentication server 12B via a network.

[0077] A notification program for notifying the customer of a visit to the store may be stored in the memory of the mobile terminal 60. When the mobile terminal 60 is the user terminal 20, the notification program may be included in the program of the registered application.

[0078] The notification program includes instructions for transmitting a store visit notification to authentication server 12B when mobile terminal 60 receives a signal transmitted from signal transmission device 51. Alternatively, mobile terminal 60 may transmit the store visit notification to management computer 52, which may then forward the store visit notification to authentication server 12B.

[0079] The store visit notification includes user information. The user information does not need to include personal information such as the user's name, but may include at least a user ID for identifying the user. 1 illustrates an example of a signal transmitting device 51 placed on the ceiling near the cash register. When the signal transmitting device 51 is placed near the cash register, the mobile terminal 60 receives a signal when the mobile terminal 60 approaches the cash register. Therefore, the mobile terminal 60 transmits a store visit notification to the authentication server 12B, which enables the authentication server 12B to detect that the user has approached the cash register.

[0080] [User authentication and gesture authentication] The authentication process executed by the authentication server 12B prior to payment will be described using an example in which a user visits a store and makes a gesture payment at a self-checkout register with reference to Fig. 5. The authentication process includes user authentication and gesture authentication.

[0081] When a user carrying the mobile terminal 60 approaches the self-checkout device, the mobile terminal 60 receives a signal transmitted from the signal transmitting device 51. Upon receiving the signal, the mobile terminal 60 transmits a store visit notification to the authentication server 12B.

[0082] The authentication server 12B receives the user information included in the store visit notification (step S21). Subsequently, the authentication server 12B authenticates the user based on the user information (step S22). More specifically, the authentication server 12B refers to the user database to detect the user who is the owner of the mobile terminal 60 that transmitted the user information.

[0083] The detected user may not be a user who will make a gesture payment, but is a candidate who is likely to make a payment. Therefore, the authentication server 12B stores the user information of the detected user together with the detection date and time as candidate data in the memory 15. The candidate data may be deleted after a certain period of time has passed, and only candidate data within the most recent specified period of time may be retained. The specified period of time may be set, for example, according to the waiting time at the cash register.

[0084] Before making a payment at the self-checkout machine, a user making a gesture payment undergoes user authentication using an authentication device 40 integrated with the self-checkout machine. This user authentication also serves as an operation instruction linked to the gesture.

[0085] First, the user makes a gesture to give an initial instruction toward the camera 44, following the guidance of the authentication device 40. The authentication device 40 captures an image of the gesture with the camera 44. As a result, the authentication device 40 acquires an authentication image that shows the hand of the user making the gesture (step S23). Next, the authentication device 40 transmits the acquired authentication image to the authentication server 12B (step S24).

[0086] The authentication server 12B receives the authentication image sent by the authentication device 40 (step S25), thereby acquiring the authentication image. Next, the authentication server 12B performs an authentication process (step S26). The authentication process includes a biometric authentication process and a gesture authentication process. More specifically, the authentication server 12B compares the authentication image with one or more reference images registered in a user database.

[0087] If one or more users are detected in step S22, the authentication server 12B may first compare the reference image and authentication image for one or more users who have been detected as visiting the store. If the authentication data in the reference image and authentication image of the detected user do not match, the authentication server 12B may compare the reference image and authentication image of other users who have not been detected as visiting the store.

[0088] In the authentication process, the authentication server 12B may first perform biometric authentication by comparing biometric data acquired from the authentication image with biometric data of one or more reference images. The biometric authentication process detects a reference image containing biometric data that matches the biometric data of the authentication image. The authentication server 12B may also compare gesture data acquired from the authentication image with gesture data of the reference images detected in the biometric authentication process. In this way, by performing authentication in two stages, using biometric data and gesture data, the accuracy of authentication can be improved.

[0089] Alternatively, authentication server 12B may perform both biometric authentication and gesture authentication using only biometric data. For example, in the case of palm authentication or vein authentication, biometric data includes characteristics of the hand when making a gesture (hand sign), and therefore may also include gesture data. In another example, authentication server 12B may perform two-stage authentication using biometric data and gesture data when the matching rate of biometric data between the authentication image and the reference image is less than a specified value.

[0090] When the reference image whose authentication data matches the authentication image is identified, the authentication process is completed. Then, the authentication server 12B outputs an instruction associated with the identified reference image (step S27).

[0091] Authentication server 12B transmits the output instruction to authentication device 40 (step S28). Upon receiving the instruction, authentication device 40 executes processing corresponding to the instruction (step S29). For example, if the instruction specifies a payment method, "credit card payment," a payment terminal integrated with authentication device 40 executes credit card payment using the credit card number registered as user data of the target user. While FIG. 5 describes that authentication device 40 integrated with the payment terminal executes step S29, if the payment terminal is separate from authentication device 40, the processing of step S29 may be executed by that separate device.

[0092] When other instructions, such as presenting a point card or charging electronic money, are also given along with the payment, the user follows the instructions of the authentication device 40 and makes the corresponding gesture toward the camera 44, and the authentication device 40 and the authentication server 12B repeat steps S23 to S29.

[0093] [Effects of the present disclosure] The technology disclosed herein allows both user authentication and gesture authentication to be performed using gesture images, which simplifies the configuration of authentication device 40 and reduces the user's workload compared to devices that perform user authentication (e.g., face authentication or iris authentication) and gesture authentication separately.

[0094] Here, if gestures corresponding to multiple instructions are predefined, it is not possible to provide many gesture options. For example, when palm authentication is adopted, there are only a limited number of gestures that can be used to display the palm. This makes it difficult to select multiple payment methods or issue multiple auxiliary instructions using gestures alone. For example, if gestures are used that involve movement, the variety of gestures can be increased, but it becomes difficult for users to remember and reproduce the predefined gestures.

[0095] In this regard, the technology disclosed herein allows a user to decide for themselves which gestures correspond to each of a plurality of instructions. Therefore, the user only needs to register gestures that correspond to the instructions that the user uses most frequently. This allows the user to execute a desired instruction from among a plurality of instructions using a gesture that is easy for the user to perform, thereby improving user convenience.

[0096] [Effects of this disclosure] According to the present disclosure, the following effects can be achieved. (1) The gestures captured in the gesture image can be arbitrarily determined by the user. Therefore, each of multiple users can adopt a gesture that is easy for them to remember and perform. Furthermore, since different users use different gestures for the same instruction, even if a gesture is made in a public place such as a storefront, a third party cannot know what instruction the gesture represents. Therefore, the method disclosed herein provides higher confidentiality than when gestures corresponding to instructions are determined in advance.

[0097] (2) Although there are more than 10 payment methods, users only need to select gestures for the payment methods they use. This allows users to prioritize and associate their favorite gestures with the payment methods they use most frequently.

[0098] (3) By linking gestures to multiple auxiliary instructions associated with payment, such as presenting a point card, using points, using a coupon, or charging electronic money, it becomes possible to perform the corresponding process without using an object such as a point card, a card for using electronic money, or a mobile terminal.

[0099] (4) By using the vein pattern of a hand while making a hand sign as biometric data, biometric authentication and gesture authentication can be performed simultaneously. (5) When the gesture image is a gesture video, the characteristics of the motion are added to the characteristics obtained from the still hand sign, which increases the variety of gestures.

[0100] (6) When the gesture image is a series of images or a three-dimensional image, gesture data and biometric data viewed from multiple angles can be acquired, thereby improving authentication accuracy. Furthermore, even if the gesture involves movement, if the gesture image is a series of images or a three-dimensional image, it is easy to acquire gesture data and biometric data.

[0101] (7) When a user freely decides a gesture, there are cases where biometric data cannot be acquired from a gesture image capturing the gesture, for example, when the palm of the hand is difficult to see. If biometric data cannot be acquired, the user can be notified of this and asked to retake the gesture image.

[0102] (8) Gesture data indicating hand shape characteristics and biometric data for biometric authentication may be obtained separately from the gesture image of 1. For example, in the case of palm or vein authentication, the biometric data includes the characteristics of the hand while making the gesture (hand sign). Therefore, the gesture may be identified using only the biometric data. However, for example, in the case of a hand sign with one or more fingers bent, it may be difficult to distinguish characteristics such as the contours or veins of the hand. In this case, authentication accuracy can be improved by performing authentication using gesture data in addition to biometric data.

[0103] (9) In addition to the standard image containing both biometric data and gesture data, the use of an auxiliary image containing only gesture data can increase the number of gesture patterns. For example, the auxiliary image does not need to contain biometric data, allowing for a wider variety of gestures.

[0104] (10) By linking the initial instructions to gestures shown in the standard image during a series of processes associated with payment, user authentication can be performed early in the process. Since user authentication has already been completed, instructions can be given using gestures shown in the supplementary image during subsequent processes.

[0105] (11) When the mobile terminal 60 receives a signal transmitted within the store and transmits user information to the authentication server 12B, the authentication server 12B can detect that the corresponding user is present within the store. Therefore, user authentication can be performed before biometric authentication to narrow down the candidate users. This can shorten the matching time and reduce the calculation load. Furthermore, authentication can be performed using two factors: store visit detection and biometric authentication, thereby improving the accuracy of authentication.

[0106] [Example of change] This embodiment can be modified as follows: This embodiment and the following modifications can be combined and implemented within the scope of technical compatibility.

[0107] [Change Example 1] A gesture involving movement may include a changing hand sign. For example, the gesture may involve a movement such as changing from an open palm to a closed palm, or a gesture involving a turning over of the hand. In this case, the movement itself may be acquired as gesture data, or the hand shapes at two or more points in time during a series of movements may be acquired as gesture data.

[0108] [Change Example 2] The process executed by gesture instruction is not limited to payment. For example, gesture authentication may be employed for one or more processes or a series of processes related to transactions such as reservations, investments, bidding, admission to facilities, and rental of items including vehicles.

[0109] [Change Example 3] When the registration device 30 is realized as a device integrated with the authentication device 40, the database 13 may be stored in the memory 32. In this case, steps S15, S16, S18, S24, S25, and S28 may be omitted, and the processing of steps S17, S26, and S27 may be performed by the registration device 30 (authentication device 40). In this case, the authentication system 11 may not include the registration server 12A, the authentication server 12B, and the database 13, and may be realized by one or more registration devices 30 (authentication devices 40).

[0110] [Change Example 4] The database 13 may be stored in the memory 42 of the authentication device 40. In this case, steps S24, S25, and S28 may be omitted, and the processing of steps S26 and S27 may be performed by the authentication device 40 instead of the authentication server 12B. In this case, the authentication system 11 may not include the authentication server 12B and the database 13.

[0111] [Change Example 5] The database 13 may be stored in the memory 54 of the management computer 52. In this case, the processes of steps S26 to S27 may be performed by the management computer 52 instead of the authentication server 12B. In this case, the authentication system 11 does not need to include the authentication server 12B and the database 13.

[0112] [Change Example 6] Steps S12 to S14 may be executed by the registration server 12A instead of the user terminal 20 or the registration device 30. In this case, the user terminal 20 or the registration device 30 may transmit the acquired gesture image to the registration server 12A after step S11.

[0113] [Specific Configuration Example 1 of the Embodiment] The processors 14, 21, 31, 41, and 53 may be, for example, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor unit (MPU), a field-programmable gate array (FPGA), or other computing devices. The processors 14, 21, 31, and 41 may be, for example, processing circuits configured to execute various software processes. The processing circuits may include dedicated hardware circuits (e.g., ASICs) that process at least a portion of the software processes. In other words, the software processes may be executed by processing circuitry that includes at least one of one or more software processing circuits and one or more dedicated hardware circuits.

[0114] [Specific Configuration Example 2 of the Embodiment] The memory 15, 22, 32, 42, 54 may be a non-transitory computer-readable medium. The memory 15, 22, 32, 42, 54 may include, for example, random access memory (RAM) or other volatile memory. The memory 15, 22, 32, 42, 54 may be configured to temporarily store programs and data. The memory 15, 22, 32, 42, 54 may also include storage for permanently storing data including programs. Programs permanently stored in the memory 15, 22, 32, 42, 54 may include an operating system (OS) for controlling the overall operation of the computer and applications for executing specific operations.

[0115] [Note] Below, aspects that can be understood from the above embodiment and modified examples are listed. [Aspect 1] a first image capture unit configured to capture one or more gesture images, each of the gesture images depicting a user's hand making a gesture arbitrarily determined by the user; and a data generation unit configured to generate reference data for the user by linking a gesture captured in each of the one or more gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; and a second image capture unit configured to capture an authentication image, the authentication image including the user's hand making a gesture; an authentication processing unit configured to perform biometric authentication and authentication of the gestures captured in the authentication image by comparing the authentication image with the one or more gesture images; an instruction output unit configured to output the one instruction associated with the authenticated gesture; An authentication system comprising:

[0116] In the above embodiment, The user terminal 20 (particularly the camera 24) or the registration device 30 (particularly the camera 34) that acquires the gesture image in step S11, or the registration server 12A that acquires the gesture image in step S16 corresponds to a first image acquisition unit, The user terminal 20 or the registration device 30 that executes steps S13 to S14 corresponds to the data generation unit, The authentication device 40 (particularly the camera 44) that takes the authentication image in step S23 or the authentication server 12B that acquires the authentication image in step S25 corresponds to the second image acquisition unit. The authentication server 12B that performs the authentication process in step S26 corresponds to the authentication processing unit. The authentication device 40 that outputs the instruction in step S27 corresponds to an instruction output unit.

[0117] [Aspect 2] the plurality of instructions include a plurality of payment instructions respectively corresponding to a plurality of different payment methods; The plurality of payment methods include at least one of electronic money payment, cash payment, credit card payment, or point payment. 2. The authentication system according to claim 1.

[0118] [Aspect 3] The plurality of instructions include at least one of presenting a point card, using points, using a coupon, or charging electronic money. 3. The authentication system according to aspect 1 or 2.

[0119] [Aspect 4] The biometric authentication is vein authentication based on the vein pattern of the palm or the back of the hand. The authentication system according to any one of aspects 1 to 3.

[0120] [Aspect 5] The one or more gesture images include a gesture video that shows a gesture in motion. The authentication system according to any one of aspects 1 to 4.

[0121] [Aspect 6] At least one of the authentication image and the one or more gesture images is a series of images including a plurality of images capturing one gesture from a plurality of different directions, or is a three-dimensional image. The authentication system according to any one of aspects 1 to 5.

[0122] [Aspect 7] The control unit further includes: The method is configured to determine whether biometric data for the biometric authentication can be acquired from the hand captured in each of the one or more gesture images, When it is determined that the biometric data cannot be acquired, the determination result is notified to the user. The authentication system according to any one of aspects 1 to 6.

[0123] In the above embodiment, the registration device 30, the user terminal 20, or the registration server 12A that executes the above determination and notification corresponds to the control unit. More specifically, the registration device 30, the user terminal 20, or the registration server 12A operates as the control unit by the processors 21, 31, and 14 executing instructions related to the determination and notification that are included in the registration program.

[0124] [Aspect 8] The system further includes one or more information acquisition units, a gesture data acquisition unit configured to acquire gesture data for identifying the gesture from each of the one or more gesture images; a biometric data acquisition unit configured to acquire biometric data for the biometric authentication from at least a portion of the one or more gesture images; Including, The authentication system according to any one of aspects 1 to 7.

[0125] In the above embodiment, the registration device 30, the user terminal 20, or the registration server 12A that acquires the authentication data in step S12 corresponds to the information acquisition unit (gesture data acquisition unit and biometric data acquisition unit).

[0126] [Aspect 9] The one or more gesture images include a standard image including both the biometric data and the gesture data, and an auxiliary image including only the gesture data out of the biometric data and the gesture data. 9. The authentication system according to claim 8.

[0127] [Aspect 10] the plurality of instructions are instructions issued by the user to execute a series of processes associated with payment, The instruction associated with the gesture shown in the standard image is an initial instruction that should be issued at the beginning of the series of processes, The instruction associated with the gesture shown in the auxiliary image is an instruction to be issued after the initial instruction. 10. The authentication system according to embodiment 9.

[0128] [Aspect 11] a signal emitting device configured to emit a signal within the store; an authentication server; Further provided with The authentication server The mobile terminal carried by the user is configured to receive user information for identifying the user when the mobile terminal receives the signal, and then, configured to authenticate the user based on the user information; The authentication system according to any one of aspects 1 to 10.

[0129] [Aspect 12] acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture of the user's choosing; Associating a gesture depicted in each of the one or more gesture images with one instruction selected by the user from among a plurality of instructions; acquiring an authentication image, the authentication image including the user's hand making a gesture; and performing biometric authentication and authenticating the gestures captured in the authentication image by comparing the authentication image with the one or more gesture images; outputting the first instruction associated with the authenticated gesture; authentication methods, including

[0130] [Aspect 13] 1. A registration device for registering one or more gesture images, comprising: one or more processors; one or more memories storing a registration program; Equipped with the registration program is configured to be executed by the one or more processors; The registered program: acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture of the user's choosing; generating reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; a registration device including instructions for:

[0131] [Aspect 14] 1. A method for registering one or more gesture images in a database, comprising: acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture arbitrarily determined by the user; generating reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; storing the reference data in the database; How to register, including:

[0132] [Aspect 15] A program including a plurality of instructions, the plurality of instructions comprising: acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture of the user's choosing; generating reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; A program containing instructions.

[0133] [Aspect 16] The plurality of instructions further comprises: acquiring an authentication image, the authentication image including the user's hand making a gesture; and performing biometric authentication and authenticating the gestures captured in the authentication image by comparing the authentication image with the one or more gesture images; outputting the first instruction associated with the authenticated gesture; 16. The program of claim 15, comprising the instructions: [Explanation of symbols]

[0134] 11...Authentication system, 12...Server, 12A...Registration server, 12B...Authentication server, 13...Database, 14...Processor, 15...Memory, 20...User terminal, 21...Processor, 22...Memory, 23...Display, 24...Camera, 25...Operation unit, 26...Photographing screen, 27...Photographing area, 28...Operation button, 30...Registration device, 31...Processor, 32...Memory, 33...Display, 34...Camera, 35...Operation unit, 36...Photographing screen, 37...Photographing Area, 38...operation button, 40...authentication device, 41...processor, 42...memory, 43...display, 44...camera, 45...operation unit, 50...store visit detection system, 51...signal transmission device, 52...management computer, 53...processor, 54...memory, 60...mobile terminal, 70, 70A, 70B...registration screen, 71...gesture image, 71a...standard image, 71b...auxiliary image, 72...display area, 73...selection button, 74...decision button, 75...delete button.

Claims

1. a first image capture unit configured to capture one or more gesture images, each of the gesture images depicting a user's hand making a gesture arbitrarily determined by the user; and a data generation unit configured to generate reference data for the user by linking a gesture captured in each of the one or more gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; and a second image capture unit configured to capture an authentication image, the authentication image including the user's hand making a gesture; and an authentication processing unit configured to perform biometric authentication and authentication of the gestures captured in the authentication image by comparing the authentication image with the one or more gesture images; an instruction output unit configured to output the one instruction associated with the authenticated gesture; An authentication system comprising:

2. the plurality of instructions include a plurality of payment instructions respectively corresponding to a plurality of different payment methods; The plurality of payment methods include at least one of electronic money payment, cash payment, credit card payment, and point payment. The authentication system of claim 1 .

3. The plurality of instructions include at least one of presenting a point card, using points, using a coupon, or charging electronic money. The authentication system of claim 1 .

4. The biometric authentication is vein authentication based on the vein pattern of the palm or the back of the hand. The authentication system of claim 1 .

5. The one or more gesture images include a gesture video that shows a gesture in motion. The authentication system of claim 1 .

6. At least one of the authentication image and the one or more gesture images is a series of images including a plurality of images capturing one gesture from a plurality of different directions, or is a three-dimensional image. The authentication system of claim 1 .

7. The control unit further includes: The method is configured to determine whether biometric data for the biometric authentication can be acquired from the hand captured in each of the one or more gesture images, When it is determined that the biometric data cannot be acquired, the determination result is notified to the user. An authentication system according to any one of claims 1 to 6.

8. The system further includes one or more information acquisition units, a gesture data acquisition unit configured to acquire gesture data for identifying the gesture from each of the one or more gesture images; a biometric data acquisition unit configured to acquire biometric data for the biometric authentication from at least a portion of the one or more gesture images; Including, An authentication system according to any one of claims 1 to 6.

9. the one or more gesture images include a standard image including both the biometric data and the gesture data, and an auxiliary image including only the gesture data of the biometric data and the gesture data; The authentication system of claim 8.

10. the plurality of instructions are instructions issued by the user to execute a series of processes associated with payment, The instruction associated with the gesture shown in the standard image is an initial instruction that should be issued at the beginning of the series of processes, The instruction associated with the gesture shown in the auxiliary image is an instruction to be issued after the initial instruction. The authentication system of claim 9.

11. a signal emitting device configured to emit a signal within the store; an authentication server; Further provided with The authentication server The mobile terminal carried by the user is configured to receive user information for identifying the user when the mobile terminal receives the signal, and then, configured to authenticate the user based on the user information; An authentication system according to any one of claims 1 to 6.

12. acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture of the user's choosing; Associating a gesture captured in each of the one or more gesture images with one instruction selected by the user from among a plurality of instructions; acquiring an authentication image, the authentication image including the user's hand making a gesture; and performing biometric authentication and authenticating the gestures captured in the authentication image by comparing the authentication image with the one or more gesture images; outputting the first instruction associated with the authenticated gesture; authentication methods, including

13. 1. A registration device for registering one or more gesture images, comprising: one or more processors; one or more memories storing a registration program; Equipped with the registration program is configured to be executed by the one or more processors; The registered program: acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture of the user's choosing; generating reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; a registration device including instructions for:

14. 1. A method for registering one or more gesture images in a database, comprising: acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture of the user's choosing; generating reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; storing the reference data in the database; How to register, including:

15. A program including a plurality of instructions, the plurality of instructions comprising: acquiring one or more gesture images, each of the gesture images showing the user's hand making a gesture of the user's choosing; generating reference data by linking the gesture depicted in each of the gesture images with one instruction selected by the user from among a plurality of instructions, the reference data including the one or more gesture images; A program containing instructions.

16. The plurality of instructions further comprises: acquiring an authentication image, the authentication image including the user's hand making a gesture; and performing biometric authentication and authenticating the gestures captured in the authentication image by comparing the authentication image with the one or more gesture images; outputting the first instruction associated with the authenticated gesture; 16. The program of claim 15, comprising the instructions:

Images