Information processor, information processing method, and program

Abstract

To improve the accuracy of face authentication.SOLUTION: An information processor includes a controller which performs: acquiring user identification information that is identification information of a user of a vehicle; acquiring an authentication face image that is an image associated with the user identification information and is an image of the face for authentication of the user of the vehicle; acquiring a user's face image that is an image generated by photographing the user of the vehicle with a camera disposed outside the vehicle; performing face authentication on the basis of a comparison between the authentication face image and the user's face image; and acquiring, from the user's face image, first information that is information used in the next and subsequent face authentications in response to the face authentication not being successful.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] It is known that image data of the faces of workers who are permitted to enter a security area is stored in advance as registration information and a face authentication process is performed (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-137959 A Summary of the Invention [Problem to be solved by the invention]

[0004] An object of the present disclosure is to improve the accuracy of face recognition. [Means for solving the problem]

[0005] One aspect of the present disclosure is Obtaining user identification information, which is identification information of a user of the vehicle; acquiring an authentication face image associated with the user identification information, the authentication face image being an authentication face image of a user of the vehicle; acquiring a user face image, the user face image being an image generated by photographing a user of the vehicle with a camera disposed outside the vehicle; performing face authentication based on a comparison between the authentication face image and the user face image; acquiring, in response to the failure of the face authentication, first information to be used in subsequent face authentication from the user face image; The information processing device is provided with a control unit that executes the above.

[0006] One aspect of the present disclosure is The computer Obtaining user identification information, which is identification information of a user of the vehicle; acquiring an authentication face image associated with the user identification information, the authentication face image being an authentication face image of a user of the vehicle; acquiring a user face image, the user face image being an image generated by photographing a user of the vehicle with a camera disposed outside the vehicle; performing face authentication based on a comparison between the authentication face image and the user face image; acquiring, in response to the failure of the face authentication, first information to be used in subsequent face authentication from the user face image; It is an information processing method for performing the above.

[0007] One aspect of the present disclosure is On the computer, Obtaining user identification information, which is identification information of a user of the vehicle; acquiring an authentication face image associated with the user identification information, the authentication face image being an authentication face image of a user of the vehicle; acquiring a user face image, the user face image being an image generated by photographing a user of the vehicle with a camera disposed outside the vehicle; performing face authentication based on a comparison between the authentication face image and the user face image; acquiring, in response to the failure of the face authentication, first information to be used in subsequent face authentication from the user face image; This is a program for executing the above.

[0008] Another aspect of the present disclosure is a storage medium that non-temporarily stores the above-described program. Effect of the Invention

[0009] According to the present disclosure, the accuracy of face authentication can be improved. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram showing a schematic configuration of a system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an overview of processing of a system according to an embodiment. [Diagram 3] A block diagram showing an example of the hardware configuration of each of the center server, the user server, the association management server, the vehicle ID server, and the payment server that constitute the system according to the embodiment. [Figure 4] 2 is a block diagram illustrating an example of a hardware configuration of each of a user terminal and an in-vehicle terminal according to the present embodiment. FIG. [Diagram 5] FIG. 2 is a diagram illustrating an example of a software configuration of a center server according to an embodiment. [Figure 6] 3A to 3C are diagrams illustrating an example of table configurations of a user information DB and an authentication face image DB according to the embodiment. [Figure 7] A diagram illustrating an example of the software configuration of a user server according to an embodiment. [Figure 8] 3A to 3C are diagrams illustrating an example of table configurations of a first information DB and a second authentication information DB according to the embodiment. [Figure 9] FIG. 2 is a diagram illustrating an example of a software configuration of an in-vehicle device according to an embodiment. [Figure 10] FIG. 2 is a diagram illustrating an example of a software configuration of a user terminal according to an embodiment. [Figure 11] 13 is a flowchart of a user registration process in the center server according to the embodiment. [Figure 12] 11 is a flowchart of a process for providing an authentication face image in the center server according to the embodiment. [Figure 13] 11 is a flowchart of a face authentication process in a user server according to an embodiment. [Figure 14] 2 is a diagram showing an example of a table configuration of a first information DB according to the embodiment. FIG. [Figure 15]11 is a flowchart of a face authentication process in a user server according to an embodiment. [Figure 16] 2 is a diagram showing an example of a table configuration of a first information DB according to the embodiment. FIG. [Figure 17] 11 is a flowchart of a face authentication process in a user server according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] According to the face recognition system, even if the subject is far from the system, the subject can be recognized as long as the subject's face can be photographed by a camera. Therefore, the face recognition makes it possible to recognize the subject simply and quickly. However, the conventional face recognition system has the following problems.

[0012] In recent years, with the advancement of machine learning technology, the number of situations in which face recognition can be implemented has increased. In addition, for example, in drive-throughs, and other situations where authentication of a subject while in a vehicle is recommended is increasing, and this has led to an increased need for face authentication. Therefore, it is assumed that a subject who is in a vehicle is photographed from outside the vehicle, and the subject is authenticated by the obtained face image. It is assumed that the system is configured to perform face authentication by comparing a face image for authentication obtained in advance with a face image obtained at the time of authentication. In this case, basically, the face image for authentication is obtained by photographing the subject at any time when the subject is not in the vehicle (for example, the subject photographs himself / herself with a mobile terminal). On the other hand, the face image obtained at the time of authentication reflects, for example, components constituting the vehicle, such as the windshield and side window. In one example, by photographing the subject through the windshield, the face of the subject may appear in a color different from the actual color in the obtained face image due to the influence of the color of the windshield. As a result, the subject's appearance in the face image obtained at the time of authentication may be significantly different from the subject's appearance in the face image for authentication, and face authentication may not be established correctly. The present disclosure solves such a problem.

[0013] Therefore, an information processing device which is one aspect of the present disclosure includes a control unit which executes the following operations: acquiring user identification information which is identification information of a user of a vehicle; acquiring an authentication face image which is an image associated with the user identification information and is an image of the face of the user of the vehicle for authentication; acquiring a user face image which is an image generated by photographing the user of the vehicle with a camera arranged outside the vehicle; performing face authentication based on a comparison between the authentication face image and the user face image; and, if the face authentication is not successful, acquiring first information from the user face image which is information to be used in the next or subsequent face authentication.

[0014] The user of the vehicle is a user who is linked to the vehicle. The user may be the owner of the vehicle. The user identification information is information that can identify the user, and may be, for example, information about a user ID that is assigned in advance. As another example, the user identification information may be information about a driver's license number. The authentication face image is an image of the user's face that has been photographed in advance. The authentication face image may be, for example, an image of the user's face that has been photographed by the user himself using a camera attached to the user's terminal. The authentication face image may also be registered in advance in association with the user identification information.

[0015] Furthermore, the control unit acquires a user face image, which is an image generated by photographing the user of the vehicle with a camera arranged outside the vehicle. The user face image is, for example, a face image of the user at the current time. Then, the control unit performs face authentication based on a comparison between the authentication face image and the user face image. At this time, even if the user is a legitimate user, face authentication may not be established. This may be caused, for example, by a cause depending on the attributes of the user, a cause depending on the attributes of the vehicle, and a cause depending on the system that performs face authentication. Causes depending on the attributes of the user are, for example, causes related to the user, such as the user wearing sunglasses or glasses, the user wearing a mask, the user wearing a hat, or the user having a beard. Causes depending on the attributes of the vehicle are, for example, causes related to the vehicle, such as the height of the vehicle (vehicle height), the size of the vehicle, the color of the windshield, the color of the side windows, the position of the wipers, the position of the sun visors, or the position of the A-pillars (pillars on both sides of the windshield). In addition, the causes depending on the system attributes are, for example, system-related causes such as the height from the ground to the camera, the distance from the camera to the vehicle, the angle of view of the camera, etc. Due to these influences, the user's face shown in the user face image may deviate from the user's face shown in the authentication face image. If this happens, there is a risk that face authentication will not be established.

[0016] Therefore, in response to the failure of face authentication, the control unit acquires, from the user's face image, first information to be used in subsequent face authentications. The first information includes information about the cause of the failure of face authentication. By using the acquired information in the next and subsequent face authentication, it is possible to prevent face authentication from failing even when the user is a legitimate user, thereby improving the accuracy of face authentication in similar situations.

[0017] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The configurations of the following embodiments are merely examples, and the present disclosure is not limited to the configurations of the embodiments.

[0018] First Embodiment Fig. 1 is a diagram showing a schematic configuration of a system 1 according to an embodiment. In the example of Fig. 1, the system 1 includes an in-vehicle device 100 mounted on a vehicle 10, a user terminal 20 carried by a user of the vehicle 10, a center server 30, a user-side server 40, a linking management server 50, a vehicle ID server 60, and a payment server 70. A camera 400 that captures images of the inside of the vehicle 10 from outside the vehicle 10 and a keypad on which a user inputs a number are connected to the user-side server 40.

[0019] The in-vehicle device 100, the user terminal 20, the center server 30, the user server 40, the association management server 50, the vehicle ID server 60, and the payment server 70 are connected to each other by a network N1. The network N1 is, for example, a global public communication network such as the Internet, and may be a Wide Area Network (WAN) or other communication network. The network N1 may also include a telephone communication network such as a mobile phone, or a wireless communication network such as Wi-Fi (registered trademark). Although FIG. 1 shows one vehicle 10 and one user terminal 20 as an example, there may be a plurality of these. The in-vehicle device 100 and the user server 40 or the camera 400 may be capable of short-distance wireless communication.

[0020] The in-vehicle device 100 is a device that provides information about the vehicle 10 to the user side server 40. The user terminal 20 is a terminal that can provide a facial image of the user. The center server 30 is a server that generates a facial image for authentication and provides it to the user side server 40. The user side server 40 is a server that performs facial authentication. The linking management server 50 is a server that manages the linking between the vehicle 10 and the user. The vehicle ID server 60 is a server that manages information about the vehicle 10. The payment server 70 is a server that performs payment.

[0021] FIG. 2 is a diagram showing an overview of the processing of the system 1 according to the embodiment. P1 to P5 are processing related to the center server 30, and U1 to U8 are processing related to the user-side server 40. In P1, a user uses the user terminal 20 to perform user registration in the center server 30. The user requests user registration to the center server 30 via the user terminal 20. At this time, a face image linked to the user ID is transmitted to the center server 30. This face image is also referred to as an authentication face image below. The authentication face image is, for example, an image of the user's face photographed by the user terminal 20. However, the authentication face image is not limited to this, and may be an image of the user's face photographed in the past and stored in the user terminal 20 or an external server. The user ID may be, for example, a number assigned by a common number system for identifying individuals in each country (for example, a social security number, or a citizen identification number) or a driver's license number. The user ID is an example of user identification information. Furthermore, at the time of user registration, payment-related information is linked to the user ID and transmitted from the user terminal 20 to the center server 30. The payment-related information is information required when the user uses a payment service, and includes, for example, a credit card number or a bank account number.

[0022] At P2, the center server 30 acquires and stores the user ID, the face image for authentication, and information related to payment from the user terminal 20. At P3, the center server 30 requests the association management server 50 to transmit the vehicle ID associated with the user ID. The association management server 50 executes a search by the user ID to extract the vehicle ID associated with the user ID. Then, the center server 30 obtains the vehicle ID linked to the user ID from the linking management server 50. The vehicle ID is a number unique to the vehicle 10, such as the vehicle body number or the number displayed on the vehicle registration number plate. Hereinafter, this number will also be referred to as the vehicle number.

[0023] At P4, the center server 30 requests the vehicle ID server 60 to transmit vehicle information linked to the vehicle ID. The vehicle ID server 60 executes a search using the vehicle ID to extract vehicle information linked to the vehicle ID. Then, the center server 30 acquires the vehicle information linked to the vehicle ID from the vehicle ID server 60. The vehicle information includes information capable of identifying the vehicle, such as information on the vehicle model, vehicle name, vehicle number, or vehicle registration plate number (vehicle number). The vehicle information may include information capable of identifying the vehicle's height, size, windshield color, side window color, wiper position, sun visor position, or A-pillar position. The vehicle information may also include information on the owner of the vehicle 10.

[0024] At P5, the center server 30 stores the vehicle information acquired from the vehicle ID server 60. At this time, attributes of the vehicle 10 may be acquired from the vehicle information and stored. For example, the color of the windshield may be acquired as an attribute of the vehicle 10 from the vehicle information.

[0025] Also, at U1, the utilization side server 40 acquires the vehicle ID and a facial image of the user. The facial image of the user is acquired by the camera 400 photographing the vehicle 10 from the outside. The facial image acquired in this manner is hereinafter also referred to as a user facial image. The vehicle ID may be acquired from the in-vehicle device 100 by communication. As another example, if the vehicle ID corresponds to the vehicle number, the vehicle number may be acquired by analyzing the image of the vehicle 10 photographed by the camera 400.

[0026] At U2, the usage server 40 queries the association management server 50 as to whether or not a linking relationship corresponding to the vehicle ID exists. At this time, the query may be made as to whether or not a valid linking relationship exists at the target date and time. The target date and time may be the date and time at the time of the query (which may be the present), or may be a past date and time. A valid linking is, for example, a linking that is within the expiration date if the linking between the vehicle ID and the user ID has an expiration date. If a valid linking exists, the user ID linked to the vehicle ID is transmitted from the association management server 50 to the usage server 40.

[0027] In U3, the utilization server 40 requests the vehicle ID server 60 to transmit vehicle information linked to the vehicle ID. The vehicle ID server 60 executes a search using the vehicle ID to extract vehicle information linked to the vehicle ID. The utilization server 40 then acquires the vehicle information linked to the vehicle ID from the vehicle ID server 60. The vehicle information includes information that can identify the vehicle 10, such as information about the vehicle model, vehicle name, vehicle number, or vehicle registration plate number (vehicle number). The vehicle information may include information that can identify the height, size, windshield color, side window color, wiper position, sun visor position, or A-pillar position of the vehicle 10. The vehicle information may also include information about the owner of the vehicle 10.

[0028] In U4, if a valid linking relationship exists, the use server 40 requests the center server 30 to transmit an authentication facial image corresponding to the user ID. The center server 30 transmits the authentication facial image corresponding to the user ID to the use server 40. At this time, the center server 30 also transmits information on the various authorities linked to the user ID to the use server 40. The various authorities are, for example, authorities for the payment service registered by the user. Note that the authorities are not limited to this and may be, for example, authorities for administrative services. The authentication face image may be stored in advance in the user-side server 40.

[0029] At U5, the user server 40 performs face authentication by comparing the user face image captured by the camera 400 with the face image for authentication. Here, a known face authentication technology can be used. The face authentication method is not limited. Image processing may be performed by any analysis process (pattern matching, edge extraction, etc.). A trained machine learning model may be used for image processing.

[0030] In U6, if face authentication is successful, the user server 40 executes the user's authority. For example, the user server 40 requests payment from the payment server 70. For example, the payment can be used for payment at a drive-through in a restaurant, payment for parking fees, or payment for toll road fees.

[0031] On the other hand, in U7, if face authentication is not established, the use server 40 performs the second authentication. For example, the use server 40 may perform the second authentication by means of inputting a password to the keypad 500, fingerprint authentication, voiceprint authentication, or the like. In this case, for example, the user may be notified to input a PIN number to the keypad 500 arranged near the stopping position of the vehicle 10. This notification may be made, for example, by playing a voice or displaying it on a signage screen. Also, a device that reads the user's fingerprint may be arranged near the stopping position of the vehicle 10, and the user may be notified to input the fingerprint using the device. Also, a microphone that inputs the user's voice may be arranged near the stopping position of the vehicle 10, and the user may be notified to input the voice into the microphone. As another example, the use server 40 may transmit a command to the user terminal 20 or the in-vehicle device 100 to acquire a password, fingerprint, or voice from the user terminal 20 or the in-vehicle device 100. The usage server 40 may then obtain a password, a fingerprint, or a voice from the user terminal 20 or the in-vehicle device 100. As another example, the usage server 40 may perform SMS authentication using the user terminal 20, authentication using a telephone call, or authentication using an e-mail. The utilization side server 40 authenticates the user through the second authentication, which is the other authentication than the above.

[0032] At U8, if the second authentication is successful, the use server 40 generates and stores first information from the user's face image. The use server 40 extracts, for example, the color of the windshield of the vehicle 10, and stores information related to the color of the windshield as the first information. The use server 40 may link the first information with the user ID and transmit it to the center server 30, where the first information is stored. After that, the use server 40 executes the user's authority in the same manner as in U6.

[0033] If the first information already exists in the use server 40, in U5, the use server 40 generates an edited authentication face image by reflecting the first information in the authentication face image. Then, the use server 40 performs face authentication by comparing the user face image captured by the camera 400 with the edited authentication face image.

[0034] The processed authentication face image may be generated by either the center server 30 or the use server 40. The processed authentication face image may be stored in either the center server 30 or the use server 40. The processed authentication face image may be generated before face authentication is performed. For example, in response to the use server 40 or the center server 30 acquiring the first information, the processed authentication face image may be generated and stored in the use server 40 or the center server 30. In addition, when one user owns multiple vehicles 10, the use server 40 or the center server 30 may generate and store the first information or the processed authentication face image for each vehicle 10.

[0035] In addition, the first information reflected when generating the processed authentication face image may be the color of the windshield or side window, or may be information depending on the attributes of the vehicle 10, such as the position and shape of the wiper, sun visor, A-pillar, or front seat, or the vehicle height. In addition, as another example, the first information may be information depending on the attributes of the user, or information depending on the attributes of the system 1. Information depending on the attributes of the user is, for example, information related to the user, such as whether the user is wearing sunglasses or glasses, whether the user is wearing a mask, whether the user is wearing a hat, or whether the user has a beard. Information depending on the attributes of the system is, for example, information related to the system, such as the height from the ground to the camera, the distance from the camera to the vehicle, and the angle of view of the camera. Any other information that has an effect on face authentication can be used as the first information. The processed authentication face image may be generated by reflecting the first information. For example, the user-side server 40 may process the authentication face image to add the wiper, sun visor, A-pillar, or front seat to generate the processed authentication face image.

[0036] Next, Figure 3 is a block diagram that shows an example of the hardware configuration of each of the center server 30, the user server 40, the linking management server 50, the vehicle ID server 60, and the payment server 70 that constitute the system 1 of the embodiment.

[0037] The center server 30 is configured as a computer having a control unit 31, a storage unit 32, and a communication module 33. The center server 30 can be configured as a computer having a processor (CPU, GPU, etc.), a main storage unit (RAM, ROM, etc.), and an auxiliary storage unit (EPROM, hard disk drive, removable media, etc.). The auxiliary storage unit stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, it is possible to realize various functions (software modules) that match a predetermined purpose, as described below. However, some or all of the modules may be realized as hardware modules by hardware circuits such as ASICs and FPGAs.

[0038] The control unit 31 is a calculation unit that executes a predetermined program to realize various functions of the center server 30. The control unit 31 can be realized by a hardware processor such as a CPU. The control unit 31 may also be configured to include a RAM, a ROM (Read Only Memory), a cache memory, and the like.

[0039] The storage unit 32 is a means for storing information, and is configured with storage media such as RAM, a magnetic disk, and a flash memory. The storage unit 32 stores programs executed by the control unit 31, data used by the programs, and the like. The storage unit 32 also stores various types of information. Details will be described later.

[0040] The communication module 33 is a communication interface for connecting the center server 30 to a network. The communication module 33 may be configured to include, for example, a network interface board, a wireless communication interface for wireless communication, etc. The center server 30 can perform data communication with other computers (for example, other server devices or each user terminal 20) via the communication module 33.

[0041] The specific hardware configuration of the center server 30 may include omissions, substitutions, and additions of components as appropriate depending on the embodiment. For example, the control unit 31 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, GPU, or the like. The center server 30 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same.

[0042] The use server 40 is configured as a computer having a control unit 41, a storage unit 42, and a communication module 43. The use server 40 can be configured as a computer having a processor (CPU, GPU, etc.), a main storage unit (RAM, ROM, etc.), and an auxiliary storage unit (EPROM, hard disk drive, removable media, etc.). The auxiliary storage unit stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, it is possible to realize each function (software module) that matches a predetermined purpose, as described below. However, some or all of the modules may be realized as a hardware module by a hardware circuit such as ASIC or FPGA. The control unit 41, storage unit 42, and communication module 43 of the use server 40 are configured similarly to the control unit 31, storage unit 32, and communication module 33 of the center server 30, so their explanations are omitted.

[0043] A camera 400 and a keypad 500 are connected to the user server 40, and are controlled by a control unit 41. The camera 400 is arranged in advance at a position where it can capture an image of the driver of the stopped vehicle 10. The camera 400 is a device that captures images using an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The keypad 500 is arranged in advance near the stopping position of the vehicle 10 so that the driver of the vehicle 10 can operate it. The keypad 500 is configured to be able to input, for example, numbers from 0 to 9. The keypad 500 may also have a display that displays the input numbers. The keypad 500 may also include a device for inputting, such as a mouse, a keyboard, or a microphone.

[0044] The association management server 50, the vehicle ID server 60, and the settlement server 70 can also be configured as a computer similar to the center server 30. The association management server 50 is a device that provides information on the association between the vehicle ID and the user ID, the timestamp of the association, and the expiration date of the association via the network N1. Therefore, the association management server 50 stores information on the vehicle ID, the user ID, the timestamp, and the expiration date in association with each other. The timestamp is, for example, the date and time when the user ID and the vehicle ID are associated with each other. The expiration date is the expiration date of the association between the user ID and the vehicle ID. The association management server 50 is managed by a business operator or a government agency that is neutral to the administrator of the center server 30 and the administrator of the user side server 40. The information stored in the association management server 50 may be provided by, for example, the user via the user terminal 20, or may be provided by the center server 30.

[0045] The vehicle ID server 60 is a device that provides information related to the vehicle number, the vehicle model, and the owner of the vehicle 10, which are linked to the vehicle ID, via the network N1. For this reason, the vehicle ID server 60 stores information related to the vehicle ID, the vehicle number, the vehicle model, and the owner of the vehicle 10, which are linked to each other. The vehicle ID server 60 is managed by a neutral business operator or administrative agency with respect to the administrator of the center server 30 and the administrator of the user side server 40. The vehicle ID server 60 may be, for example, a server managed by an administrative agency that issues license plates. When the user registers the vehicle 10, a license plate is issued, and the number (vehicle number), and information related to the vehicle model and the owner may be stored in the vehicle ID server 60.

[0046] The payment server 70 is a device that performs payment processing based on a payment request. The payment server 70 is managed by, for example, a credit card company. As another example, the payment server 70 may be a server that manages electronic money or points, or a server that manages bank deposits. The payment server 70 stores payment information. The payment information may include, for example, a user ID, It may also include information relating to the link between D and a credit card number.

[0047] Next, FIG. 4 is a block diagram illustrating an example of a hardware configuration of each of the user terminal 20 and the in-vehicle device 100 according to this embodiment.

[0048] The user terminal 20 is a small computer such as a smartphone, a mobile phone, a tablet terminal, a personal information terminal, a wearable computer (such as a smart watch), a personal computer (PC), etc. The user terminal 20 includes a control unit 21, a storage unit 22, a communication module 23, an input / output device 24, and a camera 25.

[0049] The user terminal 20 can be configured as a computer having a processor (CPU, GPU, etc.), a main memory device (RAM, ROM, etc.), and an auxiliary memory device (EPROM, hard disk drive, removable media, etc.) like the center server 30. However, some or all of the functions (software modules) may be realized as hardware modules by hardware circuits such as ASIC, FPGA, etc.

[0050] The control unit 21, memory unit 22, and communication module 23 of the user terminal 20 have the same configuration as the control unit 31, memory unit 32, and communication module 33 of the center server 30, so their explanation will be omitted.

[0051] The input / output device 24 is a means for accepting an input operation performed by the user and presenting information to the user. Specifically, the input / output device 24 includes devices for input such as a mouse, keyboard, or microphone, and devices for output such as a display and speaker. The input / output device 24 may be integrally configured with, for example, a touch panel display. The camera 25 is, for example, a CCD (Charge Coupled Device) image sensor or a CMO Using imaging elements such as S (Complementary Metal Oxide Semiconductor) image sensors It is a device that takes pictures using a microscope.

[0052] As with the center server 30, the specific hardware configuration of the user terminal 20 may be such that components are omitted, replaced, or added as appropriate depending on the embodiment.

[0053] Next, the in-vehicle device 100 is a computer such as an ECU (Electronic Control Unit), a DCM (Data Communication Module), a head unit, or a navigation system mounted on the vehicle 10. The in-vehicle device 100 is configured to include a control unit 11, a storage unit 12, a communication module 13, and an input / output device 14. The control unit 11, the storage unit 12, the communication module 13, and the input / output device 14 of the in-vehicle device 100 are similar to the control unit 21, the storage unit 22, the communication module 23, and the input / output device 24 of the user terminal 20, and therefore description thereof will be omitted.

[0054] 5 is a diagram illustrating an example of a software configuration of the center server 30 according to the embodiment. The control unit 31 is configured to have software modules of a user information acquisition unit 311, a linking information acquisition unit 312, a vehicle information acquisition unit 313, and a face image provision unit 314. Each software module may be realized by the control unit 31 (CPU) executing a program stored in the storage unit 32. Note that the information processing executed by the user information acquisition unit 311, the linking information acquisition unit 312, the vehicle information acquisition unit 313, and the face image provision unit 314 described below is synonymous with the information processing executed by the control unit 31.

[0055] The storage unit 32 of the center server 30 stores a user information database 321 (hereinafter, referred to as user information DB 321) and an authentication face image database 322 (hereinafter, referred to as authentication face image DB 322). B322). The user information DB321 and the authentication face image DB322 are constructed by a database management system (DBMS) program executed by a processor managing data stored in the auxiliary storage device. These databases are, for example, relational databases. FIG. 6 is a diagram showing an example of the table configuration of the user information DB321 and the authentication face image DB322 according to the embodiment. Details will be described later.

[0056] The user information acquisition unit 311 is configured to execute a process of receiving the user information and the authentication face image transmitted from the user terminal 20, and storing the user information and the authentication face image in the user information DB 321 and the authentication face image DB 322. As shown in FIG. 6, the user information DB 321 stores a user ID, a user's name, a user's contact information, and information related to a payment service. The user information is linked to the user ID. The user ID may be, for example, a driver's license number. The user's contact information may include the user's address, telephone number, SNS account, email address, and the like. The information related to the payment service is information related to the link between the user and the payment service, and includes information related to the payment desired by the user at the time of payment (for example, a credit card number). The user information is registered in advance in the center server 30 by the user via the user terminal 20. As shown in FIG. 6, the authentication face image DB 322 stores the authentication face image linked to the user ID. The authentication face image is a face image of the user transmitted from the user terminal 20 together with the user information, for example, a face image captured by the user himself / herself with the camera 25 of the user terminal 20. The user information acquisition unit 311 associates the authentication face image received from the user terminal 20 with the user ID and stores it in the authentication face image DB 322. As another example, the authentication face image DB 322 may store information regarding the location where the authentication face image is stored.

[0057] The association information acquisition unit 312 requests the association management server 50 to search by the user ID. Then, the association management server 50 returns information (hereinafter also referred to as association information) regarding the vehicle ID, timestamp, and expiration date associated with the user ID to the center server 30. Note that the association management server 50 may periodically update the association between the user ID and the vehicle ID. The association information acquisition unit 312 stores the association information acquired from the association management server 50 in the memory unit 32.

[0058] The vehicle information acquisition unit 313 requests the vehicle ID server 60 to search by the vehicle ID. This vehicle ID is the vehicle ID acquired from the association management server 50. The vehicle ID server 60 then returns information on the vehicle number, the vehicle model, and the owner of the vehicle 10 that are associated with the vehicle ID to the center server 30. The vehicle models are classified, for example, so that the attributes of the vehicle 10 can be determined. The information on the owner of the vehicle 10 includes, for example, information on a user ID or a name. The vehicle information acquisition unit 313 stores, in the storage unit 32, information on the vehicle number, the vehicle model, and the owner of the vehicle 10 that are associated with the vehicle ID acquired from the vehicle ID server 60.

[0059] The facial image providing unit 314 provides an authentication facial image in response to a request from the use-side server 40. The request from the use-side server 40 includes information related to the user ID or vehicle ID, and the facial image providing unit 314 extracts an authentication facial image corresponding to the user ID or vehicle ID from the authentication facial image DB 322 and transmits it to the use-side server 40.

[0060] 7 is a diagram showing an example of the software configuration of the user server 40 according to the embodiment. The control unit 41 is configured to have the following software modules: a face image acquisition unit 411, a vehicle information acquisition unit 412, a user ID acquisition unit 413, an authentication information acquisition unit 414, an execution unit 415, a second authentication unit 416, a first information generation unit 417, and a face image processing unit 418. The wear module may be realized by causing the control unit 41 (CPU) to execute a program stored in the storage unit 42. Note that the information processing executed by the facial image acquisition unit 411, the vehicle information acquisition unit 412, the user ID acquisition unit 413, the authentication information acquisition unit 414, the execution unit 415, the second authentication unit 416, the first information generation unit 417, and the facial image processing unit 418 described below is synonymous with the information processing executed by the control unit 41.

[0061] The memory unit 42 of the user server 40 stores a first information database 421 (hereinafter referred to as the first information DB421) and a second authentication information database 423 (hereinafter referred to as the second authentication information DB423). The first information DB421 and the second authentication information DB423 are constructed by a program of a database management system (DBMS) executed by a processor managing data stored in an auxiliary storage device. These databases are, for example, relational databases. FIG. 8 is a diagram showing an example of a table configuration of the first information DB421 and the second authentication information DB423 according to the embodiment. Details will be described later.

[0062] The face image acquisition unit 411 acquires an image captured by the camera 400 when the vehicle 10 is present at a predetermined position. The camera 400 is arranged in advance at a position where the driver of the vehicle 10 can be photographed. The face image acquisition unit 411, for example, constantly monitors images captured by the camera 400 and detects by image analysis that the vehicle 10 has approached a predetermined position. Then, the image of the user captured at that time is acquired as a user face image.

[0063] The vehicle information acquisition unit 412 acquires information about the vehicle 10 that is present at a predetermined position. At this time, for example, the camera 400 may be provided with a wireless communication function, and wireless communication may be performed between the camera 400 and the vehicle 10 to acquire information about the vehicle 10 (for example, the vehicle number). As another example, the vehicle number may be read by analyzing an image of the vehicle 10 captured by the camera 400.

[0064] The user ID acquisition unit 413 requests the association management server 50 to search by the vehicle ID. At this time, the user ID acquisition unit 413 inquires of the association management server 50 as to whether or not a valid association relationship corresponding to the vehicle ID exists at the target date and time. If a valid association relationship corresponding to the vehicle ID exists at the target date and time, the association management server 50 returns the user ID associated with the vehicle ID to the use-side server 40. The user ID acquisition unit 413 stores the user ID acquired from the association management server 50 in the storage unit 42. The user ID acquisition unit 413 may acquire information related to the user ID from the in-vehicle device 100 or the user terminal 20.

[0065] The authentication information acquisition unit 414 requests the center server 30 to search by the user ID. The center server 30 returns information on various authorities linked to the user ID and a face image for authentication to the user server 40. The various authorities include, for example, information on the payment service linked to the user ID.

[0066] If face authentication is successful and the authority is valid, the execution unit 415 requests payment from the payment server 70. The execution unit 415 performs face authentication by comparing the authentication face image acquired from the center server 30 or the processed authentication face image generated by the face image processing unit 418 described later with the user face image acquired from the camera 400. If the first information exists, the execution unit 415 performs face authentication based on the processed authentication face image, and if the first information does not exist, the execution unit 415 performs face authentication based on the authentication face image. The payment server 70 to which the payment is requested is selected by the execution unit 415 based on information on various authorities linked to the user ID.

[0067] If face authentication is not successful, the second authentication unit 416 performs second authentication other than face authentication. The second authentication is authentication by a method other than face authentication. The second authentication unit 416 may notify the user to input a PIN number into the keypad 500, for example. The second authentication unit 416 performs authentication by, for example, comparing the number input into the keypad 500 with the number stored in the second authentication information DB 324 of the storage unit 32. As shown in FIG. 8, the second authentication information DB stores the second authentication information in association with the user ID. Note that the second authentication is not limited to this, and may be authentication by a method other than face authentication, such as fingerprint authentication, voiceprint authentication, SMS authentication, or authentication by telephone. If the authentication by the second authentication unit 416 is established and the authority is valid, the execution unit 415 requests the payment server 70 to make a payment.

[0068] The first information generating unit 417 generates the first information when the face authentication is not established and the second authentication is established. The first information is, for example, information about the color of the windshield of the vehicle 10. The first information generating unit 417 extracts information about the color of the windshield from the user face image using a known technology to generate the first information. Then, the first information generating unit 417 associates the generated first information with the user ID and the vehicle ID and stores it in the first information DB 421. As shown in FIG. 8, the first information DB 421 stores information about the user ID, the vehicle ID, and the first information. When the same user owns multiple vehicles 10, there are multiple records of the same user ID and different vehicle IDs. The first information may be generated by the control unit 31 of the center server 30 instead of the first information generating unit 417 of the utilization side server 40. In this case, in response to the failure of the face authentication, the user face image is associated with the user ID and the vehicle ID and transmitted to the center server 30. As another example, the first information generating unit 417 may generate first information from the user face image and transmit the first information to the center server 30. Then, the first information may be stored in the storage unit 32 of the center server 30.

[0069] The face image processing unit 418 processes the authentication face image acquired from the center server 30 according to the first information, thereby generating a processed authentication face image. As shown in FIG. 8, the first information DB 421 stores the relationship between the user ID, the vehicle ID, and the first information. When the face image processing unit 418 acquires the authentication face image, it extracts the first information corresponding to the user ID and the vehicle ID from the first information DB 421. Then, the processed authentication face image is generated by reflecting the first information on the authentication face image. For example, when the color of the windshield of the vehicle 10 is stored as the first information in the first information DB 421, the face image processing unit 418 generates the processed authentication face image by reflecting the color of the windshield of the vehicle 10 on the authentication face image. That is, the processed authentication face image is generated so as to correspond to an image of the user's face photographed from outside the vehicle 10. The face image processing unit 418 associates the generated processed authentication face image with the vehicle ID and the user ID, and stores it in the storage unit 42.

[0070] 9 is a diagram showing an example of a software configuration of the in-vehicle device 100 according to the embodiment. In this embodiment, the control unit 11 of the in-vehicle device 100 is configured to have a software module of a vehicle ID transmission unit 111. The software module may be realized by causing the control unit 11 (CPU) to execute a program stored in the storage unit 12. Note that the information processing executed by the vehicle ID transmission unit 111 below is synonymous with the information processing executed by the control unit 11.

[0071] In response to a request from the utilization server 40 or the camera 400, the vehicle ID transmission unit 111 transmits the vehicle ID stored in the storage unit 12 to the utilization server 40 or the camera 400. Note that if the vehicle information acquisition unit 412 of the utilization server 40 reads and acquires the vehicle number, the vehicle ID transmission unit 111 can be omitted.

[0072] 10 is a diagram illustrating an example of a software configuration of the user terminal 20 according to the embodiment. The control unit 21 of the user terminal 20 controls the image capturing unit 211 and the user information transmitting unit 212. The system is configured with software modules. Each software module may be realized by causing the control unit 21 (CPU) to execute a program stored in the storage unit 22. Note that the information processing executed by the image capture unit 211 and the user information transmission unit 212 described below is synonymous with the information processing executed by the control unit 21.

[0073] The photographing unit 211 photographs the user with the camera 25. The photographing unit 211 photographs the user in response to a predetermined input from the user to the input / output device 24. At this time, the user adjusts the angle of the camera 25 so that the user's own face is photographed.

[0074] The user information transmission unit 212 transmits the user information input by the user via the input / output device 24 and the user face image captured by the photographing unit 211 to the center server 30. In addition, when the second authentication is SMS authentication or authentication by telephone, the user information transmission unit 212 may transmit the information input by the user to the input / output device 24 in the SMS authentication or authentication by telephone to the utilization side server 40.

[0075] Next, the user registration process in the center server 30 will be described. Fig. 11 is a flowchart of the user registration process in the center server 30 according to the embodiment. The process shown in Fig. 11 is executed at predetermined time intervals in the center server 30. Note that the process shown in Fig. 11 is executed by each software module of the control unit 31, but the following flowchart will be described assuming that the control unit 31 executes the process.

[0076] In step S101, the control unit 31 determines whether or not a user registration request has been received from the user terminal 20. An application program capable of performing user registration may be installed in the user terminal 20. The user performs a predetermined input using this application program and photographs his / her face, whereby the user terminal 20 transmits a user registration request to the center server 30. As another example, the user terminal 20 may access a web page capable of performing user registration, and the user may perform a predetermined input on the web page or upload a face image for authentication, whereby the user terminal 20 transmits a user registration request to the center server 30. If the determination in step S101 is affirmative, the process proceeds to step S102, and if the determination is negative, the routine is terminated.

[0077] In step S102, the control unit 31 acquires user information and an authentication facial image. The user information and the authentication facial image are transmitted from the user terminal 20 to the center server 30 together with a request for user registration, so the control unit 31 acquires the user information and the authentication facial image. The user information includes a user ID. Then, in step S103, the control unit 31 performs a user registration process, stores the user information in the user information DB 321, and stores the authentication facial image in the authentication facial image DB 322 in association with the user ID.

[0078] Next, the process of providing a facial image for authentication in the center server 30 will be described. Fig. 12 is a flowchart of the process of providing a facial image for authentication in the center server 30 according to the embodiment. The process shown in Fig. 12 is executed at predetermined time intervals in the center server 30. Note that the process shown in Fig. 12 is executed by each software module of the control unit 31, but the following flowchart will be described assuming that the control unit 31 executes the process.

[0079] In step S201, the control unit 31 determines whether or not a search request based on a user ID has been received from the use-side server 40. This request includes a request to transmit information on various authorities linked to the user ID and a face image for authentication to the use-side server 40. If the determination in step S201 is affirmative, the process proceeds to step S202. If the determination is negative, the process End the routine.

[0080] In step S202, the control unit 31 acquires an authentication face image stored in the authentication face image DB 322. The authentication face image acquired at this time is a face image received from the user terminal 20 at the time of user registration. The control unit 31 extracts, from the authentication face image DB 322, an authentication face image associated with the user ID acquired in step S201.

[0081] In step S203, the control unit 31 acquires the user's authority from the user information DB 321. In this embodiment, information on the payment service is acquired as the information on the user's authority. Then, in step S204, the control unit 31 transmits the authentication face image and the information on the user's authority to the use-side server 40. The information on the user's authority transmitted at this time includes, for example, information on the payment company (credit card company) registered by the user, and information on the credit card number.

[0082] Next, the face authentication process in the use server 40 will be described. Fig. 13 is a flowchart of the face authentication process in the use server 40 according to the embodiment. The process shown in Fig. 13 is executed at predetermined time intervals in the use server 40. Note that the process shown in Fig. 13 is executed by each software module of the control unit 41, but the following flowchart will be described assuming that the control unit 41 executes the process.

[0083] In step S301, the control unit 41 determines whether or not the vehicle 10 is present at a predetermined position. The predetermined position is a position where the face of the driver of the vehicle 10 can be photographed by the camera 400. The predetermined position may be, for example, a position where a product is ordered or a position where a product is received in a drive-through. Note that the predetermined position is not limited to a position where the vehicle 10 stops, and may be a position that the vehicle 10 passes while traveling. If a positive determination is made in step S301, the process proceeds to step S302, and if a negative determination is made, the routine ends.

[0084] In step S302, the control unit 41 acquires a user face image by capturing an image of the driver of the vehicle 10 with the camera 400. The control unit 41 stores the acquired user face image in the storage unit 42. In step S303, the control unit 41 acquires a vehicle ID from the vehicle 10. For example, the vehicle ID may be transmitted from the in-vehicle device 100 to the camera 400 using short-range wireless communication. As another example, the license plate of the vehicle 10 may be captured with the camera 400, and the vehicle number may be acquired by image analysis.

[0085] In step S304, the control unit 41 acquires the user ID corresponding to the vehicle ID from the association management server 50. At this time, the control unit 41 inquires of the association management server 50 as to whether or not a valid association corresponding to the vehicle ID exists at the target date and time. In this case, the control unit 41 transmits information on the vehicle ID and the target date and time to the association management server 50. For example, if the user has not registered a user ID, there is no valid association corresponding to the vehicle ID. Therefore, the association management server 50 determines whether or not a user ID corresponding to the vehicle ID exists. In addition, for example, an expiration date may be set for the association between the vehicle ID and the user ID. In this case, if the expiration date of the association between the vehicle ID and the user ID has passed, there is no valid association corresponding to the vehicle ID. Therefore, the association management server 50 determines whether or not the expiration date of the association between the vehicle ID and the user ID is within the expiration date. In addition, for example, when a valid association between the vehicle ID and the user ID is required at a specific date and time, it is determined whether or not a valid association between the vehicle ID and the user ID exists at the specific date and time using a timestamp. If a valid link between the vehicle ID and the user ID exists at the target date and time, the link management server 50 transmits the user ID to the utilization side server 40. On the other hand, if a valid link between the vehicle ID and the user ID does not exist at the target date and time, the link management server 50 transmits information to that effect to the utilization side server 40. The server 50 transmits the vehicle ID to the utilization side server 40. In this case, the control unit 41 cannot obtain the user ID corresponding to the vehicle ID. As another example, the control unit 41 may obtain the user ID from the in-vehicle device 100 or the user terminal 20. For example, the user ID may be registered in advance in the in-vehicle device 100 in association with the vehicle ID, and the user ID may be transmitted to the camera 400 together with the vehicle ID.

[0086] In step S305, the control unit 41 judges whether or not the user ID was acquired in step S304. If the judgment in step S305 is affirmative, the process proceeds to step S306, and if the judgment is negative, the routine is terminated. If the judgment in step S305 is negative, the user may be notified that payment cannot be made. For example, the use-side server 40 may be configured to display such a message on a signage or play a voice message from a speaker.

[0087] In step S306, the control unit 41 acquires an authentication face image and information related to various authorities from the center server 30. At this time, the control unit 41 transmits a search request by user ID to the center server 30. In response to this request, the center server 30 transmits an authentication face image and information related to various authorities corresponding to the user ID to the use-side server 40. Then, the control unit 41 receives the authentication face image and information related to the user's various authorities from the center server 30.

[0088] In step S307, the control unit 41 determines whether or not first information corresponding to the user exists. The control unit 41 accesses the first information DB 421 and determines whether or not a record including a user ID corresponding to the user exists. If this record exists, the control unit 41 determines that first information corresponding to the user exists. On the other hand, if this record does not exist, the control unit 41 determines that first information corresponding to the user does not exist. If a positive determination is made in step S307, the process proceeds to step S308, and if a negative determination is made, the process proceeds to step S310.

[0089] In step S308, the control unit 41 processes the authentication face image using the first information to generate a processed authentication face image. For example, the control unit 41 processes the authentication face image acquired from the center server 30 so that the processed authentication face image becomes a face image of the user photographed through the windshield. A known technique can be used for this processing. The control unit 41 stores the generated processed authentication face image in the storage unit 42.

[0090] In step S309, the control unit 41 performs face authentication by comparing the user face image acquired in step S302 with the processed authentication face image generated in step S308. Meanwhile, in step S310, the control unit 41 performs face authentication by comparing the user face image acquired in step S302 with the authentication face image acquired in step S306. Publicly known techniques can be adopted for these face authentications. The method of face authentication is not particularly limited and may be appropriately determined depending on the embodiment.

[0091] In step S311, the control unit 41 determines whether or not face authentication has been established. If an affirmative determination is made in step S311, the process proceeds to step S315, and if a negative determination is made, the process proceeds to step S312.

[0092] In step S312, the control unit 41 executes the second authentication. For example, the control unit 41 notifies the user to input a PIN into the keypad 500. For example, the use-side server 40 may be configured to display the PIN on a signage screen or play a voice message from a speaker. The control unit 41 then acquires the PIN entered by the user via the keypad 500 and checks the PIN against the user ID stored in the second authentication information DB. D, the second authentication is performed by comparing the PIN number corresponding to D with the PIN number corresponding to the payment method. If the two PIN numbers match, the second authentication is established. Then, in step S313, the control unit 41 determines whether the second authentication is established or not. If the determination in step S313 is affirmative, the process proceeds to step S314, and if the determination is negative, the routine is terminated. Note that if the determination in step S313 is negative, the control unit 41 may inform the user that payment cannot be made.

[0093] In step S314, the control unit 41 stores the first information in the first information DB 421. The first information may be information about, for example, the color of the windshield extracted from the user's face image. As another example, the first information may be the user's face image. The control unit 41 stores the first information in the first information DB 421 in association with the vehicle ID and the user ID.

[0094] In step S315, the control unit 41 determines whether or not the user is able to make a payment at the payment server 70. The control unit 41 determines whether or not the user's authority is valid based on information related to various authorities acquired from the center server 30. In this case, the control unit 41 may inquire of the payment server 70 whether or not the user is able to make a payment at the payment server 70. If a positive determination is made in step S315, the process proceeds to step S316, and if a negative determination is made, the routine is terminated.

[0095] In step S316, the control unit 41 requests the payment server 70 to make a payment. That is, the control unit 41 exercises the authority of the user. At this time, the control unit 41 transmits to the payment server 70 information on the user ID and the amount of the payment.

[0096] In this way, when face authentication by the control unit 41 is not successful, the first information is collected. Then, in the next and subsequent face authentication, face authentication is performed based on the processed authentication face image in which the first information is reflected, so that the accuracy of face authentication can be improved. That is, it is possible to suppress the discrepancy between how the user appears in the user face image obtained during face authentication and how the user appears in the processed authentication face image. Therefore, it is possible to improve the accuracy of face authentication of the user who remains in the vehicle 10.

[0097] <Second embodiment> In this embodiment, when a plurality of vehicles 10 of the same model are linked to the same user ID, the first information or the processed face image for authentication that has already been generated is reused. Even if the vehicle 10 is assigned a vehicle ID different from that of the vehicle 10 for which the first information has already been generated, the already generated first information may be reused if the vehicle model is the same. That is, if the vehicle model is the same, the attributes of the vehicle 10 are considered to have a similar effect on the user face image, so if the vehicle model and user ID are the same, the first information or the processed face image for authentication may be reused. In addition, even if the user ID is different, if the vehicle model is the same, the attributes of the vehicle 10 are considered to have a similar effect on the user face image, so the first information may be reused for other users.

[0098] When storing the first information in the first information DB 421, the first information generation unit 417 of the utilization side server 40 stores the first information in association with the user ID, the vehicle ID, and the vehicle model. Here, FIG. 14 is a diagram showing an example of a table configuration of the first information DB 421 according to the embodiment. The first information DB 421 stores the relationship between the user ID, the vehicle ID, the vehicle model, and the first information. The information related to the vehicle model is included in the information acquired by the vehicle information acquisition unit 412 from the vehicle ID server 60.

[0099] In addition, before generating the processed face image for authentication, the face image processing unit 418 of the user side server 40 Then, it is determined whether a record with the same vehicle model already exists in the first information DB 421. If a record with the same vehicle model exists, the face image processing unit 418 of the user server 40 generates a processed authentication face image based on the first information. In this way, when first information has already been generated for a vehicle 10 of the same vehicle model, new first information is not generated for the same vehicle model thereafter, but the first information is reused.

[0100] Next, the face authentication process in the use server 40 will be described. FIG. 15 is a flowchart of the face authentication process in the use server 40 according to the embodiment. The process shown in FIG. 15 is executed at predetermined time intervals in the use server 40. Note that the process shown in FIG. 15 is executed by each software module of the control unit 41, but in the following flowchart, it will be described as if the control unit 41 executes the process. Note that the steps in which the same processes as those in FIG. 13 are executed are given the same reference numerals and their explanations will be omitted. Also, since the process up to step S306 is the same as that in FIG. 13, the process from step S306 onwards is shown in FIG. 15.

[0101] In the routine shown in FIG. 15, when the process of step S306 is completed, the process proceeds to step S401. In step S401, the control unit 41 acquires vehicle information linked to the vehicle ID from the vehicle ID server 60. The vehicle information includes information capable of identifying the vehicle, such as information on the vehicle model, vehicle name, vehicle number, or vehicle registration plate number (vehicle number). The vehicle information may include information capable of identifying the vehicle's height, size, windshield color, side window color, wiper position, sun visor position, or A-pillar position. The vehicle information may also include information on the owner of the vehicle 10.

[0102] In step S402, the control unit 41 identifies the vehicle model from the vehicle information. The information related to the vehicle model is included in the vehicle information. In step S403, the control unit 41 determines whether or not the first information corresponding to the same vehicle model exists. At this time, the control unit 41 accesses the first information DB 421 and determines whether or not a record exists in which the same vehicle model as the vehicle model identified in step S402 is stored. If this record exists, the control unit 41 determines that the first information corresponding to the same vehicle model exists. On the other hand, if this record does not exist, the control unit 41 determines that the first information corresponding to the same vehicle model does not exist. If the determination in step S403 is affirmative, the process proceeds to step S404, and if the determination is negative, the process proceeds to step S310.

[0103] In step S404, the control unit 41 processes the authentication face image using the first information to generate a processed authentication face image. For example, the control unit 41 processes the authentication face image acquired from the center server 30 so that the processed authentication face image becomes a face image of the user photographed through the windshield. A known technique can be used for this processing.

[0104] In this way, by reusing the first information, a processed authentication face image corresponding to the vehicle model can be generated in advance, so that the accuracy of face authentication can be improved and the processing load for generating the first information can be reduced.

[0105] <Third embodiment> In this embodiment, when the first information includes information based on vehicle attributes, information based on user attributes, and information based on system attributes, the first information is reused. When the first information includes information based on vehicle attributes, for example, the first information may be available for other users who drive a vehicle 10 with the same attributes. For example, when face authentication of a certain user is not successful due to information based on vehicle attributes, the first information generated at that time can be reused for face authentication of other users who drive the same vehicle model.

[0106] In addition, when the first information includes information based on user attributes, for example, the first information may be available when the same user is in another vehicle. The user attributes are attributes related to the user, such as, for example, whether the user is wearing sunglasses or glasses, whether the user is wearing a mask, whether the user is wearing a hat, or whether the user has a beard. For example, assume that the user owns multiple vehicles 10. The first information generated when face authentication is not established due to information based on user attributes while the user is in a certain vehicle can be used for face authentication when the same user is in another vehicle.

[0107] Furthermore, when the first information includes information based on, for example, system attributes, the first information may be available for other users or other vehicles if the system attributes are the same. The system attributes are attributes related to the system 1, such as the mounting position of the camera 400, the angle of view of the camera 400, and the distance from the camera 400 to the vehicle 10. When face authentication is performed at the same location, the system attributes have the same effect on face authentication regardless of which user or which vehicle 10 is used. Note that the first information may be available not only at the same location, but also at different locations, such as when the relative angle between the camera 400 and the vehicle 10 is the same.

[0108] For example, if the mounting position of the camera 400 is shifted from the front of the user, the angle of the user's face in the user face image may be tilted up, down, left, or right from the state of facing forward. Therefore, the accuracy of face authentication can be improved by having the control unit 41 generate a processed authentication face image so as to correct the inclination of the face according to the mounting position of the camera 400. The mounting position of the camera 400 is the same regardless of the user or the vehicle 10.

[0109] Furthermore, the user's face captured in the user face image may be distorted depending on the angle of view of the camera 400. Therefore, the accuracy of face authentication can be improved by having the control unit 41 generate a processed authentication face image so as to correct the distortion according to the angle of view of the camera 400. The angle of view of the camera 400 is the same regardless of the user or the vehicle 10.

[0110] Furthermore, the user's face captured in the user face image may be distorted depending on the distance from the camera 400 to the vehicle 10. Therefore, the accuracy of face authentication can be improved by having the control unit 41 generate a processed authentication face image so as to correct the distortion according to the distance from the camera 400 to the vehicle 10. The distance from the camera 400 to the vehicle 10 is the same regardless of the user or the vehicle 10.

[0111] In this way, the system attributes are the same regardless of the user or the vehicle 10. Therefore, when the first information includes information based on the system attributes, the first information can be reused when performing face authentication for another user. Also, when the first information includes information based on the system attributes, the first information can be reused when the same user is in a different vehicle 10.

[0112] When generating the first information, the first information generating unit 417 of the user server 40 determines whether the first information corresponds to information based on attributes of the vehicle 10, information based on attributes of the user, or information based on attributes of the system 1. The first information generating unit 417 may make this determination, for example, by image analysis. As another example, the first information generating unit 417 may make this determination by using a trained machine learning model. When a user's face image is input, this machine learning model outputs whether the first information corresponds to information based on attributes of the vehicle 10, information based on attributes of the user, or information based on attributes of the system 1. When the first information generating unit 417 stores the first information in the first information DB 421, the first information generating unit 417 stores the first information together with information that can determine whether the first information corresponds to a vehicle attribute, a user attribute, or a system attribute.

[0113] FIG. 16 is a diagram showing an example of a table configuration of the first information DB 421 according to the embodiment. The first information DB 421 stores a user ID, a vehicle ID, a vehicle model, a camera position, a vehicle attribute, a user attribute, a system attribute, and a relationship between the first information. The vehicle model is shown as an example of a vehicle attribute. The camera position is shown as an example of an attribute of the system 1. The camera position may be indicated by, for example, the distance and angle from the ground to the camera 400. In each of the vehicle attribute, user attribute, and system attribute fields, "1" is stored if there is a correspondence, and "0" is stored if there is no correspondence. For example, in FIG. 16, the first information field of a record in which "1" is stored in the vehicle attribute stores information based on the vehicle attribute.

[0114] Next, the face authentication process in the use server 40 will be described. FIG. 17 is a flowchart of the face authentication process in the use server 40 according to the embodiment. The process shown in FIG. 17 is executed at predetermined time intervals in the use server 40. Note that the process shown in FIG. 17 is executed by each software module of the control unit 41, but in the following flowchart, it will be described as if the control unit 41 executes the process. Note that the steps in which the same processes as those in FIG. 13 or FIG. 15 are executed are given the same reference numerals and their explanations will be omitted. Since the process up to step S401 is the same as that in FIG. 15, FIG. 17 shows the process from step S401 onwards.

[0115] 17, when the process of step S401 is completed, the process proceeds to step S501. In step S501, the control unit 41 identifies vehicle attributes. The vehicle attributes are included in the vehicle information acquired in step S401.

[0116] In step S502, the control unit 41 identifies the system attributes. Information on the system attributes may be measured in advance for each camera 400 and stored in the storage unit .

[0117] In step S503, the control unit 41 determines whether or not first information corresponding to the same vehicle attribute exists in the first information DB 421. That is, the control unit 41 determines whether or not reusable first information corresponding to the vehicle attribute exists. The control unit 41 refers to the first information DB 421 and determines whether or not a record exists in which the information related to the vehicle model is the same and "1" is stored in the vehicle attribute field. If the determination in step S503 is affirmative, the process proceeds to step S506, and if the determination is negative, the process proceeds to step S504.

[0118] In step S504, the control unit 41 determines whether or not first information corresponding to the same user attribute exists in the first information DB 421. That is, the control unit 41 determines whether or not divertable first information corresponding to the user attribute exists. The control unit 41 refers to the first information DB 421 and determines whether or not a record exists in which the user ID is the same and "1" is stored in the user attribute field. If the determination in step S504 is affirmative, the process proceeds to step S506, and if the determination is negative, the process proceeds to step S505.

[0119] In step S505, the control unit 41 determines whether or not first information corresponding to the same system attribute exists in the first information DB 421. That is, the control unit 41 determines whether or not reusable first information corresponding to the system attribute exists. The control unit 41 refers to the first information DB 421 and determines whether or not a record exists in which the information regarding the camera position is the same and "1" is stored in the system attribute field. Affirmative determination in step S505 If so, the process proceeds to step S506, and if a negative determination is made, the process proceeds to step S310.

[0120] In step S506, the control unit 41 processes the authentication face image using the first information to generate a processed authentication face image. For example, the control unit 41 generates the processed authentication face image according to at least one attribute of the vehicle attribute, the user attribute, and the system attribute.

[0121] 17, if a positive determination is made in step S313, the process proceeds to step S507. In step S507, the control unit 41 stores the first information in the first information DB 421. The control unit 41 associates the first information with the vehicle ID and the user ID and stores the first information in the first information DB 421. The control unit 41 also determines corresponding attributes among the vehicle attributes, the user attributes, and the system attributes, and stores "1" in the corresponding fields of the vehicle attributes, the user attributes, and the system attributes in the first information DB 421.

[0122] As described above, according to the present embodiment, a processed authentication face image is generated according to information based on at least one of vehicle attributes, user attributes, and system attributes, so that past information can be reused. Therefore, the accuracy of face authentication can be improved. In addition, the load of generating the first information can be reduced.

[0123] <Fourth embodiment> In this embodiment, when face authentication is not established, a face image of the user is further acquired. Reacquisition of the face image of the user is started after face authentication is not established, and is performed, for example, during the period when the second authentication is being performed. That is, reacquisition of the user face image and the second authentication are performed in parallel. This allows the face image of the user performing the second authentication to be left as a record. This allows evidence to be left in the case of fraudulent use. It is also considered that the first information is more easily generated from a user face image captured later than from a user face image captured first. Therefore, the accuracy of the first information can be improved. Furthermore, by capturing a user face image in parallel with the second authentication, the time required for processing can be reduced compared to capturing a user face image after the second authentication is established. Note that the reacquired user face image does not necessarily need to be used to generate the first information. For example, the reacquired user face image may simply be stored in the storage unit 42 as evidence. As another example, the first information may be generated using only the reacquired user face image.

[0124] The control unit 41 reacquires a user face image, for example, after a negative determination is made in step S311 of the routine shown in Fig. 13 and before the processing in step S312 or step S313 is started. The reacquired user face image is associated with the user ID and the vehicle ID and stored in the storage unit 42 so that it is clear that the image is a reacquired image. In this case, the user face image may simply be stored as evidence, or may be used when generating the first information in step S314.

[0125] In addition, when the keypad 500 is disposed outside the vehicle 10, it is possible that the user may stick his / her face out of the vehicle or get off the vehicle 10 when inputting the PIN into the keypad 500. In this case, it is possible to capture a user face image without being affected by the vehicle attributes. In this case, although it is not possible to obtain information that depends on the vehicle attributes, it is possible to obtain information that depends on the user attributes and information that depends on the system attributes with a higher degree of accuracy. Therefore, it is easier to determine which of the vehicle attributes, the user attributes, and the system attributes caused the face authentication to fail, and therefore the accuracy of the first information can be improved. In addition, face authentication may be performed again based on a user face image that is reacquired without being affected by the vehicle attributes. By performing face authentication again in parallel with the second authentication, the authentication accuracy can be improved. The degree can be increased.

[0126] <Other embodiments> The above embodiment is merely an example, and the present disclosure may be modified as appropriate within the scope of the gist of the disclosure. The processes and means described in the present disclosure may be freely combined and implemented as long as no technical contradiction occurs. In addition, a process described as being performed by one device may be shared and executed by multiple devices. Alternatively, a process described as being performed by different devices may be executed by one device. In a computer system, the hardware configuration (server configuration) by which each function is realized can be flexibly changed.

[0127] For example, one server may have the functions of two or more of the center server 30, the usage side server 40, the linking management server 50, the vehicle ID server 60, and the payment server 70. For example, the center server 30 and the usage side server 40 may be one server. Similarly, the center server 30 and the linking management server 50 or the vehicle ID server 60 may be one server. At least a part of the processing of the center server 30 may be executed by the usage side server 40. At least a part of the processing of the usage side server 40 may be executed by the center server 30.

[0128] The first information may be generated by the control unit 31 of the center server 30. The processed authentication face image may be generated by the control unit 31 of the center server 30. In this case, if authentication is not established in the use server 40, the use server 40 may transmit a user face image to the center server 30. Then, the processed authentication face image generated in the center server 30 may be transmitted from the center server 30 to the use server 40.

[0129] In the above embodiment, when face authentication is successful, a payment is requested from the payment server 70. However, the use-side server 40 can execute various types of authority of the user, not limited to payment.

[0130] The present disclosure can also be realized by supplying a computer program implementing the functions described in the above embodiments to a computer, and having one or more processors of the computer read and execute the program. Such a computer program may be provided to the computer by a non-transitory computer-readable storage medium connectable to the system bus of the computer, or may be provided to the computer via a network. Non-transitory computer-readable storage media include, for example, any type of disk, such as a magnetic disk (floppy disk, hard disk drive (HDD), etc.), an optical disk (CD-ROM, DVD disk, Blu-ray disk, etc.), a read-only memory (ROM), a random access memory (RAM), an EPROM, an EEPROM, a magnetic card, a flash memory, an optical card, and any type of medium suitable for storing electronic instructions. [Explanation of symbols]

[0131] 1 System 10 Vehicles 20 User terminal 30 Center Server 31 Control Unit 32 Storage section 40 User Server 41 Control section 42 Storage section 50 Linking Management Server 60 Vehicle ID Server 70 Payment Server 100 Onboard equipment 400 Cameras

Claims

1. Obtaining user identification information, which is identification information of a user of the vehicle; acquiring an authentication face image associated with the user identification information, the authentication face image being an authentication face image of a user of the vehicle; acquiring a user face image, the user face image being an image generated by photographing a user of the vehicle with a camera disposed outside the vehicle; performing face authentication based on a comparison between the authentication face image and the user face image; acquiring, in response to the failure of the face authentication, first information to be used in subsequent face authentication from the user face image; An information processing device comprising a control unit that executes the above.

2. The control unit is If the face authentication is not successful, a second authentication other than the face authentication is performed. In response to the second authentication being established, the first information is acquired from a face image of the user. The information processing device according to claim 1 .

3. the control unit, in response to the presence of the first information, performs the face authentication based on a comparison between a processed authentication face image, which is the authentication face image reflecting the first information, and the user face image. The information processing device according to claim 1 .

4. the control unit processes the authentication face image in accordance with the first information to generate the processed authentication face image. The information processing device according to claim 3 .

5. The control unit stores the first information in a storage unit so that the first information can be used for face authentication when the user uses the vehicle. The information processing device according to claim 1 .

6. The first information includes at least one of information based on an attribute of a first user, information based on an attribute of a first vehicle, and information based on an attribute of a first system that performs the face authentication. The information processing device according to claim 1 .

7. The control unit stores the first information in a storage unit so that the first information can be used for face authentication of the first user in response to the first information including information based on an attribute of the first user. The information processing device according to claim 6.

8. the control unit, in response to the first information including information depending on the attributes of the first vehicle, stores the first information in a storage unit so as to be usable for face authentication of a second user of a second vehicle having the same attributes as the attributes of the first vehicle. The information processing device according to claim 6.

9. the control unit, in response to the first information including information depending on an attribute of the first system, stores the first information in a storage unit so as to be usable for face authentication of a second user in a second system that performs the same face authentication as the first system; The information processing device according to claim 6.

10. The control unit, in response to performing the second authentication, after the face authentication is not established, and acquiring a face image of the user again before the second authentication is completed. The information processing device according to claim 2 .

11. The computer Obtaining user identification information, which is identification information of a user of the vehicle; acquiring an authentication face image associated with the user identification information, the authentication face image being an authentication face image of a user of the vehicle; acquiring a user face image, the user face image being an image generated by photographing a user of the vehicle with a camera disposed outside the vehicle; performing face authentication based on a comparison between the authentication face image and the user face image; acquiring, in response to the failure of the face authentication, first information to be used in subsequent face authentication from the user face image; An information processing method for performing the above.

12. The computer, If the face authentication is not successful, a second authentication other than the face authentication is performed. In response to the second authentication being established, the first information is acquired from a face image of the user. The information processing method according to claim 11.

13. the computer performs the face authentication based on a comparison between a processed authentication face image, which is the authentication face image reflecting the first information, and the user face image, in response to the presence of the first information; The information processing method according to claim 11.

14. the computer processes the authentication face image in accordance with the first information to generate the processed authentication face image; The information processing method according to claim 13.

15. The computer stores the first information in a storage unit so that the first information can be used for face authentication when the user uses the vehicle. The information processing method according to claim 11.

16. The first information includes at least one of information based on an attribute of a first user, information based on an attribute of a first vehicle, and information based on an attribute of a first system that performs the face authentication. The information processing method according to claim 11.

17. The computer stores the first information in a storage unit so that the first information can be used for face authentication of the first user in response to the first information including information based on an attribute of the first user. The information processing method according to claim 16.

18. the computer, in response to the first information including information based on attributes of the first vehicle, stores the first information in a storage unit so as to be usable for face authentication of a second user of a second vehicle having attributes identical to the attributes of the first vehicle; The information processing method according to claim 16.

19. the computer, in response to the first information including information depending on an attribute of a first system that performs the facial authentication, stores the first information in a storage unit so as to be usable for facial authentication of a second user in a second system that performs the facial authentication identical to the first system; The information processing method according to claim 16.

20. On the computer, Obtaining user identification information, which is identification information of a user of the vehicle; acquiring an authentication face image associated with the user identification information, the authentication face image being an authentication face image of a user of the vehicle; acquiring a user face image, the user face image being an image generated by photographing a user of the vehicle with a camera disposed outside the vehicle; performing face authentication based on a comparison between the authentication face image and the user face image; acquiring, in response to the failure of the face authentication, first information to be used in subsequent face authentication from the user face image; A program for executing the above.

Images