Information processing apparatus, information processing method, and non-transitory storage medium

Abstract

The present disclosure relates to an information processing apparatus, an information processing method, and a non-transitory storage medium for improving the accuracy of face authentication. The vehicle control device is provided with a control unit that executes: user identification information that is identification information of a user of a vehicle; a face image acquisition unit that acquires a face image for authentication that is an image associated with the user identification information and that is an image of a face for authentication of a user of the vehicle; acquiring a user face image, which is an image generated by capturing an image of a user of the vehicle using a camera disposed outside the vehicle; performing face authentication on the basis of a comparison between the face image for authentication and the user face image; and acquiring, from the user face image, first information that is information to be used in the next and subsequent face authentication in response to the face authentication being not established.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, and a non-transitory storage medium storing a program. Background Art

[0002] It is known that face image data of workers who are allowed to enter a safety area is stored in advance as registration information and a face authentication process is performed (for example, refer to Patent Document 1).

[0003] Patent Document 1: Japanese Patent Application Publication No. 2022-137959 Summary of the invention

[0004] The purpose of the present disclosure is to improve the accuracy of face authentication.

[0005] One of the aspects of the present disclosure is an information processing device including a control unit that performs the following processing:

[0006] Obtaining identification information of a user of the vehicle, i.e., user identification information;

[0007] obtaining an authentication facial image, the authentication facial image being an image associated with the user identification information and being an authentication facial image of a user of the vehicle;

[0008] obtaining a user facial image, the user facial image being an image generated by photographing a user of the vehicle using a camera disposed outside the vehicle;

[0009] performing facial authentication based on comparison of the authentication facial image with the user facial image; and

[0010] In response to the failure of the facial authentication, first information is acquired from the user's facial image as information to be used in the next facial authentication or later.

[0011] One of the aspects of the present disclosure is an information processing method, in which a computer performs the following processing:

[0012] Obtaining identification information of a user of the vehicle, i.e., user identification information;

[0013] obtaining an authentication facial image, the authentication facial image being an image associated with the user identification information and being an authentication facial image of a user of the vehicle;

[0014] obtaining a user facial image, the user facial image being an image generated by photographing a user of the vehicle using a camera disposed outside the vehicle;

[0015] performing facial authentication based on comparison of the authentication facial image with the user facial image; and

[0016] In response to the failure of the facial authentication, first information is acquired from the user's facial image as information to be used in the next facial authentication or later.

[0017] One aspect of the present disclosure is a non-transitory storage medium storing a program for causing a computer to execute the following processing:

[0018] Obtaining identification information of a user of the vehicle, i.e., user identification information;

[0019] obtaining an authentication facial image, the authentication facial image being an image associated with the user identification information and being an authentication facial image of a user of the vehicle;

[0020] obtaining a user facial image, the user facial image being an image generated by photographing a user of the vehicle using a camera disposed outside the vehicle;

[0021] performing facial authentication based on comparison of the authentication facial image with the user facial image; and

[0022] In response to the failure of the facial authentication, first information is acquired from the user's facial image as information to be used in the next facial authentication or later.

[0023] In addition, another aspect of the present disclosure is the above-mentioned program.

[0024] According to the present disclosure, the accuracy of face authentication can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a diagram showing a schematic configuration of a system according to an embodiment.

[0026] Figure 2 This is a diagram showing an overview of the processing of the system according to the embodiment.

[0027] Figure 3 This is a block diagram schematically showing an example of the hardware configuration of each of the center server, the user-side server, the association management server, the vehicle ID server, and the settlement server that constitute the system involved in the embodiment.

[0028] Figure 4 This is a block diagram schematically showing an example of the hardware configuration of each of the user terminal and the vehicle-mounted device according to the present embodiment.

[0029] Figure 5 This is a diagram schematically showing an example of the software configuration of the center server according to the embodiment.

[0030] Figure 6It is a diagram showing an example of the table structure of the user information DB and the authentication facial image DB according to the embodiment.

[0031] Figure 7 This is a diagram schematically showing an example of the software configuration of the user-side server according to the embodiment.

[0032] Figure 8 It is a diagram showing an example of the table structure of the first information DB and the second authentication information DB according to the embodiment.

[0033] Fig. 9 It is a diagram schematically showing an example of the software configuration of the vehicle-mounted device according to the embodiment.

[0034] Fig.10 It is a diagram schematically showing an example of the software configuration of the user terminal according to the embodiment.

[0035] Fig.11 This is a flowchart of a user registration process in the center server according to the embodiment.

[0036] Fig.12 This is a flowchart of a process of providing a facial image for authentication in a center server according to an embodiment.

[0037] Fig.13 This is a flowchart of the facial authentication process in the user-side server according to the embodiment.

[0038] Fig.14 It is a diagram showing an example of the table structure of the first information DB involved in the embodiment.

[0039] Fig.15 This is a flowchart of the facial authentication process in the user-side server according to the embodiment.

[0040] Fig.16 It is a diagram showing an example of the table structure of the first information DB involved in the embodiment.

[0041] Fig.17 This is a flowchart of the facial authentication process in the user-side server according to the embodiment. DETAILED DESCRIPTION

[0042] According to the face recognition system, even if the target person leaves the system, as long as the face of the target person can be photographed by the camera, the target person can be recognized. Therefore, according to the face recognition, the target person can be recognized easily and quickly. However, the existing face recognition system has the following problems.

[0043] That is, in recent years, with the advancement of machine learning technology, the scenarios in which facial recognition can be implemented have increased. In addition, the scenarios in which the recognition of the subject person is recommended in the state of riding in the vehicle, such as drive through, have increased, and thus the demand for facial recognition has increased. In view of this, it is assumed that the subject person riding in the vehicle is photographed from the outside of the vehicle and the subject person is recognized by the obtained facial image. It is assumed that the system is configured to perform facial recognition by comparing the facial image for recognition obtained in advance with the facial image obtained during the recognition. In this case, the facial image for recognition is basically obtained by photographing the subject person at any time when not riding in the vehicle (for example, the subject person uses a mobile terminal to photograph himself). On the other hand, the facial image obtained during the recognition, for example, will reflect the components constituting the vehicle, such as the front windshield and the side window glass. In one example, because the subject person is photographed across the front windshield, the facial image obtained will be affected by the color of the front windshield, and the face of the subject person may be reflected in a color different from the actual color. As a result, the reflection state of the subject person in the facial image obtained during the recognition is greatly deviated from the reflection state of the subject person in the facial image for recognition, and there is a possibility that facial recognition will not be accurately established. In the present disclosure, the above-mentioned problems are solved.

[0044] In view of this, an information processing device as one of the methods of the present invention includes a control unit that performs the following processing: obtaining identification information of a user of a vehicle, that is, user identification information; obtaining an authentication facial image, which is an image associated with the above-mentioned user identification information and is an image of the face of the user of the above-mentioned vehicle for authentication; obtaining a user facial image, which is an image generated by photographing the user of the above-mentioned vehicle using a camera arranged outside the above-mentioned vehicle; performing facial authentication based on comparison between the above-mentioned authentication facial image and the above-mentioned user facial image; and obtaining first information from the above-mentioned user facial image as information to be used in the next facial authentication or later based on the above-mentioned facial authentication not being successful.

[0045] The user of the vehicle is a user who has established an association with the vehicle. The user may be the owner of the vehicle. User identification information is information that can identify the user, and may be, for example, information related to a pre-assigned user ID. In addition, as another example, the user identification information may be information related to the number of a driver's license. The facial image for authentication is an image obtained by pre-capturing the user's face. The facial image for authentication may be, for example, an image of the user's face captured by a camera attached to the user's terminal. In addition, the facial image for authentication may be pre-registered by establishing an association with the user identification information.

[0046] Furthermore, the control unit obtains an image generated by photographing the user of the vehicle using a camera arranged outside the vehicle, that is, a user face image. The user face image is, for example, a face image of the user at the current moment. Furthermore, the control unit performs face authentication based on the comparison between the authentication face image and the user face image. At this time, even if the user is a regular user, there is a case where face authentication is not established. Among the reasons, for example, reasons based on the attributes of the user, reasons based on the attributes of the vehicle, and reasons based on the system for performing face authentication can be considered. Reasons based on the attributes of the user are, for example, reasons related to the user, such as the user wearing sunglasses or glasses, the user wearing a mask, the user wearing a hat, and the user growing a beard. In addition, reasons based on the attributes of the vehicle are, for example, reasons related to the vehicle, such as the height of the vehicle (vehicle height), the size of the vehicle, the color of the front windshield, the color of the side window glass, the position of the wiper, the position of the sun visor, or the position of the A-pillar (the pillar located on both sides of the front windshield). In addition, reasons based on the attributes of the system are, for example, reasons related to the system, such as the height from the ground to the camera, the distance from the camera to the vehicle, and the viewing angle of the camera. Due to these influences, the user's face shown in the user face image may deviate from the user's face shown in the authentication face image. This may cause facial authentication to fail.

[0047] In view of this, the control unit obtains the first information from the user's facial image as information used in the next facial authentication in accordance with the failure of facial authentication. The first information includes information related to the reason why facial authentication was not successful. By collecting the information related to the reason why facial authentication was not successful and using it in the next facial authentication in accordance with the failure of facial authentication, it is possible to prevent the situation where facial authentication is not successful even if the user is a legitimate user. That is, the accuracy of facial authentication in similar situations can be improved.

[0048] Hereinafter, embodiments of the present disclosure will be described based on the drawings. The configurations of the following embodiments are illustrative only, and the present disclosure is not limited to the configurations of the embodiments.

[0049] <First embodiment>

[0050] Figure 1 1 is a diagram showing a schematic configuration of a system 1 according to an embodiment of the present invention. Figure 1 In the example of FIG. 1 , the system 1 includes an in-vehicle device 100 mounted on a vehicle 10, a user terminal 20 held by a user of the vehicle 10, a center server 30, a user-side server 40, an association management server 50, a vehicle ID server 60, and a settlement server 70. The user-side server 40 is connected to a camera 400 for photographing the interior of the vehicle 10 from the outside of the vehicle 10 and a numeric keypad for the user to input a number.

[0051] The vehicle-mounted device 100, the user terminal 20, the central server 30, the user-side server 40, the association establishment management server 50, the vehicle ID server 60, and the settlement server 70 are connected to each other via a network N1. The network N1 is, for example, a world-scale public communication network such as the Internet, and a WAN (Wide Area Network) or other communication network may be used. In addition, the network N1 may include a telephone communication network such as a mobile phone, a wireless communication network such as Wi-Fi (registered trademark), etc. Figure 1 Although one vehicle 10 and one user terminal 20 are shown as examples, there may be a plurality of them. In addition, the vehicle-mounted device 100 and the user-side server 40 or the camera 400 may also be capable of short-distance wireless communication.

[0052] The vehicle-mounted device 100 is a device that provides information related to the vehicle 10 to the user-side server 40. The user terminal 20 is a terminal that can provide a facial image of a user. The central server 30 is a server that generates a facial image for authentication and provides it to the user-side server 40. The user-side server 40 is a server that performs facial authentication. The association establishment management server 50 is a server that manages the association establishment between the vehicle 10 and the user. The vehicle ID server 60 is a server that manages information related to the vehicle 10. The settlement server 70 is a server that performs settlement.

[0053] Figure 2 1 is a diagram showing an overview of the processing of the system 1 involved in the embodiment. P1 to P5 are processes in which the central server 30 participates, and U1 to U8 are processes in which the user-side server 40 participates. In P1, the user uses the user terminal 20 to register the user with the central server 30. The user requests the central server 30 to register the user via the user terminal 20. At this time, the facial image associated with the user ID is sent to the central server 30. Hereinafter, the facial image is also referred to as a facial image for authentication. The facial image for authentication is, for example, an image of the user's face taken by the user terminal 20. However, it is not limited to this, and the facial image for authentication may also be an image of the user's face taken in the past and stored in the user terminal 20 or an external server. Among them, the user ID may be, for example, a number assigned according to a common numbering system for identifying individuals in various countries (for example, my number, social security number, or citizen identity number) or a driver's license number. The user ID is an example of user identification information. In addition, when the user registers, the information related to the settlement is associated with the user ID from the user terminal 20 and sent to the central server 30. The information involved in settlement is the information required when the user uses the settlement service, including, for example, a credit card number or a bank account number.

[0054] In P2, the central server 30 obtains the user ID, the facial image for authentication, and the information related to the settlement from the user terminal 20 and stores them. In P3, the central server 30 requests the association establishment management server 50 to send the vehicle ID associated with the user ID. The association establishment management server 50 performs a search based on the user ID and extracts the vehicle ID associated with the user ID. Then, the central server 30 obtains the vehicle ID associated with the user ID from the association establishment management server 50. The vehicle ID is a number inherent to the vehicle 10, such as a vehicle body number or a number displayed on a car registration plate (number plate). Hereinafter, this number is also referred to as a vehicle number.

[0055] In P4, the central server 30 requests the vehicle ID server 60 to send the vehicle information associated with the vehicle ID. The vehicle ID server 60 performs a search based on the vehicle ID to extract the vehicle information associated with the vehicle ID. Then, the central server 30 obtains the vehicle information associated with the vehicle ID from the vehicle ID server 60. The vehicle information includes information that can identify the vehicle, such as the model, name, body number or the number of the car registration plate (vehicle number). In addition, the vehicle information may also include information that can determine the height, size, color of the front windshield, color of the side window glass, position of the wiper, position of the sun visor or position of the A-pillar, etc. of the vehicle. In addition, the vehicle information may include information related to the owner of the vehicle 10.

[0056] In P5, the vehicle information obtained from the vehicle ID server 60 is stored in the center server 30. At this time, the attributes of the vehicle 10 can be obtained from the vehicle information and stored. For example, the color of the front windshield can be obtained as the attribute of the vehicle 10 from the vehicle information.

[0057] In addition, in U1, the side server 40 is used to obtain the vehicle ID and the user's facial image. The user's facial image is obtained by photographing the vehicle 10 from the outside with the camera 400. Hereinafter, the facial image obtained in this way is also referred to as the user's facial image. In addition, the vehicle ID can be obtained from the vehicle-mounted device 100 through communication. In addition, as another example, when the vehicle ID corresponds to the vehicle number, the vehicle number can be obtained by analyzing the image of the vehicle 10 photographed by the camera 400.

[0058] In U2, the utilization side server 40 inquires the association establishment management server 50 whether there is an association establishment relationship corresponding to the vehicle ID. At this time, it can also be inquired whether there is a valid association establishment relationship at the target date and time. The target date and time can be the date and time of the inquiry time (which can be the current time) or the date and time in the past. A valid association establishment is, for example, an association establishment within the validity period when there is a validity period for the association establishment between the vehicle ID and the user ID. When there is a valid association establishment, the user ID associated with the vehicle ID is sent from the association establishment management server 50 to the utilization side server 40.

[0059] In U3, the utilization side server 40 requests the vehicle ID server 60 to send the vehicle information associated with the vehicle ID. The vehicle ID server 60 performs a search based on the vehicle ID and extracts the vehicle information associated with the vehicle ID. Moreover, the utilization side server 40 obtains the vehicle information associated with the vehicle ID from the vehicle ID server 60. The vehicle information includes information that can identify the vehicle 10, such as information related to the model, name, body number or the number of the car registration plate (vehicle number). In addition, the vehicle information may also include information that can determine the height, size, color of the front windshield, color of the side window glass, position of the wiper, position of the sun visor or position of the A-pillar of the vehicle 10. In addition, the vehicle information may include information related to the owner of the vehicle 10.

[0060] In U4, when there is a valid association relationship, the user-side server 40 requests the central server 30 to send the authentication facial image corresponding to the user ID. The central server 30 sends the authentication facial image corresponding to the user ID to the user-side server 40. At this time, the central server 30 also sends information related to various permissions associated with the user ID to the user-side server 40. Various permissions are, for example, permissions related to settlement services registered by the user. In addition, the permissions are not limited to this, and may also be, for example, permissions related to administrative services. In addition, the authentication facial image may be pre-stored in the user-side server 40.

[0061] In U5, the side server 40 performs facial authentication by comparing the user's facial image captured by the camera 400 with the facial image for authentication. Here, known facial authentication techniques can be used. The method of facial authentication is not limited. In addition, image processing can be performed by any analytical processing (pattern matching, edge extraction, etc.). In addition, a learned machine learning model can be used in image processing.

[0062] In U6, when facial authentication is successful, the user-side server 40 executes the user's authority. For example, the user-side server 40 requests settlement from the settlement server 70. For example, it can be used for settlement of drive-through at a restaurant, settlement of parking fees, or settlement of toll road fees.

[0063] On the other hand, in U7, when facial authentication is not established, the side server 40 performs a second authentication. For example, the side server 40 can perform a second authentication by means such as inputting a password to the numeric keypad 500, fingerprint authentication, and voiceprint authentication. In this case, for example, the user can be reported as inputting a password to the numeric keypad 500 configured near the stop position of the vehicle 10. The report can be made, for example, by playing a sound or displaying a screen on a sign. In addition, a device for reading the user's fingerprint can be pre-configured near the stop position of the vehicle 10, and it can be reported that the fingerprint is input through the device. In addition, a microphone for inputting the user's voice can also be configured near the stop position of the vehicle 10, and it can be reported that the voice is input to the microphone. In addition, as another example, the side server 40 can send an instruction to the user terminal 20 or the vehicle-mounted device 100 to obtain a password, fingerprint, or voice from the user terminal 20 or the vehicle-mounted device 100. Moreover, the side server 40 can also obtain a password, fingerprint, or voice from the user terminal 20 or the vehicle-mounted device 100. Furthermore, as another example, the use-side server 40 may perform SMS authentication, telephone authentication, or email authentication using the user terminal 20. In this way, the use-side server 40 performs user authentication through the second authentication that is other authentication than facial authentication.

[0064] In U8, when the second authentication is established, the user-side server 40 generates and stores the first information based on the user's facial image. The user-side server 40 extracts, for example, the color of the front windshield of the vehicle 10 and stores information related to the color of the front windshield as the first information. In addition, the user-side server 40 may associate the first information with the user ID and send it to the central server 30, and store the first information in the central server 30. Then, the user-side server 40 may execute the user's authority in the same manner as U6.

[0065] In addition, when the first information already exists in the user-side server 40, in U5, the user-side server 40 generates a processed authentication facial image by reflecting the first information in the authentication facial image. Furthermore, the user-side server 40 performs facial authentication by comparing the user's facial image captured by the camera 400 with the processed authentication facial image.

[0066] In addition, the processed facial image for authentication can be generated by either the central server 30 or the user-side server 40. In addition, the processed facial image for authentication can be stored in either the central server 30 or the user-side server 40. In addition, the processed facial image for authentication only needs to be generated before facial authentication. For example, the user-side server 40 or the central server 30 can generate the processed facial image for authentication based on the first information obtained, and store it in the user-side server 40 or the central server 30. In addition, when a user owns multiple vehicles 10, the user-side server 40 or the central server 30 can generate the first information or the processed facial image for authentication for each vehicle 10 and store it.

[0067] In addition, the first information reflected when the processed facial image for authentication is generated may be the color of the front windshield or side window glass, or the position and shape of the wiper, sun visor, A-pillar or front seat, or the vehicle height, etc., which are information based on the attributes of the vehicle 10. In addition, as another example, the first information may be information based on the attributes of the user, or information based on the attributes of the system 1. Information based on the attributes of the user may be information related to the user, such as the user wearing sunglasses or glasses, the user wearing a mask, the user wearing a hat, and the user growing a beard. In addition, information based on the attributes of the system may be information related to the system, such as the height from the ground to the camera, the distance from the camera to the vehicle, and the viewing angle of the camera. In addition, any information that affects facial authentication can be used as the first information. Moreover, the processed facial image for authentication may be generated to reflect the first information. For example, the server 40 on the utilization side may generate the processed facial image for authentication by processing the facial image for authentication by adding wipers, sun visors, A-pillars or front seats.

[0068] Next, Figure 3 This is a block diagram schematically showing an example of the hardware configuration of each of the center server 30 , the user-side server 40 , the association management server 50 , the vehicle ID server 60 , and the settlement server 70 constituting the system 1 according to the embodiment.

[0069] The central server 30 is configured as a computer having a control unit 31, a storage unit 32, and a communication module 33. The central server 30 can be configured as a computer having a processor (CPU, GPU, etc.), a main storage device (RAM, ROM, etc.), and an auxiliary storage device (EPROM, hard disk drive, removable media, etc.). The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that meet the prescribed purpose as described later can be realized. However, some or all of the modules can also be implemented as hardware modules by hardware circuits such as ASIC and FPGA, for example.

[0070] The control unit 31 is a computing unit that realizes various functions of the center server 30 by executing a predetermined program. The control unit 31 can be realized by a hardware processor such as a CPU. In addition, the control unit 31 can be configured to include a RAM, a ROM (Read Only Memory), a cache memory, and the like.

[0071] The storage unit 32 is a means for storing information, and is composed of a storage medium such as a RAM, a magnetic disk, and a flash memory. The storage unit 32 stores programs executed by the control unit 31, data used by the programs, and the like. In addition, the storage unit 32 stores various information. Details will be described later.

[0072] The communication module 33 is a communication interface for connecting the central server 30 to the network. The communication module 33 may be configured to include, for example, a network interface board, a wireless communication interface for wireless communication, etc. The central server 30 can communicate data with other computers (for example, other server devices or user terminals 20) via the communication module 33.

[0073] In addition, the specific hardware configuration of the central server 30 can be appropriately omitted, replaced, and added according to the implementation method. For example, the control unit 31 may include multiple hardware processors. The hardware processor may be composed of a microprocessor, an FPGA, a GPU, etc. In addition, the central server 30 may be composed of multiple computers. In this case, the hardware configuration of each computer may be consistent or inconsistent.

[0074] The user-side server 40 is configured as a computer having a control unit 41, a storage unit 42, and a communication module 43. The user-side server 40 can be configured as a computer having a processor (CPU, GPU, etc.), a main storage device (RAM, ROM, etc.), and an auxiliary storage device (EPROM, hard disk drive, removable media, etc.). The auxiliary storage device stores an operating system (OS), various programs, various tables, etc., and by executing the programs stored therein, various functions (software modules) that meet the prescribed purpose as described later can be realized. However, some or all of the modules can also be implemented as hardware modules by hardware circuits such as ASIC, FPGA, etc. Regarding the control unit 41, storage unit 42 and communication module 43 of the user-side server 40, since the configuration is the same as the control unit 31, storage unit 32 and communication module 33 of the center server 30, the description is omitted.

[0075] A camera 400 and a numeric keypad 500 are connected to the utilization side server 40, and they are controlled by the control unit 41. The camera 400 is pre-configured at a position where it can photograph the driver of the stopped vehicle 10. The camera 400 is, for example, a device that uses a photographing element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor to take pictures. The numeric keypad 500 is pre-configured near the stop position of the vehicle 10 so that the driver of the vehicle 10 can operate it. The numeric keypad 500 is configured to be able to input numbers 0 to 9, for example. In addition, the numeric keypad 500 may have a display that displays the input numbers. In addition, the numeric keypad 500 may include a device for inputting, such as a mouse, a keyboard, or a microphone.

[0076] The association management server 50, the vehicle ID server 60, and the settlement server 70 can also be configured as the same computer as the center server 30. The association management server 50 is a device that provides information related to the association between the vehicle ID and the user ID, the timestamp of the association, and the validity period of the association via the network N1. Therefore, the information related to each of the vehicle ID, the user ID, the timestamp, and the validity period is associated and stored in the association management server 50. The timestamp is, for example, the date and time when the association between the user ID and the vehicle ID is performed. The validity period is the validity period of the association between the user ID and the vehicle ID. The association management server 50 is managed by a company or an administrative agency that is neutral to the administrator of the center server 30 and the administrator of the user-side server 40. The information stored in the association management server 50 can be provided by the user via the user terminal 20, for example, or by the center server 30.

[0077] The vehicle ID server 60 is a device that provides information related to each of the vehicle number, vehicle model, and owner of the vehicle 10 associated with the vehicle ID via the network N1. Therefore, the vehicle ID, vehicle number, vehicle model, and information related to each of the owner of the vehicle 10 are associated and stored in the vehicle ID server 60. The vehicle ID server 60 is managed by a company or administrative agency that is neutral to the manager of the center server 30 and the manager of the user-side server 40. The vehicle ID server 60 may be a server managed by an administrative agency that issues number plates, for example. When a user registers the vehicle 10, a number plate may be issued, and its number (vehicle number) and vehicle model and information related to the owner may be stored in the vehicle ID server 60.

[0078] The settlement server 70 is a device that performs settlement processing based on settlement requirements. The settlement server 70 is managed by, for example, a credit card company. As another example, the settlement server 70 may be a server that manages electronic money and points, or a server that manages bank deposits. Settlement information is stored in the settlement server 70. The settlement information may include, for example, information related to establishing an association between a user ID and a credit card number.

[0079] Next, Figure 4 This is a block diagram schematically showing an example of the hardware configuration of each of the user terminal 20 and the vehicle-mounted device 100 according to the present embodiment.

[0080] The user terminal 20 is a small computer such as a smartphone, a mobile phone, a tablet terminal, a personal information terminal, a wearable computer (smart watch, etc.), a personal computer (PC), etc. The user terminal 20 is configured to include a control unit 21, a storage unit 22, a communication module 23, an input / output device 24, and a camera 25.

[0081] The user terminal 20 can be configured as a computer having a processor (CPU, GPU, etc.), a main storage device (RAM, ROM, etc.), and an auxiliary storage device (EPROM, hard disk drive, removable media, etc.) similarly to the center server 30. However, part or all of the functions (software modules) may be implemented as hardware modules by hardware circuits such as ASIC and FPGA, for example.

[0082] The control unit 21 , the storage unit 22 , and the communication module 23 of the user terminal 20 have the same configuration as the control unit 31 , the storage unit 32 , and the communication module 33 of the center server 30 , and thus description thereof will be omitted.

[0083] The input-output device 24 is a unit that accepts input operations performed by the user and presents information to the user. Specifically, the input-output device 24 includes a device for input such as a mouse, a keyboard, or a microphone, and a device for output such as a display and a speaker. The input-output device 24 can be integrally formed by, for example, a touch panel display. The camera 25 is a device that uses a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor or other imaging element for imaging.

[0084] In addition, the specific hardware configuration of the user terminal 20 is the same as that of the center server 30 , and components can be omitted, replaced, or added as appropriate according to the embodiment.

[0085] Next, the vehicle-mounted device 100 is, for example, a computer such as an ECU (Electronic Control Unit), a DCM (Data Communication Module), a head unit, or a navigation system mounted on the vehicle 10. The vehicle-mounted device 100 is configured to include a control unit 11, a storage unit 12, a communication module 13, and an input-output device 14. The control unit 11, the storage unit 12, the communication module 13, and the input-output device 14 of the vehicle-mounted device 100 are the same as the control unit 21, the storage unit 22, the communication module 23, and the input-output device 24 of the user terminal 20, and thus description thereof is omitted.

[0086] Figure 5 1 is a diagram schematically showing an example of the software configuration of the center server 30 according to the embodiment. The control unit 31 is configured to include software modules including a user information acquisition unit 311, an association establishment information acquisition unit 312, a vehicle information acquisition unit 313, and a face image providing unit 314. Each software module can be realized by the control unit 31 (CPU) executing a program stored in the storage unit 32. In addition, the information processing performed by the user information acquisition unit 311, the association establishment information acquisition unit 312, the vehicle information acquisition unit 313, and the face image providing unit 314 below is synonymous with the information processing performed by the control unit 31.

[0087] The storage unit 32 of the center server 30 stores a user information database 321 (hereinafter referred to as user information DB321) and an authentication facial image database 322 (hereinafter referred to as authentication facial image DB322). The user information DB321 and the authentication facial image DB322 are constructed by managing the data stored in the auxiliary storage device by a program of a database management system (DBMS) executed by a processor. These databases are, for example, relational databases. Figure 6 This is a diagram showing an example of the table structure of the user information DB 321 and the authentication facial image DB 322 according to the embodiment. The details will be described later.

[0088] The user information acquisition unit 311 is configured to perform the following processing: receiving the user information and the authentication face image sent from the user terminal 20, and storing the user information and the authentication face image in the user information DB 321 and the authentication face image DB 322. Figure 6 As shown, the user information DB 321 stores the user ID, the user's name, the user's contact information, and information related to the settlement service. The user information is associated with the user ID. The user ID is, for example, the number of a car driver's license. In addition, the user's contact information may include the user's address, telephone number, SNS account, email address, etc. The information related to the settlement service is the information involved in establishing the association between the user and the settlement service, including the information related to the settlement that the user wishes to settle (for example, a credit card number). The user information is pre-registered in the central server 30 by the user via the user terminal 20. In addition, as Figure 6 As shown, the facial image for authentication is associated with the user ID and stored in the facial image DB 322 for authentication. The facial image for authentication is a facial image of the user sent from the user terminal 20 together with the user information, for example, a facial image captured by the user himself using the camera 25 of the user terminal 20. The user information acquisition unit 311 associates the facial image for authentication received from the user terminal 20 with the user ID and stores it in the facial image DB 322 for authentication. In addition, as another example, the facial image DB 322 may store information related to the location where the facial image for authentication is stored.

[0089] The association information acquisition unit 312 requests the association management server 50 to perform a search based on the user ID. The association management server 50 returns information related to each of the vehicle ID, the time stamp, and the validity period associated with the user ID to the center server 30 (hereinafter, also referred to as association information). In addition, the association management server 50 may periodically update the association between the user ID and the vehicle ID. The association information acquisition unit 312 causes the storage unit 32 to store the association information acquired from the association management server 50.

[0090] The vehicle information acquisition unit 313 requests a search based on the vehicle ID to the vehicle ID server 60. The vehicle ID is the vehicle ID acquired from the association establishment management server 50. The vehicle ID server 60 returns information related to each of the vehicle number, vehicle model, and owner of the vehicle 10 associated with the vehicle ID to the central server 30. The vehicle model is, for example, classified so as to be able to determine the attributes of the vehicle 10. The information related to the owner of the vehicle 10 includes, for example, information related to the user ID or name. The vehicle information acquisition unit 313 causes the storage unit 32 to store information related to each of the vehicle number, vehicle model, and owner of the vehicle 10 associated with the vehicle ID acquired from the vehicle ID server 60.

[0091] The facial image providing unit 314 provides the facial image for authentication in response to a request from the user-side server 40. The request from the user-side server 40 includes information related to the user ID or the vehicle ID, and the facial image providing unit 314 extracts the facial image for authentication corresponding to the user ID or the vehicle ID from the facial image DB 322 and transmits it to the user-side server 40.

[0092] Figure 7 4 is a diagram schematically showing an example of the software configuration of the user-side server 40 according to the embodiment. The control unit 41 is configured to include software modules such as a facial image acquisition unit 411, a vehicle information acquisition unit 412, a user ID acquisition unit 413, an authentication information acquisition unit 414, an execution unit 415, a second authentication unit 416, a first information generation unit 417, and a facial image processing unit 418. Each software module can be realized by the control unit 41 (CPU) executing a program stored in the storage unit 42. In addition, the information processing performed by the facial image acquisition unit 411, the vehicle information acquisition unit 412, the user ID acquisition unit 413, the authentication information acquisition unit 414, the execution unit 415, the second authentication unit 416, the first information generation unit 417, and the facial image processing unit 418 is synonymous with the information processing performed by the control unit 41.

[0093] The storage unit 42 of the user-side server 40 stores a first information database 421 (hereinafter referred to as the first information DB421) and a second authentication information database 423 (hereinafter referred to as the second authentication information DB423). The first information DB421 and the second authentication information DB423 are constructed by managing the data stored in the auxiliary storage device by a program of a database management system (DBMS) executed by a processor. These databases are, for example, relational databases. Figure 8 This is a diagram showing an example of the table structure of the first information DB 421 and the second authentication information DB 423 according to the embodiment. The details will be described later.

[0094] The facial image acquisition unit 411 acquires an image captured by the camera 400 when the vehicle 10 is at a predetermined position. The camera 400 is pre-arranged at a position where it can capture an image of the driver of the vehicle 10. The facial image acquisition unit 411, for example, always monitors the image captured by the camera 400 and detects that the vehicle 10 approaches the predetermined position by image analysis. Then, the image of the user captured at this time is acquired as the user facial image.

[0095] The vehicle information acquisition unit 412 acquires information related to the vehicle 10 existing at a predetermined position. In this case, for example, the camera 400 has a wireless communication function, and the camera 400 can wirelessly communicate with the vehicle 10 to acquire information related to the vehicle 10 (for example, the vehicle number). In addition, as another example, the vehicle number can be read by analyzing the image of the vehicle 10 captured by the camera 400.

[0096] The user ID acquisition unit 413 requests the association establishment management server 50 to search based on the vehicle ID. At this time, the user ID acquisition unit 413 inquires the association establishment management server 50 whether there is a valid association establishment relationship corresponding to the vehicle ID at the target date and time. If there is a valid association establishment relationship corresponding to the vehicle ID at the target date and time, the association establishment management server 50 returns the user ID associated with the vehicle ID to the use-side server 40. The user ID acquisition unit 413 causes the storage unit 42 to store the user ID acquired from the association establishment management server 50. In addition, the user ID acquisition unit 413 can acquire information related to the user ID from the vehicle-mounted device 100 or the user terminal 20.

[0097] The authentication information acquisition unit 414 requests a search based on the user ID to the center server 30. The center server 30 returns information related to various authorities associated with the user ID and the authentication face image to the user-side server 40. The various authorities include, for example, information related to the settlement service associated with the user ID.

[0098] If the facial authentication is successful and the authority is valid, the execution unit 415 requests the settlement server 70 to settle the account. The execution unit 415 performs facial authentication by comparing the authentication facial image obtained from the center server 30 or the processed authentication facial image generated by the facial image processing unit 418 described later with the user's facial image obtained from the camera 400. The execution unit 415 performs facial authentication based on the processed authentication facial image when the first information exists, and performs facial authentication based on the authentication facial image when the first information does not exist. The settlement server 70 to which the settlement is requested is selected by the execution unit 415 based on the information related to the various authorities associated with the user ID.

[0099] If the facial authentication is not successful, the second authentication unit 416 performs a second authentication other than facial authentication. The second authentication is an authentication based on a method other than facial authentication. The second authentication unit 416 may, for example, report to the user that a password has been entered on the numeric keypad 500. The second authentication unit 416 performs authentication by, for example, comparing the number entered on the numeric keypad 500 with the number stored in the second authentication information DB 324 of the storage unit 32. Figure 8 As shown, the second authentication information is associated with the user ID and stored in the second authentication information DB. In addition, the second authentication is not limited to this, as long as it is fingerprint authentication, voiceprint authentication, SMS authentication, telephone-based authentication, etc., based on authentication methods other than facial authentication. If the authentication based on the second authentication unit 416 is established and the authority is valid, the execution unit 415 requests settlement to the settlement server 70.

[0100] When the facial authentication fails and the second authentication succeeds, the first information generating unit 417 generates the first information. The first information is, for example, information related to the color of the front windshield of the vehicle 10. The first information generating unit 417 generates the first information by extracting the information related to the color of the front windshield from the user's facial image using a known technique. Furthermore, the first information generating unit 417 associates the generated first information with the user ID and the vehicle ID and stores them in the first information DB 421. Figure 8As shown, the first information DB 421 stores information related to the user ID, the vehicle ID, and the first information. In the case where the same user owns multiple vehicles 10, there are multiple records of the same user ID but different vehicle IDs. In addition, the first information can be generated by the control unit 31 of the central server 30 instead of the first information generation unit 417 of the utilization side server 40. In this case, the user's facial image is associated with the user ID and the vehicle ID based on whether the facial authentication is successful and sent to the central server 30. In addition, as another example, the first information generation unit 417 can generate the first information based on the user's facial image and send the first information to the central server 30. Moreover, the first information can be stored in the storage unit 32 of the central server 30.

[0101] The facial image processing unit 418 generates a processed facial image for authentication by processing the facial image for authentication obtained from the center server 30 based on the first information. Figure 8 As shown, the relationship between the user ID and the vehicle ID and the first information is stored in the first information DB 421. If the facial image processing unit 418 obtains the facial image for authentication, it extracts the first information corresponding to the user ID and the vehicle ID from the first information DB 421. Moreover, the processed facial image for authentication is generated by reflecting the first information on the facial image for authentication. For example, when the color of the front windshield of the vehicle 10 is stored as the first information in the first information DB 421, the facial image processing unit 418 generates the processed facial image for authentication by reflecting the color of the front windshield of the vehicle 10 on the facial image for authentication. That is, the processed facial image for authentication is generated in a manner corresponding to the image obtained by taking a picture of the user's face from the outside of the vehicle 10. The facial image processing unit 418 associates the generated processed facial image for authentication with the vehicle ID and the user ID and stores them in the storage unit 42.

[0102] Fig. 9 1 is a diagram schematically showing an example of the software configuration of the vehicle-mounted device 100 according to the embodiment. In the present embodiment, the control unit 11 of the vehicle-mounted device 100 is configured to include a software module called a vehicle ID transmitter 111. The software module can be implemented by the control unit 11 (CPU) executing a program stored in the storage unit 12. In addition, the information processing performed by the vehicle ID transmitter 111 below is synonymous with the information processing performed by the control unit 11.

[0103] The vehicle ID transmitter 111 transmits the vehicle ID stored in the storage unit 12 to the user-side server 40 or the camera 400 in response to a request from the user-side server 40 or the camera 400. If the vehicle information acquisition unit 412 of the user-side server 40 acquires the vehicle number by reading it, the vehicle ID transmitter 111 can be omitted.

[0104] Fig.10 1 is a diagram schematically showing an example of the software configuration of the user terminal 20 involved in the embodiment. The control unit 21 of the user terminal 20 is configured to include software modules such as a photographing unit 211 and a user information sending unit 212. Each software module can be implemented by the control unit 21 (CPU) executing a program stored in the storage unit 22. In addition, the information processing performed by the photographing unit 211 and the user information sending unit 212 below can be synonymous with the information processing performed by the control unit 21.

[0105] The imaging unit 211 captures the user through the camera 25. The imaging unit 211 captures the user when the user inputs a predetermined input to the input / output device 24. At this time, the user adjusts the angle of the camera 25 so as to capture the user's own face.

[0106] The user information sending unit 212 sends the user information input by the user via the input / output device 24 and the user's facial image captured by the capturing unit 211 to the center server 30. In addition, when the second authentication is SMS authentication or telephone-based authentication, the user information sending unit 212 may send the information input by the user to the input / output device 24 in the SMS authentication or telephone-based authentication to the utilization-side server 40.

[0107] Next, the user registration process in the center server 30 will be described. Fig.11 FIG. 1 is a flowchart of a user registration process in the center server 30 according to the embodiment. Fig.11 In addition, Fig.11 The processes shown are executed by the software modules of the control unit 31 , but in the following flowcharts, the control unit 31 is described as executing the processes.

[0108] In step S101, the control unit 31 determines whether a user registration request is received from the user terminal 20. An application capable of performing user registration may be installed in the user terminal 20. The user sends a user registration request from the user terminal 20 to the center server 30 by using the application to perform a predetermined input and take a picture of the face. In addition, as another example, a user registration request may be sent from the user terminal 20 to the center server 30 by accessing a web page capable of performing user registration from the user terminal 20, and the user performs a predetermined input and uploads a facial image for authentication on the web page. If an affirmative determination is made in step S101, the routine proceeds to step S102, and if a negative determination is made, the routine ends.

[0109] In step S102, the control unit 31 obtains user information and a facial image for authentication. Since the user information and the facial image for authentication are sent from the user terminal 20 to the center server 30 together with the user registration request, the control unit 31 obtains these user information and the facial image for authentication. The user information includes a user ID. Then, in step S103, the control unit 31 performs user registration processing, stores the user information in the user information DB 321, and associates the facial image for authentication with the user ID and stores it in the facial image DB 322 for authentication.

[0110] Next, the processing of providing the facial image for authentication in the center server 30 will be described. Fig.12 FIG. 1 is a flowchart of the process of providing the authentication facial image in the center server 30 according to the embodiment. Fig.12 In addition, Fig.12 The processes shown are executed by the software modules of the control unit 31 , but in the following flowcharts, the control unit 31 is described as executing the processes.

[0111] In step S201, the control unit 31 determines whether a search based on the user ID is requested from the user-side server 40. The request includes a request to send information related to various permissions associated with the user ID and a facial image for authentication to the user-side server 40. If an affirmative determination is made in step S201, the routine proceeds to step S202, and if a negative determination is made, the routine ends.

[0112] In step S202, the control unit 31 obtains the authentication facial image stored in the authentication facial image DB 322. The authentication facial image obtained at this time is a facial image received from the user terminal 20 when the user registered. The control unit 31 extracts the authentication facial image associated with the user ID obtained in step S201 from the authentication facial image DB 322.

[0113] In step S203, the control unit 31 obtains the user's authority from the user information DB 321. In this embodiment, information related to the settlement service is obtained as the information related to the user's authority. In step S204, the control unit 31 sends the authentication facial image and the information related to the user's authority to the user-side server 40. The information related to the user's authority sent at this time includes, for example, information related to the settlement company (credit card company) registered by the user and information related to the credit card number.

[0114] Next, the face authentication process in the user-side server 40 will be described. Fig.13FIG. 4 is a flowchart of the face authentication process in the user-side server 40 according to the embodiment. Fig.13 In addition, Fig.13 The processes shown are executed by the software modules of the control unit 41 , but in the following flowcharts, it is described that the control unit 41 executes the processes.

[0115] In step S301, the control unit 41 determines whether the vehicle 10 is at a predetermined position. The predetermined position is a position where the face of the driver of the vehicle 10 can be photographed by the camera 400. The predetermined position may be, for example, a position where goods are ordered without getting off the vehicle or a position where goods are received. In addition, the predetermined position is not limited to a position where the vehicle 10 stops, but may also be a position where the vehicle 10 passes while driving. If an affirmative determination is made in step S301, the routine proceeds to step S302, and if a negative determination is made, the routine ends.

[0116] In step S302, the control unit 41 obtains a user face image by photographing the driver of the vehicle 10 using the camera 400. The control unit 41 causes the storage unit 42 to store the obtained user face image. In step S303, the control unit 41 obtains a vehicle ID from the vehicle 10. For example, the vehicle ID may be transmitted from the vehicle-mounted device 100 to the camera 400 using short-range wireless communication. As another example, the number plate of the vehicle 10 may be photographed using the camera 400, and the vehicle number may be obtained by image analysis.

[0117] In step S304, the control unit 41 obtains the user ID corresponding to the vehicle ID from the association establishment management server 50. At this time, the control unit 41 inquires the association establishment management server 50 whether there is a valid association establishment relationship corresponding to the vehicle ID at the target date and time. In this case, the control unit 41 sends the vehicle ID and the information related to the target date and time to the association establishment management server 50. For example, if the user has not registered the user ID, there is no valid association establishment relationship corresponding to the vehicle ID. Therefore, in the association establishment management server 50, it is determined whether there is a user ID corresponding to the vehicle ID. In addition, for example, there is also a case where an effective period is set for the association establishment of the vehicle ID and the user ID. In this case, if the effective period of the association establishment of the vehicle ID and the user ID is exceeded, there is no valid association establishment relationship corresponding to the vehicle ID. Therefore, in the association establishment management server 50, it is determined whether it is within the effective period of the association establishment of the vehicle ID and the user ID. In addition, for example, when a valid association establishment relationship between the vehicle ID and the user ID is required at a specific date and time, a timestamp is used to determine whether there is a valid association establishment relationship between the vehicle ID and the user ID at the specific date and time. When there is a valid association relationship between the vehicle ID and the user ID at the target date and time, the association management server 50 sends the user ID to the use-side server 40. On the other hand, when there is no valid association relationship between the vehicle ID and the user ID at the target date and time, the association management server 50 sends the information of the content to the use-side server 40. In this case, the control unit 41 cannot obtain the user ID corresponding to the vehicle ID. In addition, as another example, the control unit 41 can obtain the user ID from the vehicle-mounted device 100 or the user terminal 20. For example, the user ID can be registered in the vehicle-mounted device 100 by associating it with the vehicle ID in advance, and the user ID can be sent to the camera 400 together with the vehicle ID.

[0118] In step S305, the control unit 41 determines whether the user ID was obtained in step S304. If a positive determination is made in step S305, the routine proceeds to step S306, and if a negative determination is made, the routine ends. In addition, if a negative determination is made in step S305, the fact that settlement cannot be performed may be reported to the user. For example, the use-side server 40 may be configured to display the content on a sign or play the sound of the content from a speaker.

[0119] In step S306, the control unit 41 obtains the facial image for authentication and information related to various permissions from the central server 30. At this time, the control unit 41 sends a request for search based on the user ID to the central server 30. In response to the request, the central server 30 sends the facial image for authentication and information related to various permissions corresponding to the user ID to the user-side server 40. In addition, the control unit 41 receives the facial image for authentication and information related to various permissions of the user from the central server 30.

[0120] In step S307, the control unit 41 determines whether the first information corresponding to the user exists. The control unit 41 accesses the first information DB 421 to determine whether there is a record including the user ID corresponding to the user. If the record exists, the control unit 41 determines that the first information corresponding to the user exists. On the other hand, if the record does not exist, the control unit 41 determines that the first information corresponding to the user does not exist. If a positive determination is made in step S307, the process proceeds to step S308, and if a negative determination is made, the process proceeds to step S310.

[0121] In step S308, the control unit 41 generates a processed authentication facial image by processing the authentication facial image using the first information. For example, the control unit 41 processes the authentication facial image obtained from the center server 30 so that the processed authentication facial image becomes a facial image of the user when the user is photographed through the front windshield. This processing can be performed using a known technique. The control unit 41 causes the storage unit 42 to store the generated processed authentication facial image.

[0122] In step S309, the control unit 41 performs facial authentication by comparing the user facial image obtained in step S302 with the processed authentication facial image generated in step S308. On the other hand, in step S310, the control unit 41 performs facial authentication by comparing the user facial image obtained in step S302 with the authentication facial image obtained in step S306. These facial authentications can adopt well-known technologies. The method of facial authentication is not particularly limited and can be appropriately determined according to the embodiment.

[0123] In step S311, the control unit 41 determines whether the facial authentication is successful. If an affirmative determination is made in step S311, the process proceeds to step S315, and if a negative determination is made, the process proceeds to step S312.

[0124] In step S312, the control unit 41 performs the second authentication. For example, the control unit 41 reports to the user that a password has been entered into the numeric keypad 500. For example, the utilization side server 40 can be configured to display the content on a sign or to play the sound of the content from a speaker. Moreover, the control unit 41, which has obtained the password entered by the user via the numeric keypad 500, performs the second authentication by comparing it with the password corresponding to the user ID stored in the second authentication information DB. If the two passwords are consistent, the second authentication is established. Moreover, in step S313, the control unit 41 determines whether the second authentication is established. When an affirmative determination is made in step S313, the routine proceeds to step S314, and when a negative determination is made, the routine ends. In addition, when a negative determination is made in step S313, the control unit 41 can report to the user that settlement cannot be performed.

[0125] In step S314, the control unit 41 causes the first information DB 421 to store the first information. The first information may be information extracted from the user's facial image, such as information related to the color of the front windshield. As another example, the first information may be the user's facial image. The control unit 41 may associate the first information with the vehicle ID and the user ID and store them in the first information DB 421.

[0126] In step S315, the control unit 41 determines whether the user can make a payment in the settlement server 70. The control unit 41 determines whether the user's authority is valid based on the information related to various authorities obtained from the center server 30. In this case, the control unit 41 may inquire the settlement server 70 whether the user can make a payment in the settlement server 70. If an affirmative determination is made in step S315, the routine proceeds to step S316, and if a negative determination is made, the routine ends.

[0127] In step S316 , the control unit 41 requests the settlement server 70 to make a settlement request. In other words, the user's authority is exercised. At this time, information related to the user ID and the amount to be settled is sent to the settlement server 70 .

[0128] In this way, when the facial authentication by the control unit 41 is not successful, the first information is collected. In addition, since the facial authentication is performed based on the processed authentication facial image reflecting the first information in the next facial authentication, the accuracy of facial authentication can be improved. That is, the difference between the user's appearance in the user's facial image obtained during facial authentication and the user's appearance in the processed authentication facial image can be reduced. Therefore, the accuracy of facial authentication of the user riding in the vehicle 10 can be improved.

[0129] <Second embodiment>

[0130] In this embodiment, when a plurality of vehicles 10 of the same model are associated with the same user ID, the first information or the processed facial image for authentication that has been generated is used. Even if a vehicle 10 is assigned a vehicle ID different from the vehicle 10 for which the first information has been generated, as long as the vehicle 10 is of the same model, the first information that has been generated can be used. That is, as long as the model is the same, since it can be assumed that the attributes of the vehicle 10 have the same effect on the user's facial image, the first information or the processed facial image for authentication can be used as long as the model and user ID are the same. In addition, even if the user ID is different, as long as the model is the same, since it can be assumed that the attributes of the vehicle 10 have the same effect on the user's facial image, the first information can be used for other users.

[0131] When the first information generating unit 417 of the user-side server 40 causes the first information DB 421 to store the first information, the first information is stored in association with the user ID, the vehicle ID, and the vehicle model. Fig.14 4 is a diagram showing an example of a table structure of the first information DB 421 according to the embodiment. The first information DB 421 stores the relationship between the user ID, the vehicle ID, the vehicle model, and the first information. The information related to the vehicle model is included in the information obtained by the vehicle information acquisition unit 412 from the vehicle ID server 60.

[0132] Furthermore, before generating the processed authentication facial image, the facial image processing unit 418 of the user-side server 40 determines whether a record of the same vehicle model already exists in the first information DB 421. If a record of the same vehicle model already exists, the facial image processing unit 418 of the user-side server 40 generates the processed authentication facial image based on the first information. In this way, if the first information has already been generated for the vehicle 10 of the same vehicle model, the first information is not newly generated for the same vehicle model and is used.

[0133] Next, the face authentication process in the user-side server 40 will be described. Fig.15 FIG. 4 is a flowchart of the face authentication process in the user-side server 40 according to the embodiment. Fig.15 In addition, Fig.15 The processing shown in the figure is executed by each software module of the control unit 41, but in the following flowchart, it is described that the control unit 41 executes the processing. Fig.13 The same processing steps are marked with the same reference numerals and the description thereof is omitted. Fig.13 The same, so Fig.15 The processing after step S306 is shown in FIG.

[0134] exist Fig.15 In the routine shown, if the processing of step S306 is completed, the process proceeds to step S401. In step S401, the control unit 41 obtains the vehicle information associated with the vehicle ID from the vehicle ID server 60. The vehicle information includes information that can identify the vehicle, such as information related to the model, name, body number or the number of the car registration plate (vehicle number). In addition, the vehicle information may include information that can determine the height, size, color of the front windshield, color of the side window glass, position of the wiper, position of the sun visor or position of the A-pillar, etc. of the vehicle. In addition, the vehicle information may include information related to the owner of the vehicle 10.

[0135] In step S402, the control unit 41 determines the vehicle model based on the vehicle information. Information related to the vehicle model is included in the vehicle information. In step S403, the control unit 41 determines whether there is first information corresponding to the same vehicle model. At this time, the control unit 41 accesses the first information DB421 to determine whether there is a record storing the same vehicle model as the vehicle model determined in step S402. If the record exists, the control unit 41 determines that there is first information corresponding to the same vehicle model. On the other hand, if the record does not exist, the control unit 41 determines that there is no first information corresponding to the same vehicle model. When an affirmative determination is made in step S403, the process proceeds to step S404, and when a negative determination is made, the process proceeds to step S310.

[0136] In step S404, the control unit 41 generates a processed authentication facial image by processing the authentication facial image using the first information. For example, the control unit 41 processes the authentication facial image obtained from the center server 30 so that the processed authentication facial image becomes a facial image of the user when the user is photographed through the front windshield. This processing can use a known technique.

[0137] In this way, since a processed authentication facial image pre-matched with the vehicle type can be generated by using the first information, the accuracy of facial authentication can be improved. In addition, the load of the process of generating the first information can be reduced.

[0138] <Third embodiment>

[0139] In this embodiment, when the first information includes information based on vehicle attributes, information based on user attributes, and information based on system attributes, the first information is used. When the first information includes information based on vehicle attributes, for example, the first information may be used for other users riding in the vehicle 10 of the same attribute. For example, when the facial recognition of a certain user fails due to information based on vehicle attributes, the first information generated at this time may be used for facial recognition of other users riding in the same vehicle model.

[0140] In addition, when the first information includes information based on user attributes, for example, the first information may be used when the same user is riding in other vehicles. User attributes are attributes associated with the user, such as the user wearing sunglasses or glasses, the user wearing a mask, the user wearing a hat, and the user having a beard. For example, it is assumed that the user owns multiple vehicles 10. The first information generated when facial recognition is not established due to information based on user attributes when riding in a certain vehicle can be used for facial recognition when riding in other vehicles as long as the same user is used.

[0141] Furthermore, in the case where the first information includes information based on system attributes, as long as the system attributes are the same, the first information can be used for other users or other vehicles. System attributes are, for example, attributes related to the system 1, such as the installation position of the camera 400, the viewing angle of the camera 400, and the distance from the camera 400 to the vehicle 10. When facial authentication is performed in the same place, the system attributes have the same effect on facial authentication regardless of the user or the vehicle 10. In addition, it is not limited to the same place. Even if the places are different, the first information can be used if the relative angle between the camera 400 and the vehicle 10 is equal.

[0142] For example, if the installation position of the camera 400 deviates from the front of the user, the angle of the user's face reflected in the user's facial image may change from a state facing the front to a state tilted upward, downward, leftward, or rightward. Therefore, the control unit 41 generates a processed facial image for authentication in a manner that corrects the tilt of the face according to the installation position of the camera 400, thereby improving the accuracy of facial authentication. Moreover, the installation position of the camera 400 is the same regardless of the user or the vehicle 10.

[0143] In addition, the face of the user reflected in the user face image may be deformed depending on the viewing angle of the camera 400. Therefore, the control unit 41 generates a processed authentication face image in a manner that corrects the deformation according to the viewing angle of the camera 400, thereby improving the accuracy of face authentication. Moreover, the viewing angle of the camera 400 is the same regardless of the user or the vehicle 10.

[0144] In addition, the face of the user shown in the user face image may be deformed depending on the distance from the camera 400 to the vehicle 10. Therefore, the control unit 41 generates a processed authentication face image in a manner that corrects the deformation according to the distance between the camera 400 and the vehicle 10, thereby improving the accuracy of face authentication. Moreover, the distance from the camera 400 to the vehicle 10 is the same regardless of the user or the vehicle 10.

[0145] In this way, the system attributes are the same regardless of the user or the vehicle 10. Therefore, when the first information includes information based on the system attributes, the first information can be used when performing facial authentication of other users. In addition, when the first information includes information based on the system attributes, the first information can be used when the same user rides in different vehicles 10.

[0146] When generating the first information, the first information generating unit 417 of the utilization side server 40 determines whether the first information corresponds to the information based on the attributes of the vehicle 10, the information based on the attributes of the user, or the information based on the attributes of the system 1. The first information generating unit 417 can make the determination, for example, by image analysis. In addition, as another example, the first information generating unit 417 can make the determination using a learned machine learning model. The machine learning model can be a model that outputs the information based on the attributes of the vehicle 10, the information based on the attributes of the user, or the information based on the attributes of the system 1 when a user's facial image is input. Moreover, when the first information generating unit 417 causes the first information DB 421 to store the first information, it also stores information that can determine which of the vehicle attributes, user attributes, and system attributes the first information corresponds to.

[0147] Fig.16This is a diagram showing an example of the table structure of the first information DB 421 involved in the implementation method. The first information DB 421 stores the relationship between the user ID, vehicle ID, vehicle model, camera position, vehicle attributes, user attributes, system attributes, and the first information. The vehicle model is shown as an example of a vehicle attribute. The camera position is shown as an example of an attribute of the system 1. The camera position can be represented by the distance and angle from the ground to the camera 400, for example. In the respective fields of the vehicle attribute, user attribute, and system attribute, "1" is stored in the case of correspondence, and "0" is stored in the case of non-correspondence. For example, in Fig.16 In the first information field of the record in which "1" is stored in the vehicle attribute, information according to the vehicle attribute is stored.

[0148] Next, the face authentication process in the user-side server 40 will be described. Fig.17 FIG. 4 is a flowchart of the face authentication process in the user-side server 40 according to the embodiment. Fig.17 In addition, Fig.17 The processing shown is executed by each software module of the control unit 41, but in the following flowchart, it is described that the control unit 41 executes the processing. Fig.13 or Fig.15 The same processing steps are marked with the same reference numerals and the description thereof is omitted. Fig.15 The same, so Fig.17 The processing after step S401 is shown in FIG.

[0149] exist Fig.17 In the illustrated routine, when the processing of step S401 is completed, the process proceeds to step S501. In step S501, the control unit 41 determines the vehicle attributes. The vehicle attributes are included in the vehicle information acquired in step S401.

[0150] In step S502 , the control unit 41 determines the system properties. Information on the system properties may be measured in advance for each camera 400 , for example, and may be stored in the storage unit 42 .

[0151] In step S503, the control unit 41 determines whether the first information corresponding to the same vehicle attribute exists in the first information DB 421. That is, the control unit 41 determines whether there is first information corresponding to the vehicle attribute that can be used. The control unit 41 refers to the first information DB 421 to determine whether there is a record with the same information related to the vehicle model and "1" stored in the vehicle attribute field. If a positive determination is made in step S503, the process proceeds to step S506, and if a negative determination is made, the process proceeds to step S504.

[0152] In step S504, the control unit 41 determines whether the first information corresponding to the same user attribute exists in the first information DB 421. That is, the control unit 41 determines whether there is a usable first information corresponding to the user attribute. The control unit 41 refers to the first information DB 421 to determine whether there is a record with the same user ID and "1" stored in the user attribute field. If a positive determination is made in step S504, the process proceeds to step S506, and if a negative determination is made, the process proceeds to step S505.

[0153] In step S505, the control unit 41 determines whether the first information corresponding to the same system attribute exists in the first information DB 421. That is, the control unit 41 determines whether there is first information corresponding to the system attribute that can be used. The control unit 41 refers to the first information DB 421 to determine whether there is a record in which the information related to the position of the camera is the same and "1" is stored in the system attribute field. If a positive determination is made in step S505, the process proceeds to step S506, and if a negative determination is made, the process proceeds to step S310.

[0154] In step S506, the control unit 41 generates a processed authentication facial image by processing the authentication facial image using the first information. For example, the control unit 41 generates the processed authentication facial image based on at least one of the vehicle attribute, the user attribute, and the system attribute.

[0155] In addition, Fig.17 In the routine shown, when a positive determination is made in step S313, the process proceeds to step S507. In step S507, the control unit 41 causes the first information DB 421 to store the first information. The control unit 41 causes the first information DB 421 to associate the first information with the vehicle ID and the user ID and store the first information. In addition, the control unit 41 determines the corresponding attributes among the vehicle attributes, the user attributes, and the system attributes, and stores "1" in the corresponding fields among the vehicle attributes, the user attributes, and the system attributes of the first information DB 421.

[0156] As described above, according to this embodiment, by generating a processed facial image for authentication based on information based on at least one of vehicle attributes, user attributes, and system attributes, past information can be used. Therefore, the accuracy of facial authentication can be improved. In addition, the load of generating the first information can be reduced.

[0157] <Fourth embodiment>

[0158] In the present embodiment, the facial image of the user is further acquired when the facial authentication is not established. The re-acquisition of the facial image of the user begins after the facial authentication is not established, for example, during the second authentication. That is, the re-acquisition of the facial image of the user and the second authentication are performed in parallel. As a result, the image of the face of the user who has performed the second authentication can be retained as a record. Therefore, evidence in the case of illegal use can be retained. In addition, it can also be considered that the facial image of the user captured later than the facial image of the user captured for the first time is easier to generate the first information. Therefore, the accuracy of the first information can be improved. Moreover, by capturing the facial image of the user in parallel with the second authentication, the time required for processing can be shortened compared to capturing the facial image of the user after the second authentication is established. In addition, the re-acquired facial image of the user does not necessarily need to be used in the generation of the first information. For example, the re-acquired user image as evidence can be simply stored in the storage unit 42. In addition, as another example, the first information can be generated using only the re-acquired facial image of the user.

[0159] For example, when Fig.13 After a negative determination is made in step S311 of the routine shown, the control unit 41 reacquires the user's facial image before starting the process of step S312 or step S313. The reacquired user's facial image is associated with the user ID and the vehicle ID in a manner that the user knows that it is a reacquired image and is stored in the storage unit 42. In this case, the user's facial image may be stored only as evidence or may be used when generating the first information in step S314.

[0160] Furthermore, in the case where the numeric keypad 500 is disposed outside the vehicle 10, it can also be considered that the user extends his face outside the vehicle and temporarily gets off the vehicle 10 when entering the password on the numeric keypad 500. In this way, the user's facial image can be captured without being affected by the vehicle's attributes. In this case, although information based on the vehicle's attributes cannot be obtained, information based on the user's attributes and information based on the system's attributes can be obtained with higher accuracy. Therefore, since it becomes easy to determine whether facial authentication is not established due to any of the vehicle's attributes, user's attributes, and system's attributes, the accuracy of the first information can be improved. In addition, facial authentication can be performed again based on the user's facial image that is reacquired without being affected by the vehicle's attributes. By performing facial authentication again in parallel with the second authentication, the authentication accuracy can be improved.

[0161] <Other embodiments>

[0162] The above-mentioned implementation is only an example, and the present disclosure can be implemented by appropriately changing it within the scope of its gist. As long as no technical contradiction occurs, the processing and means described in the present disclosure can be freely combined for implementation. In addition, the processing described as being performed by one device can be shared and executed by multiple devices. Alternatively, the processing described as being performed by different devices can also be executed by one device. In a computer system, it is possible to flexibly change which hardware composition (server composition) is used to implement each function.

[0163] For example, one server may have the functions of two or more of the central server 30, the user-side server 40, the association establishment management server 50, the vehicle ID server 60, and the settlement server 70. For example, the central server 30 and the user-side server 40 may be one server. Similarly, the central server 30 and the association establishment management server 50 or the vehicle ID server 60 may also be one server. At least a part of the processing of the central server 30 may be executed on the user-side server 40. At least a part of the processing of the user-side server 40 may be executed on the central server 30.

[0164] In addition, the first information may be generated by the control unit 31 of the central server 30. In addition, the processed authentication facial image may be generated by the control unit 31 of the central server 30. In this case, when the authentication is not established in the user-side server 40, the user-side server 40 may send the user's facial image to the central server 30. Furthermore, the processed authentication facial image generated in the central server 30 may be sent from the central server 30 to the user-side server 40.

[0165] In the above-described embodiment, when facial authentication is successful, the settlement server 70 is requested to settle the transaction. However, the transaction is not limited to settlement, and the user-side server 40 can execute various rights of the user.

[0166] The present disclosure can also be implemented by supplying a computer program installed with the functions described in the above-mentioned embodiments to a computer, and having one or more processors of the computer read out the program and execute it. Such a computer program can be provided to the computer via a non-temporary computer-readable storage medium that can be connected to the system bus of the computer, or can be provided to the computer via a network. Non-temporary computer-readable storage media include, for example, any type of disks such as magnetic disks (floppy (registered trademark) disks, hard disk drives (HDDs), etc.), optical disks (CD-ROMs, DVD disks, Blu-ray disks, etc.), read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic cards, flash memories, optical cards, and any type of medium suitable for storing electronic commands.

[0167] Description of reference numerals:

[0168] 1…system; 10…vehicle; 20…user terminal; 30…central server; 31…control unit; 32…storage unit; 40…utilization side server; 41…control unit; 42…storage unit; 50…association establishment management server; 60…vehicle ID server; 70…settlement server; 100…vehicle-mounted device; 400…camera.

Claims

1. An information processing device, wherein: A control unit is provided to execute the following processing: Obtaining identification information of a user of the vehicle, i.e., user identification information; acquiring an authentication facial image, the authentication facial image being an image associated with the user identification information and being an authentication facial image of a user of the vehicle; acquiring a user facial image, the user facial image being an image generated by photographing a user of the vehicle using a camera disposed outside the vehicle; performing facial authentication based on comparison of the authentication facial image with the user facial image; and In response to the failure of the facial authentication, first information is acquired from the user's facial image as information to be used in the next facial authentication or later.

2. The information processing device according to claim 1, wherein: The control unit performs a second authentication other than the facial authentication in response to the facial authentication not being successful. And based on the establishment of the second authentication, the first information is obtained from the user's facial image.

3. The information processing device according to claim 1, wherein: The control unit performs the facial authentication based on the presence of the first information, based on a comparison between the processed authentication facial image, which is the authentication facial image reflecting the first information, and the user facial image.

4. The information processing device according to claim 3, wherein: The control unit generates the processed authentication facial image by processing the authentication facial image based on the first information.

5. The information processing device according to claim 1, wherein: The control unit causes the storage unit to store the first information so as to be usable in facial authentication when the user uses the vehicle.

6. The information processing device according to claim 1, wherein: The first information includes at least one of information based on an attribute of a first user, information based on an attribute of a first vehicle, and information based on an attribute of a first system that performs the facial authentication.

7. The information processing device according to claim 6, wherein: The control unit causes the storage unit to store the first information so as to be usable for facial authentication of the first user, based on the first information including information according to the attribute of the first user.

8. The information processing device according to claim 6, wherein: The control unit causes the storage unit to store the first information so that the first information can be used for facial authentication of a second user of a second vehicle having the same attribute as the first vehicle, based on the first information including the information according to the attribute of the first vehicle.

9. The information processing device according to claim 6, wherein: The control unit causes the storage unit to store the first information so that the first information can be used for facial authentication of a second user in a second system that performs the same facial authentication as the first system, based on the first information including information according to the attribute of the first system.

10. The information processing device according to claim 2, wherein: The control unit obtains the user's facial image again after the facial authentication is not completed and before the second authentication is completed in accordance with the second authentication.

11. An information processing method, wherein: The following processing is performed by the computer: Obtaining identification information of a user of the vehicle, i.e., user identification information; acquiring an authentication facial image, the authentication facial image being an image associated with the user identification information and being an authentication facial image of a user of the vehicle; acquiring a user facial image, the user facial image being an image generated by photographing a user of the vehicle using a camera disposed outside the vehicle; performing facial authentication based on comparison of the authentication facial image with the user facial image; and In response to the failure of the facial authentication, first information is acquired from the user's facial image as information to be used in the next facial authentication or later.

12. The information processing method according to claim 11, wherein: The computer performs a second authentication other than the facial authentication in response to the facial authentication not being successful. And based on the establishment of the second authentication, the first information is obtained from the user's facial image.

13. The information processing method according to claim 11, wherein: The computer performs the facial authentication based on the presence of the first information and comparison between the processed facial authentication image, which is the authentication facial image reflecting the first information, and the user facial image.

14. The information processing method according to claim 13, wherein: The computer generates the processed authentication facial image by processing the authentication facial image based on the first information.

15. The information processing method according to claim 11, wherein: The computer causes the storage unit to store the first information so as to be usable for facial authentication when the user uses the vehicle.

16. The information processing method according to claim 11, wherein: The first information includes at least one of information based on an attribute of a first user, information based on an attribute of a first vehicle, and information based on an attribute of a first system that performs the facial authentication.

17. The information processing method according to claim 16, wherein: The computer causes the storage unit to store the first information so that the first information can be used for facial authentication of the first user, based on the fact that the first information includes information according to the attribute of the first user.

18. The information processing method according to claim 16, wherein: The computer causes the storage unit to store the first information so that it can be used for facial authentication of a second user of a second vehicle having the same attribute as the first vehicle, based on the first information including information according to the attribute of the first vehicle.

19. The information processing method according to claim 16, wherein: The computer causes the storage unit to store the first information so that the first information can be used for facial authentication of a second user in a second system that performs the same facial authentication as the first system, based on the first information including information according to the attribute of the first system that performs the facial authentication.

20. A non-transitory storage medium, wherein: A program for causing a computer to execute the following processing is stored: Obtaining identification information of a user of the vehicle, i.e., user identification information; acquiring an authentication facial image, the authentication facial image being an image associated with the user identification information and being an authentication facial image of a user of the vehicle; acquiring a user facial image, the user facial image being an image generated by photographing a user of the vehicle using a camera disposed outside the vehicle; performing facial authentication based on a comparison between the authentication facial image and the user facial image; as well as In response to the failure of the facial authentication, first information is acquired from the user's facial image as information to be used in the next facial authentication or later.

Images