In-vehicle system, information processing method and program

Abstract

An in-vehicle system (10) is provided with: an image acquisition unit (12) for acquiring a captured image generated by capturing an image of a passenger positioned around a vehicle or in the interior of the vehicle; an identification unit (13) for identifying the passenger by controlling face authentication on the basis of the captured image; an assessment unit (14) for assessing whether or not the passenger is a registered user of the vehicle; and, a transmission control unit (16) for allowing a transmission means to transmit a user ID that corresponds to the passenger to an external device if the passenger is assessed to be a registered user of the vehicle.

Description

[Technical Field]

[0001] The present disclosure relates to an in-vehicle system, an information processing method, and a non-transitory computer-readable medium. [Background technology]

[0002] In recent years, there has been a growing need to enjoy city services without having to face an employee, without even rolling down the windows of the vehicle. For example, it has been proposed to use drive-through services with electronic payment using the vehicle's onboard Electronic Toll Collection (ETC) system.

[0003] Here, the use of face recognition has been proposed as a means of identity verification during payment. For example, Patent Document 1 discloses an image recognition device for a vehicle that photographs the face of a passenger using a vehicle camera and permits control of vehicle accessories on the condition that the photographed image is determined to be identical to the image of the vehicle user. It also describes that the vehicle accessories may be accessories used when making electronic money payments using an ETC on-board unit. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-205923 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the method described in Patent Document 1 above has the disadvantage that the toll is billed to the owner of the ETC card currently inserted in the ETC system, regardless of the passenger currently on board. Therefore, in order to bill the passenger, it is necessary to replace the passenger's ETC card with the ETC system every time the passenger changes, which is time-consuming.

[0006] In view of the above-mentioned problems, the object of the present disclosure is to provide an in-vehicle system, an information processing method, and a non-transitory computer-readable medium that can reduce the burden on passengers in verifying their identity when using a drive-through service. [Means for solving the problem]

[0007] An in-vehicle system according to one aspect of the present disclosure includes: an image acquisition means for acquiring a captured image generated by photographing a passenger positioned around the vehicle or inside the vehicle; an identification unit that identifies the passenger by controlling face authentication based on the captured image; a determination means for determining whether the passenger is a registered user of the vehicle; a transmission control means for permitting a transmission means to transmit a user ID corresponding to the passenger to an external device when it is determined that the passenger is a registered user of the vehicle; Equipped with.

[0008] An information processing method according to one aspect of the present disclosure includes: Acquire a captured image generated by capturing an image of a passenger positioned around the vehicle or inside the vehicle; Identifying the passenger by controlling face authentication based on the captured image; determining whether the passenger is a registered user of the vehicle; If it is determined that the passenger is the registered user of the vehicle, the transmitting means is permitted to transmit a user ID corresponding to the passenger to an external device.

[0009] According to one aspect of the present disclosure, there is provided a non-transitory computer-readable medium, comprising: an image acquisition process for acquiring a captured image generated by capturing an image of a passenger positioned around the vehicle or inside the vehicle; an identification process for identifying the passenger by controlling face authentication based on the captured image; a determination process for determining whether the passenger is a registered user of the vehicle; a transmission control process that allows a transmission means to transmit a user ID corresponding to the passenger to an external device when it is determined that the passenger is a registered user of the vehicle; A program for causing a computer to execute the above is stored. [Effects of the Invention]

[0010] The present disclosure provides an in-vehicle system, an information processing method, and a non-transitory computer-readable medium that can reduce the burden on passengers in verifying their identity when using a drive-through service. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a configuration of an in-vehicle system according to a first embodiment. [Figure 2] 1 is a flowchart showing the flow of an information processing method according to the first embodiment. [Figure 3] FIG. 10 is a block diagram showing the overall configuration of a system according to a second embodiment. [Figure 4] FIG. 10 is a block diagram showing the configuration of a face authentication device according to a second embodiment. [Figure 5] 10 is a flowchart showing the flow of face information registration processing according to the second embodiment. [Figure 6] 10 is a flowchart showing the flow of face authentication processing according to the second embodiment. [Figure 7] FIG. 10 is a block diagram showing the configuration of a store terminal according to a second embodiment. [Figure 8] FIG. 10 is a block diagram showing the configuration of an in-vehicle system according to a second embodiment. [Figure 9] FIG. 10 is a sequence diagram showing the flow of a registration process according to the second embodiment. [Figure 10] FIG. 10 is a sequence diagram showing the flow of a call origination process according to the second embodiment. [Figure 11] FIG. 10 is a diagram showing an example of a display on a display unit according to a second embodiment. [Figure 12] FIG. 10 is a diagram showing an example of a display on a display unit according to a second embodiment. [Figure 13] FIG. 10 is a diagram showing an example of a display on a display unit according to a second embodiment. [Figure 14] FIG. 10 is a block diagram showing the overall configuration of a system according to a third embodiment. [Figure 15] FIG. 10 is a block diagram showing the configuration of an in-vehicle system according to a third embodiment. [Figure 16] FIG. 11 is a sequence diagram showing the flow of a call origination process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each drawing, the same or corresponding elements are designated by the same reference numerals, and for clarity of explanation, duplicate explanations will be omitted as necessary.

[0013] <Embodiment 1> First, a first embodiment of the present disclosure will be described. Fig. 1 is a block diagram showing the configuration of an in-vehicle system 10 according to the first embodiment. The in-vehicle system 10 is a computer system mounted on a vehicle that transmits identification information (ID) of a passenger to the outside. The in-vehicle system 10 includes an image acquisition unit 12, an identification unit 13, a determination unit 14, and a transmission control unit 16.

[0014] The image acquisition unit 12 is also called image acquisition means. The image acquisition unit 12 acquires a captured image from a camera that captures the scenery around the vehicle or the interior of the vehicle. Therefore, the captured image includes an image area showing the scenery around the vehicle or the interior of the vehicle. When there is a passenger in the vehicle interior or when a passenger gets into the vehicle, the passenger positioned around the vehicle or inside the vehicle is captured by the camera, and the captured image includes the passenger's face area. Note that, hereinafter, a person before getting into the vehicle may also be referred to as a "passenger."

[0015] The identification unit 13 is also referred to as an identification means. When the captured image includes a facial region of a passenger, the identification unit 13 identifies the passenger included in the captured image by facial recognition based on the captured image. For example, the identification unit 13 identifies an ID that identifies the passenger by facial recognition.

[0016] The determination unit 14 is also called a determination means. The determination unit 14 determines whether or not the passenger is a registered user of the vehicle. The registered user of the vehicle is determined in advance. For example, the determination unit 14 determines whether or not an ID identifying the passenger is registered in advance as a registered user of the vehicle.

[0017] The outgoing call control unit 16 is also called outgoing call control means. When the outgoing call control unit 16 determines that the passenger is a registered user of the vehicle, it permits the transmitting unit (not shown) to transmit a user ID corresponding to the passenger to an external device (not shown). On the other hand, when the outgoing call control unit 16 does not determine that the passenger is a registered user of the vehicle, it does not permit the transmitting unit to transmit a user ID corresponding to the passenger to an external device. In other words, the outgoing call control unit 16 restricts or prohibits the transmitting unit from transmitting a user ID corresponding to the passenger to an external device. The user ID is information that identifies a user. The user ID may be an ID identified by facial recognition. However, instead of this, the user ID may be the user's credit card number, etc., linked to an ID identified by facial recognition.

[0018] The external device is a computer device installed outside the vehicle, which performs electronic payments based on the user ID and provides services according to the user ID.

[0019] If the outgoing call control unit 16 permits the outgoing call, the outgoing call unit transmits the passenger's user ID to the external device. On the other hand, if the outgoing call control unit 16 does not permit the outgoing call, the outgoing call unit transmits the passenger's user ID to the external device.

[0020] FIG. 2 is a flowchart showing the flow of the information processing method according to the first embodiment. First, the image acquisition unit 12 of the in-vehicle system 10 acquires a captured image from the camera (S10). If the captured image includes a facial region of a passenger, the identification unit 13 identifies the passenger by face authentication based on the captured image (S11). Next, the determination unit 14 determines whether the identified passenger is a registered user of the vehicle (S12). If the identified passenger is a registered user of the vehicle (YES in S12), the transmission control unit 16 permits the transmission unit to transmit the passenger's user ID to an external device (S13), and ends the process. On the other hand, if the identified passenger is not a registered user of the vehicle (NO in S12), the transmission control unit 16 restricts the transmission unit from transmitting the passenger's user ID to an external device (S14), and ends the process.

[0021] As described above, according to the first embodiment, the in-vehicle system 10 transmits the user ID corresponding to the passenger to the external device in a hands-free manner. In this in-vehicle system 10, even if the passenger changes, there is no need to replace the physical card. Therefore, the burden on the passenger in identity verification when using the drive-through service can be reduced.

[0022] Furthermore, the in-vehicle system 10 does not transmit the user ID if the passenger is not a registered user of the vehicle, thereby preventing unauthorized passengers from using the drive-through service.

[0023] <Embodiment 2> Next, a second embodiment of the present disclosure will be described. FIG. 3 is a block diagram showing the overall configuration of a system 1000 according to the second embodiment. The system 1000 is a computer system that enables a store terminal to remotely verify the identity of a vehicle occupant and execute processing according to the occupant. The system 1000 includes a face authentication device 100, an in-vehicle system 20, and a store terminal 400. At least the face authentication device 100 and the in-vehicle system 20 are connected to each other via a network N. The network N is a wireless communication line, but may also include a wired communication line.

[0024] The face authentication device 100 is a face authentication device that stores facial feature information of multiple people. The face authentication device 100 registers the facial feature information of a user during user registration, which will be described later. The face authentication device 100 also has a face authentication function that, in response to a face authentication request received from the in-vehicle system 20, compares the facial image or facial feature information included in the request with the facial feature information of each user. In the second embodiment, the face authentication device 100 acquires a facial image of a passenger (sometimes referred to as a user) who has visited the store from the in-vehicle device 200 of the in-vehicle system 20, and performs face authentication using the facial image. The face authentication device 100 then returns a comparison result (face authentication result) to the in-vehicle device 200.

[0025] The in-vehicle system 20 is an example of the above-mentioned in-vehicle system 10. The in-vehicle system 20 is mounted on a vehicle V. The in-vehicle system 20 includes an in-vehicle device 200, a camera 250, a display unit 260, an input unit 270, and a transmission unit 280. The camera 250, the display unit 260, the input unit 270, and the transmission unit 280 are connected to the in-vehicle device 200 by wire or wirelessly.

[0026] In the second embodiment, the camera 250 is an in-vehicle camera that captures images of the interior of the vehicle V. For example, the camera 250 may be mounted on the inner mirror of the vehicle V. However, the camera 250 may also be an in-vehicle camera that captures images of the scenery around the vehicle. The display unit 260 is a display device also called a display means. The input unit 270 is also called an input means and is an input device that accepts input from a passenger. The display unit 260 and the input unit 270 may be integrally configured like a touch panel. Furthermore, the display unit 260 and the input unit 270 may be a touch panel of a smartphone or tablet terminal connected to the in-vehicle device 200. For example, the display unit 260 and the input unit 270 may be provided in an area of ​​the interior of the vehicle V that is easily visible and allows easy input operation from the driver's seat or passenger seat, such as a dashboard. The transmission unit 280 is also called a transmission means and is a transmitter that transmits a user ID designated by the in-vehicle device 200 to the store terminal 400 via wireless communication. In the second embodiment, the wireless communication performed by the transmitting unit 280 is short-range wireless communication such as Bluetooth (registered trademark) or BLE (Bluetooth Low Energy). However, the transmitting unit 280 may employ any other wireless communication.

[0027] The in-vehicle device 200 is a computer device that controls transmission of the transmitting unit 280. The in-vehicle device 200 controls the camera 250 to capture a scene including the passenger U riding in the vehicle V. The in-vehicle device 200 performs face authentication on the facial area of ​​the passenger U via the face authentication device 100. When the passenger U whose face has been successfully authenticated satisfies a predetermined condition, the in-vehicle device 200 controls the transmitting unit 280 to transmit the user ID of the passenger U to the store terminal 400.

[0028] The store terminal 400 is an example of the external device described above. The store terminal 400 is an information terminal installed in a store that provides a drive-through service. For example, the store is a gas station, an EV (Electric Vehicle) charging station, or a fast food restaurant. The store terminal 400 receives the user ID transmitted from the transmitting unit 280 and verifies the identity of the occupant of the vehicle V. The store terminal 400 then performs electronic payment, with the occupant of the vehicle V being billed. Alternatively, the store terminal 400 provides a service according to the individual occupant of the vehicle V or the attributes of the occupant. For example, providing a service according to the individual occupant or the attributes of the occupant may be providing promotional information or preferential treatment information according to the individual occupant or the attributes to the in-vehicle system 20. For another example, providing a service according to the individual occupant or the attributes of the occupant may be notifying a store staff member that the occupant of the vehicle V has a predetermined attribute (for example, a VIP member of the store).

[0029] FIG. 4 is a block diagram showing the configuration of a face authentication device 100 according to the second embodiment. The face authentication device 100 includes a face information database (DB) 110, a face detection unit 120, a feature point extraction unit 130, a registration unit 140, and an authentication unit 150. The face information DB 110 stores a user ID 111 and facial feature information 112 corresponding to the user ID in association with each other. The facial feature information 112 is a collection of feature points extracted from a face image and is an example of face information. The face authentication device 100 may delete the facial feature information 112 from the facial feature DB 110 in response to a request from a registered user of the facial feature information 112. Alternatively, the face authentication device 100 may delete the facial feature information 112 after a certain period of time has elapsed since the registration of the facial feature information 112.

[0030] The face detection unit 120 detects a face area included in a registration image for registering face information and supplies the detected face area to the feature point extraction unit 130. The feature point extraction unit 130 extracts feature points from the face area detected by the face detection unit 120 and supplies face feature information to the registration unit 140. The feature point extraction unit 130 also extracts feature points included in a face image received from the in-vehicle device 200 and supplies the face feature information to the authentication unit 150.

[0031] The registration unit 140 issues a new user ID 111 when registering facial feature information. The registration unit 140 associates the issued user ID 111 with facial feature information 112 extracted from the registered image and registers them in the facial information DB 110. The authentication unit 150 performs facial authentication using the facial feature information 112. Specifically, the authentication unit 150 compares the facial feature information extracted from the facial image with the facial feature information 112 in the facial information DB 110. The authentication unit 150 returns a response indicating whether or not the facial feature information matches to the in-vehicle device 200. The presence or absence of a match in the facial feature information corresponds to the success or failure of authentication. Note that the facial feature information matches (matches) when the degree of match is equal to or greater than a predetermined value.

[0032] FIG. 5 is a flowchart showing the flow of face information registration processing according to the second embodiment. First, the face authentication device 100 acquires a registration image of the user included in a face registration request (S21). For example, the face authentication device 100 receives the face registration request from the in-vehicle device 200 or a user terminal used by the user via the network N. Next, the face detection unit 120 detects a face area included in the registration image (S22). Next, the feature point extraction unit 130 extracts feature points from the face area detected in step S22 and provides facial feature information to the registration unit 140 (S23). Finally, the registration unit 140 issues a user ID 111 and associates the user ID 111 with facial feature information 112 to register the user ID 111 in the face information DB 110 (S24). Note that the face authentication device 100 may receive the facial feature information 112 from the source of the face registration request and register the facial feature information 112 in the face information DB 110 in association with the user ID 111.

[0033] FIG. 6 is a flowchart showing the flow of face authentication processing according to the second embodiment. First, the feature point extraction unit 130 acquires facial feature information for authentication (S31). For example, the face authentication device 100 receives a face authentication request from the in-vehicle device 200 via the network N and extracts facial feature information from the face image included in the face authentication request as in steps S21 to S23. Alternatively, the face authentication device 100 may receive the facial feature information from the in-vehicle device 200. Next, the authentication unit 150 compares the acquired facial feature information with the facial feature information 112 in the face information DB 110 (S32). If the facial feature information matches, that is, if the degree of match between the facial feature information is equal to or greater than a predetermined value (Yes in S33), the authentication unit 150 identifies the user ID 111 of the user whose facial feature information matches (S34). Then, the authentication unit 150 returns a message indicating that face authentication was successful and the identified user ID 111 as a face authentication result to the in-vehicle device 200 (S35). If there is no matching facial feature information (No in S33), the authentication unit 150 returns a message to the in-vehicle device 200 indicating that the facial authentication has failed as a facial authentication result (S36).

[0034] 7 is a block diagram showing the configuration of the store terminal 400 according to embodiment 2. The store terminal 400 includes a storage unit 420, a communication unit 430, a display unit 440, an input unit 450, and a control unit 460.

[0035] The memory unit 420 is a storage device that stores programs for realizing each function of the store terminal 400. The communication unit 430 is a communication interface for communicating with the in-vehicle system 20. The display unit 440 is a display device. The input unit 450 is an input device that accepts input. The display unit 440 and the input unit 450 may be configured integrally, such as a touch panel. The control unit 460 controls the hardware of the store terminal 400. Note that the display unit 440 and the input unit 450 may be omitted.

[0036] 8 is a block diagram showing the configuration of the in-vehicle system 20 according to the second embodiment. As described above, the in-vehicle system 20 includes the in-vehicle device 200. The in-vehicle device 200 is connected to a camera 250, a display unit 260, an input unit 270, and a transmission unit 280.

[0037] The in-vehicle device 200 includes a storage unit 210, a memory 220, a communication unit 230, and a control unit 240. The storage unit 210 is a storage device such as a hard disk or a flash memory. The storage unit 210 stores a program 211 and a registered user list 212. The program 211 is a computer program that implements the processing of the information processing method according to the second embodiment.

[0038] The registered user list 212 stores the user IDs 2121 of registered users. The user IDs 2121 stored in the registered user list 212 are user IDs of users who are authorized as legitimate passengers of the vehicle V, among the user IDs 111 issued by the face authentication device 100 and stored in the face information DB 110.

[0039] The memory 220 is a volatile storage device such as a RAM (Random Access Memory), and is a storage area for temporarily storing information during operation of the control unit 240. The communication unit 230 is a communication interface with the network N.

[0040] The control unit 240 is a processor, i.e., a control device, that controls each component of the in-vehicle device 200. The control unit 240 loads the program 211 from the storage unit 210 into the memory 220 and executes the program 211. In this way, the control unit 240 realizes the functions of a registration unit 241, an image acquisition unit 242, an identification unit 243, a determination unit 244, an output control unit 245, and a transmission control unit 246.

[0041] The registration unit 241 is also called a registration means. When the registration unit 241 acquires a registration image, which is a captured image of a user to be registered, from the camera 250 or an administrator terminal used by an administrator of the in-vehicle system 20, the registration unit 241 transmits a face registration request to the face authentication device 100. Then, when the face authentication device 100 registers face information and issues a user ID, the registration unit 241 registers the user ID as a registered user in the registered user list 212. Furthermore, when there is a request from the administrator to add a registered user, the registration unit 241 may add the user ID of the designated registered user to the registered user list 212. On the other hand, when there is a request from the administrator to delete a registered user, the registration unit 241 may delete the user ID of the designated registered user from the registered user list 212.

[0042] The image acquisition unit 242 is an example of the above-mentioned image acquisition unit 12. The image acquisition unit 242 acquires a captured image of a passenger positioned inside or around the vehicle V from the camera 250, and supplies the captured image to the identification unit 243.

[0043] The identification unit 243 is an example of the above-mentioned identification unit 13. The identification unit 243 controls face authentication based on the captured image to identify the user ID of the passenger detected from the captured image. Specifically, when the image acquisition unit 242 acquires a captured image, the identification unit 243 transmits a face authentication request including the captured image or facial feature information extracted from the captured image to the face authentication device 100 via the network N. Then, the identification unit 243 acquires a face authentication result from the face authentication device 100. When face authentication is successful, the identification unit 243 extracts the user ID from the face authentication result.

[0044] Here, when multiple passengers are detected from the captured image, the identification unit 243 identifies the user IDs of each of the multiple passengers. That is, the identification unit 243 transmits a face recognition request for the face region of each of the multiple passengers to the face recognition device 100, and acquires the face recognition results of each of the multiple passengers from the face recognition device 100.

[0045] The determination unit 244 is an example of the above-mentioned determination unit 14. The determination unit 244 refers to the registered user list 212 and determines whether or not the user ID identified by the identification unit 243 is included in the registered user list 212. When the identification unit 243 identifies the user ID of each of the multiple passengers, the determination unit 244 determines, for each of the multiple passengers, whether or not the passenger is a registered user of the vehicle.

[0046] The output control unit 245 is also called output control means. When the passenger is a registered user included in the registered user list 212, the output control unit 245 displays information for prompting the passenger to perform a call permission operation on the display unit 260, and receives a call permission operation from the passenger via the input unit 270.

[0047] Here, when there are multiple passengers who have been determined to be registered users of the vehicle, the output control unit 245 may cause the display unit 260 to display a list of the passengers who have been determined to be registered users of the vehicle V. Then, the output control unit 245 may accept, via the input unit 270, an operation from the passenger to select a passenger who is to transmit a user ID from among the passengers who are registered users.

[0048] The outgoing call control unit 246 is an example of the outgoing call control unit 16 described above. When the output control unit 245 receives an outgoing call permission operation from a passenger, the outgoing call control unit 246 controls the transmitting unit 280 to transmit the passenger's user ID to the store terminal 400 via short-range wireless communication. That is, in the second embodiment, even when the determination unit 244 determines that the passenger is a registered user of the vehicle V, the outgoing call control unit 246 restricts the transmitting unit 280 from transmitting the user ID to the store terminal 400 until the determination unit 244 receives an outgoing call permission operation from the passenger. This makes it possible to prevent the user ID from being transmitted to the outside without the passenger's consent, and electronic payments and the like from being made without permission.

[0049] In addition, when there are multiple passengers determined to be registered users of vehicle V, the transmission control unit 246 may restrict the transmission of user IDs by the transmission unit 280 until one of the passengers of vehicle V selects at least one passenger from the list. The transmission control unit 246 may then permit the transmission of the user IDs of all passengers determined to be registered users of vehicle V to the store terminal 400, or may permit only the user IDs of selected passengers to be transmitted to the store terminal 400. That is, in the latter case, the transmission control unit 246 may restrict the transmission of user IDs of passengers who were not selected to the store terminal 400, even if they are registered users of vehicle V. This makes it possible to limit the destinations for billing and service provision to only some of the passengers determined to be registered users, thereby improving convenience.

[0050] FIG. 9 is a sequence diagram showing the flow of registration processing according to the second embodiment. First, when a registration request is received from an administrator, the registration unit 241 of the in-vehicle device 200 acquires a registration image of a user to be registered (S501). For example, the registration unit 241 of the in-vehicle device 200 may acquire the registration image from the camera 250 or an administrator terminal. Next, the registration unit 241 of the in-vehicle device 200 transmits a face registration request including the registration image to the face authentication device 100 (S502). Then, the face authentication device 100 registers face information (facial feature information) of the user to be registered based on the registration image included in the received face registration request (S503). Then, the face authentication device 100 notifies the in-vehicle device 200 of the issued user ID (S504). The registration unit 241 of the in-vehicle device 200 registers the notified user ID in the registered user list 212 as the user ID of the registered user (S506).

[0051] 10 is a sequence diagram showing the flow of the calling process according to the second embodiment. This diagram shows, as an example, the calling process when making an electronic payment at a store.

[0052] First, the image acquisition unit 242 controls the camera 250 to capture an image of the passenger U (S521) and supplies the captured image to the identification unit 243. At this time, the image acquisition unit 242 may stop supplying the captured image to the identification unit 243 until a predetermined face recognition trigger is detected. Here, the face recognition trigger may be the detection of a person's face area in the captured image, particularly the detection of a face area large enough to perform face recognition. The identification unit 243 of the in-vehicle device 200 transmits a face recognition request for the face area of ​​the passenger U in the captured image to the face recognition device 100 (S522). Then, the face recognition device 100 performs face recognition on the face area of ​​the passenger U in the face image included in the received face recognition request (S523). Here, it is assumed that a user ID for which face recognition was successful exists. The face recognition device 100 transmits a face recognition result including a notification that face recognition was successful and the user ID to the in-vehicle device 200 (S524). At this time, the in-vehicle device 200 may display on the display unit 260 whether or not the face authentication has been successful.

[0053] The determination unit 244 of the in-vehicle device 200, which has received the face authentication result, refers to the registered user list 212 and determines whether the identified user ID is included in the registered user list 212 (S525). The processes of S521 to S525 may be performed when the passenger U gets into the vehicle V, while driving, or immediately before making an electronic payment at a store. If the identified user ID is included in the registered user list 212, the determination unit 244 determines whether a predetermined transmission trigger has been detected, and if a predetermined transmission trigger has been detected (S526), ​​the process proceeds to S527. Here, the transmission trigger is a trigger for transmitting the user ID of the passenger U. For example, the transmission trigger may be receipt of an input from the passenger U requesting electronic payment via the input unit 270, or receipt of a signal inquiring about the availability of electronic payment from the store terminal 400. Next, the output control unit 245 of the in-vehicle device 200 causes the display unit 260 to display a list of passengers U who have been determined to be registered users (S527).

[0054] Here, the processing of S528 to S530 when there is one passenger U determined to be a registered user will be described. In this case, when the passenger U performs a call permission operation (S528), the call control unit 246 of the in-vehicle device 200 sends the user ID of the passenger determined to be a registered user to the store terminal 400 via the calling unit 280 (S529). The store terminal 400, which has received the user ID, executes a payment process with the person identified by the user ID as the payment recipient (S530).

[0055] FIG. 11 is a diagram showing an example of the display of the display unit 260 according to the second embodiment. This diagram shows the display screen in S528 when one passenger U is determined to be a registered user. For example, the display unit 260 may display the name of the passenger determined to be a registered user and a message confirming whether it is okay to make an electronic payment with that passenger as the payee. When the passenger U performs an input operation to authorize the payment, the input operation is detected as a call permission operation. Then, the user ID of the passenger determined to be a registered user is transmitted to the store terminal 400.

[0056] Next, the processing of S528 to S530 when there are multiple passengers U determined to be registered users will be described. In this case, when any one of the passengers U performs a passenger selection operation and then a call permission operation (S528), the call control unit 246 of the in-vehicle device 200 transmits the user ID of the selected passenger to the store terminal 400 via the transmission unit 280 (S529). The store terminal 400, which has received the user ID, executes a payment process with the person identified by the user ID as the payment recipient (S530).

[0057] FIG. 12 is a diagram showing an example of the display of the display unit 260 according to the second embodiment. This diagram shows the display screen in S528 when there are multiple passengers determined to be registered users. For example, the display unit 260 displays a list of the names of passengers determined to be registered users and a message for selecting which passenger to make the payment to. When a passenger performs an operation to select a passenger and selects "OK," it is recognized that a transmission permission operation has been performed to permit transmission of the user ID of the selected passenger. Then, the user ID of the selected passenger is transmitted to the store terminal 400.

[0058] Here, when a passenger of vehicle V specifies a payment method, the transmission control unit 246 may control the transmission unit 280 to transmit information about the payment method together with the user ID to the store terminal 400. The information about the payment method may include information such as whether to pay in full or in installments, which account to debit from, whether to use points, whether one person or multiple people will pay, and in what proportions if multiple people will pay.

[0059] The payment method may be specified in S528 of FIG. FIG. 13 is a diagram illustrating an example of a display on the display unit 260 according to the second embodiment. For example, the display unit 260 displays a list of names of passengers determined to be registered users and an input area for inputting each passenger's payment percentage. The payment percentage input area accepts an addition or subtraction operation from the passenger to change the payment percentage. Each payment percentage may be adjusted so that the total payment percentage is 1. When the passenger selects the "OK" input area, information about the payment method is transmitted to the store terminal 400 along with the user ID. By allowing the passenger to specify the payment method each time, convenience is improved. This screen may be displayed every time in S528 of FIG. 10, or may be displayed when multiple passengers are selected in FIG. 12 and not when a single passenger is selected. This screen may also be displayed only when an input operation to change the payment percentage is performed in S528 of FIG. 10. In this case, if no input operation to change the payment percentage is performed, the payment percentage may be automatically set to an equal percentage for each passenger (a so-called split payment).

[0060] Note that the second embodiment has the same effects as the first embodiment. That is, in the in-vehicle system 20, there is no need to replace the physical card even if the passenger changes. Therefore, the burden on the passenger U in identity verification when using the drive-through service can be reduced. Furthermore, the in-vehicle system 20 does not transmit the user ID if the passenger U is not a registered user of the vehicle V. Therefore, it is possible to prevent a passenger U who boards the vehicle fraudulently from using the drive-through service.

[0061] Furthermore, stores do not need to install a dedicated ETC system receiver, as long as they have a general-purpose receiver for short-range wireless communication such as Bluetooth, which reduces costs.

[0062] <Embodiment 3> Next, a third embodiment of the present disclosure will be described. The third embodiment is characterized in that an in-vehicle system unlocks the opening and closing members of a vehicle by facial recognition, and the registered user determination result for the call control process is used in the unlocking control process. In the third embodiment, the opening and closing members are the front and rear doors, but are not limited thereto and may be a hood, a fuel filler cover, door glass (windows), a trunk cover, etc. Hereinafter, the opening and closing members will be simply referred to as "doors."

[0063] 14 is a block diagram showing the overall configuration of a system 1000a according to the third embodiment. The system 1000a differs from the system 1000 of the second embodiment in that the system 1000a includes an in-vehicle system 20a instead of the in-vehicle system 20.

[0064] The in-vehicle system 20a differs from the in-vehicle system 20 in that the in-vehicle system 20a has a camera 250a instead of the camera 250 and an in-vehicle device 200a instead of the in-vehicle device 200.

[0065] Camera 250a is an in-vehicle camera that captures the surrounding scenery outside vehicle V. Camera 250a captures at least the face of a passenger U who approaches to unlock the door of vehicle V, which is locked, and get in.

[0066] The in-vehicle device 200a executes door unlocking control using face authentication in addition to the functions of the in-vehicle device 200. The in-vehicle device 200a uses the registered user determination result for the outgoing call control process for the door unlocking control process.

[0067] 15 is a block diagram showing the configuration of an in-vehicle system 20a according to the third embodiment. An in-vehicle device 200a of the in-vehicle system 20a includes a storage unit 210a instead of the storage unit 210 and a control unit 240a instead of the control unit 240. The storage unit 210a stores a program 211a instead of the program 211. The program 211a is a computer program that implements the processing of the information processing method according to the third embodiment.

[0068] The control unit 240a has a release control unit 247. When the determination unit 244 determines, based on the captured image acquired from the camera 250a, that the passenger U is a registered user of the vehicle V, the release control unit 247 transmits a control signal to a drive unit (not shown) to unlock the doors of the vehicle V. This causes the doors to be unlocked. Note that the doors to be unlocked may be all doors or only some of the doors.

[0069] FIG. 16 is a sequence diagram showing the flow of the transmission process according to the third embodiment. The steps in FIG. 16 include S540 and S541 instead of S521 in FIG. 10. First, the image acquisition unit 242 controls the camera 250a to capture an image of the passenger U positioned in front of the vehicle V (S540) and supplies the captured image to the identification unit 243. Then, the same processes as those in S522 to S525 are executed. Here, it is assumed that the registered user determination by the determination unit 244 is successful. In this case, in S541, the release control unit 247 transmits a control signal for unlocking the doors of the vehicle V to a drive unit (not shown), thereby unlocking the doors. After the passenger U gets into the vehicle V, the same processes as those in S526 to S530 are executed. That is, when the determination unit 244 detects a transmission trigger (S526), ​​the output control unit 245 and the transmission control unit 246 execute the same processes as those in the second embodiment based on the determination result by the determination unit 244.

[0070] In this way, in the third embodiment, the face authentication and registered user determination for unlocking the door and the face authentication and registered user determination for the outgoing call control process described in the second embodiment may be performed at the same time. Therefore, face photography may be performed by the outside-vehicle camera (camera 250a), and face photography by the inside-vehicle camera as shown in FIG. 3 may be omitted.

[0071] Thus, according to embodiment 3, the in-vehicle device 200a reuses the registered user determination results for the outgoing call control process for the unlocking control process, thereby enhancing functionality while reducing the processing load and the increased psychological burden on the passenger U due to photographing the passenger U again.

[0072] Although the above-described embodiments have been described as hardware configurations, the present disclosure is not limited to such configurations. Any processing in the present disclosure can also be realized by causing a processor to execute a computer program.

[0073] In the above examples, the program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable medium or tangible storage medium includes random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technology, CD-ROM, digital versatile disc (DVD), Blu-ray® disc or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device. The program may also be transmitted on a transitory computer-readable medium or communication medium. By way of example and not limitation, transitory computer-readable medium or communication medium includes electrical, optical, acoustic, or other forms of propagated signals.

[0074] It should be noted that the present disclosure is not limited to the above-described embodiment, and can be modified as appropriate within the scope of the present disclosure. For example, in the above-described embodiment, the face authentication function is provided in the face authentication device 100, but instead of or in addition to the face authentication device 100, the registration unit 241 and the identification unit 243 of the in-vehicle device 200 may have the face authentication function.

[0075] In the above-described embodiment, the camera 250, 250a for capturing a face image is installed in the vehicle V, and the in-vehicle device 200, 200a performs face authentication based on the image captured by the camera 250, 250a to identify the user. However, instead of this, the passenger U may capture a face image using a user terminal with a camera used by the passenger U, and the in-vehicle device 200, 200a may identify the user based on the captured image thus generated. The user terminal with a camera is, for example, a smartphone or a tablet terminal.

[0076] For example, the following method can be used to identify a user based on an image captured by a user terminal. (Method 1) First, the user terminal transmits captured image data to the face authentication device 100 and receives a face authentication result from the face authentication device 100. Alternatively, the user terminal performs face authentication using a face authentication function within the terminal. Then, the user terminal transmits the face authentication result and the user ID to the in-vehicle system 20, 20a via short-range wireless communication such as Bluetooth. At this time, the user terminal may transmit the captured image to the in-vehicle system 20, 20a in addition to the face authentication result and the user ID for confirmation. (Method 2) The user terminal sends the captured image to the in-vehicle system 20a. The in-vehicle device 200, 200a of the in-vehicle system 20, 20a that receives the captured image controls face authentication to identify the user. By adopting the above method, the cameras 250 and 250a of the in-vehicle systems 20 and 20a can be omitted.

[0077] In the above-described embodiment, the transmission control unit 246 controlled the transmission unit 280 to transmit the user ID corresponding to the passenger U. However, if the passenger U is using a vehicle V that has been rented from a car sharing management company, that is, if the vehicle V is a shared car, the transmission control unit 246 may also control the transmission unit 280 to transmit the ID of the car sharing management company.

[0078] Here, when the vehicle V is a shared car, the transmission control unit 246 may determine whether to transmit the user ID corresponding to the passenger U or the ID of the shared car management company depending on the type of store visited by the vehicle V. This allows, for example, at a service station such as a gas station, the payment can be made to the shared car management company instead of the passenger U, while at a drive-through of a fast food restaurant, the payment can be made to the passenger U.

[0079] Specifically, the following process may be performed. (1) The in-vehicle device 200, 200a acquires a store ID from the store terminal 400 and identifies the service type or service provider of the store. This allows the in-vehicle device 200 to identify, for example, whether the store is a service station or a fast food restaurant. The in-vehicle device 200, 200a then determines whether the service type or service provider of the store is a predetermined service type or service provider. (2) Next, when the service type or service provider of the store is a predetermined service type or service provider, the in-vehicle device 200, 200a determines whether or not the vehicle V is a shared car. For example, when the store is a service station, the in-vehicle device 200, 200a determines whether or not the vehicle V is a shared car. As a method for determining whether or not the vehicle V is a shared car, the in-vehicle device 200, 200a may use a vehicle number stored in advance, or, if other information indicating that the vehicle is installed in a shared car is stored in advance, that information may be used. (3) Next, if the vehicle V is a shared car, the transmission control unit 246 of the in-vehicle device 200, 200a controls the transmission unit 280 to transmit the shared car ID or the shared car management company ID instead of the personal ID. Then, in the payment process, the shared car ID or the shared car management company ID is used.

[0080] Note that (2) may be performed before (1). That is, the in-vehicle device 200, 200a may determine whether the vehicle V is a shared car, and if it is a shared car, may acquire a store ID from the store terminal 400 and identify the service type or service provider of the store.

[0081] In this way, the transmission control unit 246 of the in-vehicle device 200, 200a may control the transmission unit 280 to transmit the ID of the shared car management company when the vehicle V is a shared car and the store's service type or service provider is a predetermined one. On the other hand, the transmission control unit 246 may control the transmission unit 280 to transmit the user ID corresponding to the passenger U when the vehicle V is not a shared car or when the vehicle V is a shared car but the store's service type or service provider is not a predetermined one. [Explanation of symbols]

[0082] 10,20,20a In-vehicle systems 100 Facial Recognition Device 110 Face Information DB 111 User ID 112 Facial Feature Information 120 Face detection unit 130 Feature point extraction unit 140 Registration Department 150 Authentication Department 200,200a Onboard equipment 210,210a Storage section 211,211a Program 212 Registered User List 2121 User ID 220 memory 230 Communications Department 240,240a Control unit 241 Registration Department 12,242 Image acquisition unit 13,243 Specific section 14,244 Judgment section 245 Output control section 16,246 Outgoing control section 247 Release control section 250,250a camera 260 Display 270 Input section 280 Transmission Department 400 store terminals 420 Storage section 430 Communications Department 440 Display section 450 Input section 460 Control Unit 1000,1000a system N Network

Claims

1. an image acquisition means for acquiring a captured image generated by photographing a passenger positioned around the vehicle or inside the vehicle; an identification unit that identifies the passenger by controlling face authentication based on the captured image; a determination means for determining whether the passenger is a registered user of the vehicle; a transmission control means for controlling the transmission means to transmit the user ID to an external device in accordance with the determination result of the determination means; an output control means for causing a display means to display a list of passengers determined to be the registered users of the vehicle when there are multiple passengers determined to be the registered users of the vehicle; Equipped with the identification means, when a plurality of passengers are detected from the captured image, identifies each of the plurality of passengers; The determination means determines, for each of the plurality of passengers, whether the passenger is a registered user of the vehicle; the transmission control means restricts the transmission of the user ID by the transmission means to the external device until at least one passenger is selected from the list; the transmission control means permits the transmission means to transmit the user ID to the external device when at least one passenger is selected from the list. In-vehicle systems.

2. The vehicle further includes an unlocking control unit that transmits a control signal for controlling unlocking of at least one of the opening and closing members of the vehicle when it is determined that the passenger is the registered user of the vehicle based on the captured image. The in-vehicle system according to claim 1 .

3. When a payment method is designated, the transmission control means controls the transmission means to transmit information about the payment method together with the user ID to the external device.

3. The in-vehicle system according to claim 1 or 2.

4. Acquire a captured image generated by capturing an image of a passenger positioned around a vehicle or inside the vehicle; Identifying the passenger by controlling face authentication based on the captured image; determining whether the passenger is a registered user of the vehicle; controlling a transmitting means to transmit a user ID to an external device according to a determination result of whether or not the passenger is the registered user of the vehicle; If there are a plurality of passengers who are determined to be the registered users of the vehicle, displaying a list of the passengers who are determined to be the registered users of the vehicle on a display means; If a plurality of passengers are detected from the captured image, each of the plurality of passengers is identified; determining, for each of the plurality of occupants, whether the occupant is a registered user of the vehicle; restricting the transmission of the user ID to the external device by the transmission means until at least one passenger is selected from the list; When at least one passenger is selected from the list, the transmitting means is permitted to transmit the user ID to the external device. Information processing methods.

5. If it is determined based on the captured image that the passenger is a registered user of the vehicle, a control signal is transmitted to control the unlocking of at least one of the opening and closing mechanisms of the vehicle. The information processing method according to claim 4.

6. When a payment method is designated, the transmitting means is controlled to transmit information about the payment method together with the user ID to the external device.

6. The information processing method according to claim 4 or 5.

7. an image acquisition process for acquiring a captured image generated by capturing an image of a passenger positioned around the vehicle or inside the vehicle; an identification process for identifying the passenger by controlling face authentication based on the captured image; a determination process for determining whether the passenger is a registered user of the vehicle; a transmission control process for controlling a transmission means to transmit a user ID to an external device according to a determination result of whether or not the passenger is a registered user of the vehicle; a display process for displaying, on a display means, a list of passengers determined to be the registered users of the vehicle when there are a plurality of passengers determined to be the registered users of the vehicle; on the computer, If a plurality of passengers are detected from the captured image, each of the plurality of passengers is identified; determining, for each of the plurality of occupants, whether the occupant is a registered user of the vehicle; restricting the transmission of the user ID to the external device by the transmission means until at least one passenger is selected from the list; a program for causing a computer to execute a process for permitting the transmitting means to transmit the user ID to the external device when at least one passenger is selected from the list;

8. When it is determined based on the captured image that the passenger is the registered user of the vehicle, the computer is caused to execute a process of transmitting a control signal for controlling the unlocking of at least one of the opening and closing members of the vehicle. The program according to claim 7.

9. When a payment method is specified, the computer executes a process of controlling the transmitting means to transmit information about the payment method together with the user ID to the external device. The program according to claim 7 or 8.

Images