Vehicle door control device, method, and program

Abstract

This invention is intended to make it possible to improve user convenience in vehicle door control. A door determination means (11) performs a first door determination process, and determines a door to be controlled on the basis of an image of a user attempting to board a vehicle having a plurality of doors. A door control means (12) performs an operation of the door determined by the door determination means (11). An inquiry means (13) inquires of the user whether or not the door determined by the door determination means (11) can be operated. When the result of the inquiry indicates negative, the door control means (12) performs an operation canceling the operation performed on the door determined by the door determination means (11), and performs an operation of a door different from the determined door.

Description

Vehicle door control device, method, and computer-readable medium

[0001] The present disclosure relates to a vehicle door control device, a method, and a computer-readable medium.

[0002] As a related technique, Patent Document 1 discloses a smart entry system. In the smart entry system described in Patent Document 1, a user approaches a vehicle while carrying an electronic key. An electronic key authentication device installed inside the vehicle wirelessly communicates with the electronic key carried by the user and authenticates the electronic key. A camera installed inside the vehicle captures an image of the user carrying the electronic key. The user directs their gaze toward an area corresponding to the door they wish to select from multiple areas pre-defined for each door in the vehicle. A computing device detects the user's gaze from the image captured by the camera and determines the door selected by the user based on the detected gaze.

[0003] The computing device uses the audio output device to signal to the user that the door is ready for operation. The audio output device 20 outputs a confirmation message such as "Are you sure you want to open the driver's door?" to ask the user whether to unlock and open the determined door. If the user does not look away from the area where they were looking, the door unlocking device unlocks the door selected by the user. If the user looks away from the area where they were looking, the computing device determines that the user has denied the signal that the door is ready for operation. In this case, the user manually unlocks the door by touching the handle of the door they want to open.

[0004] Japanese Patent Application Laid-Open No. 2008-2095

[0005] In Patent Document 1, the computing device sends a signal to the user indicating that the door is ready to be operated, and the user can hear a confirmation message to know which door will be operated. If the door determined by the computing device is different from the door the user wants to select, the user can avoid operating the wrong door by looking away. However, in Patent Document 1, if the user denies the signal indicating that the door is ready to be operated, the doors are not automatically unlocked, and the user must touch the door they want to open and manually unlock it. Therefore, it is desirable to further improve the convenience of devices that control vehicle doors.

[0006] In view of the above circumstances, an object of the present disclosure is to provide a vehicle door control device, a method, and a computer-readable medium that can improve user convenience in vehicle door control.

[0007] To achieve the above object, the present disclosure provides, as a first aspect, a vehicle door control device. The vehicle door control device includes: a door determination unit that performs a first door determination process to determine a door to be controlled based on an image of a user attempting to enter a vehicle having multiple doors; a door control unit that operates the determined door; and an inquiry unit that inquires of the user whether or not to operate the door. If the inquiry result indicates a negative, the door control unit performs an operation to cancel the performed operation on the determined door and operates a door other than the determined door.

[0008] In a second aspect, the present disclosure provides a vehicle door control method, which includes: performing a first door determination process to determine a door to be controlled based on an image of a user attempting to enter a vehicle having multiple doors; operating the determined door; inquiring of the user as to whether or not to operate the door; and, if the result of the inquiry indicates no, performing an operation to cancel the operation performed on the determined door; and operating a door other than the determined door.

[0009] In a third aspect, the present disclosure provides a computer-readable medium storing a program for causing a processor to execute a process including: performing a first door determination process for determining a door to be controlled based on an image of a user attempting to enter a vehicle having multiple doors; operating the determined door; inquiring of the user about whether or not to operate the door; and, if the inquiry result indicates no, performing an operation to cancel the operation performed on the determined door; and operating a door other than the determined door.

[0010] The vehicle door control device, method, and computer-readable medium according to the present disclosure can improve user convenience in vehicle door control.

[0011] The present disclosure relates to a vehicle door control device, a vehicle door control system, a vehicle door control device according to an embodiment of the present disclosure ...

[0012]

[0023] Before describing the embodiments of the present disclosure, an overview of the present disclosure will be described. Fig. 1 shows a schematic configuration example of a vehicle door control device according to the present disclosure. The vehicle door control device 10 includes a door determination unit 11, a door control unit 12, and an inquiry unit 13.

[0013] The door determination means 11 performs a first door determination process based on an image of a user attempting to get into the vehicle, and determines which of the vehicle's multiple doors is to be controlled. The door control means 12 operates the door determined by the door determination means 11. The inquiry means 13 inquires of the user whether or not the door operation to be performed by the door control means 12 is acceptable. If the inquiry result indicates a negative result, the door control means 12 performs an operation to cancel the operation performed on the door determined by the first door determination process. Furthermore, the door control means 12 operates a door other than the door determined by the first door determination process.

[0014] In the present disclosure, if the result of the inquiry acquired by the inquiry means 13 indicates a negative result, the door control means 12 performs an operation to cancel the operation performed on the door determined in the first door determination process and performs an operation on a door other than the door determined in the first door determination process. In the present disclosure, if the door determined as the door to be controlled in the first door determination process is different from the door the user wants to operate, the door control means 12 performs an operation on the other door. In this manner, even if the door determined as the door to be controlled in the first door determination process is different from the door the user wants to operate, the user does not need to touch the vehicle. Therefore, the vehicle door control device 10 can improve user convenience in vehicle door control.

[0015] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. In addition, in each drawing, the same or similar elements are designated by the same reference numerals, and duplicate explanations are omitted as necessary.

[0016] FIG. 2 shows a vehicle door control device according to an embodiment of the present disclosure. The vehicle door control device 100 includes an occupant detection unit 101, a face authentication unit 102, a door determination unit 103, a door control unit 104, and an inquiry unit 105. The vehicle door control device 100 may be physically configured as a device including, for example, one or more memories and one or more processors. At least a portion of the functions of each unit in the vehicle door control device 100 may be realized by the one or more processors operating in accordance with instructions read from the one or more memories. The vehicle door control device 100 is mounted on, for example, a vehicle. The vehicle door control device 100 corresponds to the vehicle door control device 10 shown in FIG. 1.

[0017] The occupant detection unit 101 acquires a camera image from the camera 130. The camera 130 is mounted on a vehicle and captures an image of the area around the vehicle. The camera 130 captures an image of a person who is about to get into the vehicle, i.e., an occupant, around the vehicle. The occupant of the vehicle is also called a user. The camera 130 may be embedded in, for example, a B-pillar of the vehicle, i.e., a pillar between the front door and the rear door.

[0018] 2 illustrates only one camera 130, the number of cameras 130 is not limited to one. The occupant detection unit 101 may acquire camera images from each of the multiple cameras 130. For example, the occupant detection unit 101 may acquire camera images from a camera that captures an area on the left side of the vehicle, a camera that captures an area on the right side of the vehicle, and a camera that captures an area behind the vehicle.

[0019] The occupant detection unit 101 detects a user who is about to get into the vehicle using a camera image. The occupant detection unit 101 extracts, for example, a face image or a face area of ​​a person near the vehicle from the camera image. When a face image or a face area is extracted, the occupant detection unit 101 determines that a user has been detected.

[0020] The face authentication unit 102 authenticates the user detected by the occupant detection unit 101. In this embodiment, the face authentication unit 102 authenticates the user through face authentication using a camera image from the camera 130. For example, the face authentication unit 102 registers face images of people permitted to board the vehicle or feature amounts extracted from the face images. The face authentication unit 102 compares the face image included in the camera image or the feature amounts extracted from the face image with the registered face image or feature amounts. The face authentication unit 102 calculates a matching score indicating the similarity between the face image included in the camera image or the feature amounts extracted from the face image and the registered face image or feature amounts. If the matching score is equal to or greater than a predetermined threshold, the face authentication unit 102 determines that the user's face authentication has been successful.

[0021] The door determination unit 103 performs a first door determination process to determine a door to be controlled based on a camera image. In the first door determination process, the door determination unit 103 detects, for example, at least one of a user's position, a user's line of sight, and a user's gesture from the camera image. The door determination unit 103 determines a door to be controlled based on the detected at least one of the user's position, the user's line of sight, and the user's gesture. The door determination unit 103 may determine a door to be controlled based on a combination of the detected user's position, the user's line of sight, and the user's gesture.

[0022] When determining the door to be controlled based on the user's position, the door determination unit 103 detects the user's position from a camera image. For example, the door determination unit 103 determines whether the user is approaching the vehicle from the right side or the left side of the vehicle. The door determination unit 103 may further determine whether the user is moving toward the front door or the rear door of the vehicle. For example, the door determination unit 103 stores a table indicating the correspondence between the user's position and the door to be controlled. The door determination unit 103 may use the table to determine the door to be controlled according to the user's position. Alternatively, the door determination unit 103 may determine the door closest to the user's position as the door to be controlled.

[0023] When determining the door to be controlled based on the user's line of sight, the door determination unit 103 detects the user's eyes from an image of a face area in a camera image, i.e., the face image, and detects the direction in which the user's eyes are facing. Any known algorithm may be used for the line of sight detection. A plurality of areas corresponding to the plurality of doors 150 are set in the vehicle. The door determination unit 103 determines whether the user is looking at an area corresponding to a specific door based on the detected line of sight of the user. For example, if the user has been looking at an area corresponding to a specific door for a predetermined period of time or more, the door determination unit 103 may determine that specific door as the door to be controlled.

[0024] When determining a door to be controlled based on a user's gesture, the door determination unit 103 detects physical expressions, such as posture, head movement, hand movement, and foot movement, from a camera image. For example, the door determination unit 103 stores a table showing a correspondence between gestures and doors to be controlled. The door determination unit 103 refers to the table and determines whether the user's physical expression indicates a specific gesture associated with a specific door. If the user's physical expression indicates a specific gesture, the door determination unit 103 determines the door associated with the specific gesture as the door to be controlled. The door determination unit 103 may determine one or more doors 150 of a plurality of doors 150 provided in the vehicle as the doors to be controlled. The door determination unit 103 corresponds to the door determination means 11 shown in FIG. 1.

[0025] If the user is successfully authenticated by the face authentication unit 102, the door control unit 104 performs the door operation determined by the door determination unit 103. The door control unit 104, for example, unlocks the door determined by the door determination unit 103. In addition to unlocking the door, the door control unit 104 may also open the door. For example, if the door is an electric sliding door, the door control unit 104 sends a control signal to a controller such as an ECU (Electronic Control Unit) that controls the electric sliding door, and controls a motor that drives the electric sliding door. For example, if the door is a hinged door, the door control unit 104 may open the door via some mechanism. The door control unit 104 corresponds to the door control means 12 shown in FIG. 1.

[0026] The inquiry unit 105 inquires of the user as to whether or not the door determined by the door determination unit 103 can be operated. The inquiry unit 105 notifies the user of the door to be operated, for example, by using sound, light, or the like, and inquires of the user as to whether or not the operation can be performed. The inquiry unit 105 may notify the user of the door to be operated by causing the door control unit 104 to perform an operation to half-open the door determined by the door determination unit 103.

[0027] The inquiry unit 105 acquires the user's response to the inquiry as the inquiry result. The inquiry unit 105 acquires the inquiry result based on, for example, a camera image. The inquiry unit 105 may detect a user gesture from, for example, the camera image, and acquire the inquiry result according to the detected gesture. For example, when the user nods their head, the inquiry unit 105 acquires "yes" as the inquiry result. On the other hand, when the user shakes their head from side to side, the inquiry unit 105 acquires "no" as the inquiry result. The inquiry unit 105 corresponds to the inquiry means 13 shown in FIG. 1 .

[0028] If the result of the inquiry obtained by the inquiry unit 105 indicates a negative result, the door control unit 104 performs an operation to cancel the operation performed on the door determined as the door to be controlled in the first door determination process. Furthermore, the door control unit 104 performs an operation on a door other than the door determined as the door to be controlled in the first door determination process. For example, the door control unit 104 may perform an operation to close the door that was left half-open in the inquiry to the user, and open a door other than that door. If the result of the inquiry indicates a positive result, the door control unit 104 may open the door that was left half-open in the inquiry to the user.

[0029] If the result of the inquiry obtained by the inquiry unit 105 indicates a negative result, the door determination unit 103 may perform a second door determination process based on the camera image to re-determine the door to be controlled. In the second door determination process, the door determination unit 103 detects, for example, at least one of the user's position, the user's line of sight, and the user's gesture from the camera image. The door determination unit 103 re-determines the door to be controlled based on the detected at least one of the user's position, the user's line of sight, and the user's gesture. In the second door determination process, the door determination unit 103 can determine a door different from the door determined in the first door determination process as the door to be controlled. The door control unit 104 performs the operation of the door determined as the door to be controlled in the second door determination process.

[0030] Here, the correspondence between the user's position, the user's line of sight, and the user's gesture and the door to be controlled may be different between the first door determination process and the second door determination process. For example, the action performed by the user to specify a certain door in the first door determination process may be different from the action performed by the user to specify that door in the second door determination process. The door determination unit 103 may detect the user's line of sight and the user's gesture in a shorter time in the second door determination process than in the first door determination process. The second door determination process is considered to be performed when the user is closer to the vehicle than in the first door determination process. If the user's line of sight and the user's gesture are detected in a shorter time, the time the user has to wait until another door is operated can be shortened.

[0031] In the first door determination process, the door determination unit 103 may determine a first candidate door and a second candidate door as candidates for the door to be controlled, and determine the first candidate door as the door to be controlled. In this case, if the result of the inquiry obtained by the inquiry unit 105 indicates a negative result, the door determination unit 103 may determine the second candidate door as the door to be controlled without performing the second door determination process.

[0032] In this embodiment, when a user approaches a vehicle, the door determination unit 103 may perform the first door determination process after the distance between the user and the vehicle becomes within a first distance. In other words, the door determination unit 103 may perform the first door determination process and determine a door to be controlled when the user approaches the vehicle to the first distance. Similarly, the face authentication unit 102 may authenticate the user by face authentication after the distance between the user and the vehicle becomes within the first distance. The determination of a door to be controlled by the door determination unit 103 and the face authentication by the face authentication unit 102 may be performed in parallel.

[0033] The door control unit 104 may operate the door after the distance between the user and the vehicle becomes within a second distance shorter than the first distance. Furthermore, the inquiry unit 105 may notify the user of the door determined to be the door to be controlled in the first door determination process and inquire of the user whether or not to operate the door before the door control unit 104 operates the door to be controlled. In other words, the inquiry unit 105 may inquire of the user whether or not to operate the door when the distance between the user and the vehicle is longer than the second distance.

[0034] For example, when the user approaches within 5 meters of the vehicle, the door determination unit 103 and the face authentication unit 102 determine which door is to be controlled and perform face authentication, respectively. The inquiry unit 105 inquires of the user whether or not to operate the door determined by the door determination unit 103. In response to the inquiry, the door control unit 104 may half-open the door determined by the door determination unit 103.

[0035] The door control unit 104 operates the door when the user approaches within 1 meter of the vehicle. For example, the door control unit 104 operates to open a half-open door. In this case, the determination of the door to be controlled, the face authentication, and the acquisition of the inquiry result can be completed by the time the user approaches the vehicle, and the door can be operated only after the user has sufficiently approached the vehicle.

[0036] If the inquiry unit 105 obtains an inquiry result indicating that the user denies the door operation, and if an operation has already been performed on the door determined in the first door determination process, the door control unit 104 performs an operation to cancel the performed operation. For example, the door control unit 104 performs an operation to close the door that was left half-open in the inquiry. The door determination unit 103 performs a second door determination process to determine the door to be controlled. The door control unit 104 performs the operation on the door determined in the second door determination process.

[0037] Next, the operation procedure will be described. FIG. 3 shows the operation procedure of the vehicle door control device 100. The operation procedure of the vehicle door control device 100 corresponds to a vehicle door control method. The occupant detection unit 101 detects a user around the vehicle based on a camera image from the camera 130 (step S1). The face authentication unit 102 performs face authentication on the user detected in step S1 (step S2). The door determination unit 103 performs a first door determination process and determines a door to be controlled (step S3). In step S3, the door determination unit 103 may detect the user's line of sight or gesture and determine a door to be controlled based on the detected line of sight or gesture.

[0038] The door control unit 104 performs the operation of the door determined in step S3 (step S4). The inquiry unit 105 inquires of the user whether or not to operate the door determined in step S3 (step S5). For example, the inquiry unit 105 may cause the door control unit 104 to perform an operation to half-open the door determined in step S3 in step S4, and notify the user of the door determined in step S3. In step S5, the inquiry unit 105 may output a message such as "The driver's door will be opened. Is this OK?" from a speaker, and inquire of the user whether or not to operate the door.

[0039] The user performs an action indicating a reaction to the inquiry. If the door the user wants to open is the door determined in step S3, the user performs an action to affirm the door operation. On the other hand, if the door the user wants to open is different from the door determined in step S3, the user performs an action to deny the door operation. The inquiry unit 105 determines whether the user has denied the door operation (step S6). If the user has affirmed the door operation, the processing ends. If the user has affirmed the door operation, the door control unit 104 may perform an operation to open the half-open door.

[0040] If the user denies the door operation, the user performs an operation to specify another door. If the user denies the door operation, the door determination unit 103 performs a second door determination process to re-determine the door to be controlled (step S7). The operation based on the inquiry result indicating denial and the operation to specify another door may be performed simultaneously. Furthermore, if the user performs an operation to specify another door, the inquiry unit 105 may determine in step S6 that the user has denied the operation of the door determined in the first door determination process. The door control unit 104 performs an operation to cancel the operation performed in step S4 and performs an operation on the door determined in step S7 (step S8). For example, in step S8, the door control unit 104 may close the door that was left ajar in the inquiry and open the door determined in step S7.

[0041] In this embodiment, when an occupant is detected, the door determination unit 103 performs a first door determination process to determine a door to be controlled. The door control unit 104 operates the door determined in the first door determination process. The inquiry unit 105 inquires of the user whether or not to operate the door determined in the first door determination process. The door determination unit 103 performs a second door determination process to re-determine a door to be controlled. If the inquiry result indicates a negative result, the door control unit 104 performs an operation to cancel the operation of the door determined in the first door determination process and operates the door determined in the second door determination process. In this manner, even if the door determined as the door to be controlled in the first door determination process is different from the door the user wants to operate, the vehicle door control device 100 can operate a different door in accordance with the user's response to the inquiry. In this embodiment, the user can operate the door without touching the vehicle, thereby improving user convenience in vehicle door control.

[0042] Next, the hardware configuration of the vehicle door control device 100 will be described. Fig. 4 shows an example of the hardware configuration of the vehicle door control device 100. The vehicle door control device 100 has a processor (CPU: Central Processing Unit) 501, a ROM (read only memory) 502, and a RAM (random access memory) 503. In the vehicle door control device 100, the processor 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. Although not shown, the vehicle door control device 100 may include other circuits such as peripheral circuits, communication circuits, and interface circuits.

[0043] The ROM 502 is a non-volatile storage device. For example, a semiconductor storage device with a relatively small capacity, such as a flash memory, is used as the ROM 502. The ROM 502 stores the programs executed by the processor 501.

[0044] The program includes instructions (or software code) that, when loaded into a computer, cause the computer to perform one or more functions described in the embodiments. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. By way of example and not limitation, computer-readable media or tangible storage media include RAM, ROM, flash memory, solid-state drive (SSD) or other memory technologies, compact discs (CDs), digital versatile discs (DVDs), Blu-ray discs or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices. The program may also be transmitted on a transitory computer-readable medium or a communication medium. By way of example and not limitation, transitory computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0045] The RAM 503 is a volatile storage device. Various semiconductor memory devices such as a dynamic random access memory (DRAM) or a static random access memory (SRAM) are used for the RAM 503. The RAM 503 can be used as an internal buffer for temporarily storing data and the like.

[0046] The processor 501 loads a program stored in the ROM 502 into the RAM 503 and executes the program. When the CPU 501 executes the program, the functions of each unit in the vehicle door control device 100 can be realized.

[0047] The vehicle door control device 100 does not necessarily have to be a single device. The vehicle door control device 100 may be configured using multiple physically separated devices. Furthermore, all functions of the vehicle door control device 100 do not necessarily have to be installed in the vehicle. For example, in the vehicle door control device 100, at least one of the door determination unit 103 and the face authentication unit 102 may be arranged outside the vehicle. In this case, the functional unit arranged in the vehicle and the functional unit arranged outside the vehicle may transmit and receive information via a wireless communication network.

[0048] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the above-described embodiments, and changes and modifications to the above-described embodiments that do not deviate from the spirit of the present disclosure are also included in the present disclosure.

[0049] 10: Vehicle door control device 11: Door determination means 12: Door control means 13: Inquiry means 100: Vehicle door control device 101: Occupant detection unit 102: Face authentication unit 103: Door determination unit 104: Door control unit 105: Inquiry unit 130: Camera 150: Door 501: Processor 502: ROM 503: RAM 504: Bus

Claims

1. a door determination means for performing a first door determination process for determining a door to be controlled based on an image of a user attempting to get into a vehicle having multiple doors; a door control means for executing the determined door operation; an inquiry means for inquiring of a user as to whether or not the door can be operated, If the result of the inquiry indicates a negative result, the door control means performs an operation to cancel the performed operation on the determined door and performs an operation on a door other than the determined door.

2. When the result of the inquiry indicates a negative result, the door determination means performs a second door determination process to determine, based on the image, a door different from the door determined in the first door determination process as a door to be controlled; The vehicle door control device according to claim 1 , wherein the door control means carries out the operation of the door determined in the second door determination process.

3. 3. The vehicle door control device according to claim 2, wherein the door determination means detects at least one of the user's position, the user's line of sight, and the user's gesture from the image in the first door determination process and the second door determination process, and determines the door to be controlled based on the detected at least one of the user's position, the user's line of sight, and the user's gesture.

4. 4. The vehicle door control device according to claim 3, wherein a correspondence between at least one of the user's position, the user's line of sight, and the user's gesture and the door to be controlled in the first door determination process is different from a correspondence between at least one of the user's position, the user's line of sight, and the user's gesture and the door to be controlled in the second door determination process.

5. The vehicle door control device according to claim 1 , wherein the inquiry unit acquires a result of the inquiry based on the image.

6. 4. The vehicle door control device according to claim 1, wherein the inquiry unit notifies the user of the door to be operated by causing the door control unit to perform an operation to half-open the door determined in the first door determination process.

7. 7. The vehicle door control device according to claim 6, wherein the door control means opens the half-open door when the result of the inquiry indicates a positive result.

8. The vehicle door control device according to claim 6, wherein if the result of the inquiry indicates a negative result, an operation to close the half-open door is carried out.

9. performing a first door determination process for determining a door to be controlled based on an image of a user attempting to get into a vehicle having multiple doors; Implementing the determined door operation; Inquiring of a user as to whether or not the door can be operated; If the result of the inquiry indicates a negative result, an operation is performed on the determined door to undo the performed operation; A vehicle door control method comprising: performing an operation on a door different from the determined door.

10. performing a first door determination process for determining a door to be controlled based on an image of a user attempting to get into a vehicle having multiple doors; Implementing the determined door operation; Inquiring of a user as to whether or not the door can be operated; If the result of the inquiry indicates a negative result, an operation is performed on the determined door to undo the performed operation; A program for causing a processor to execute a process including performing an operation on a door different from the determined door.