Vehicle door control device, method, and program

Abstract

This invention is intended to make it possible to improve user convenience in vehicle door control. A door determination means (11) performs a first door determination process, and determines a door to be controlled on the basis of an image of a user attempting to board a vehicle having a plurality of doors. A door control means (12) performs an operation of the door determined by the door determination means (11). An inquiry means (13) inquires of the user whether or not the door determined by the door determination means (11) can be operated. When the result of the inquiry indicates negative, the door control means (12) performs an operation canceling the operation performed on the door determined by the door determination means (11), and performs an operation of a door different from the determined door.

Description

Technical Field

[0001] The present disclosure relates to a vehicle door control device, method, and computer-readable medium.

Background Art

[0002] As related art, Patent Document 1 discloses a smart entry system. In the smart entry system described in Patent Document 1, a user holds an electronic key and approaches the vehicle. An electronic key authentication device arranged inside the vehicle performs wireless communication with the electronic key held by the user and authenticates the electronic key. Further, a camera arranged inside the vehicle captures an image of the user holding the electronic key. The user sends a line of sight to a region corresponding to a door that the user wants to select among a plurality of regions set in advance for each door in the vehicle. An arithmetic device detects the line of sight of the user from the image captured by the camera, and determines the door selected by the user based on the detected line of sight.

[0003] The arithmetic device uses an audio output device to send a signal indicating that the door operation is ready to the user. The audio output device 20 outputs a confirmation message such as "May I open the driver's door?" and asks the user whether the determined door can be unlocked and opened. When the user does not shift the line of sight from the region where the line of sight was sent, the door unlocking device unlocks the door selected by the user. When the user shifts the line of sight from the region where the line of sight was sent, the arithmetic device determines that the user has denied the signal indicating that the door operation is ready. In that case, the user manually unlocks the door by touching the handle of the door to be opened.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] In Patent Document 1, a computing device sends a signal to the user indicating that the door is ready for operation, and the user can know which door will be operated by listening to a confirmation message. If the door determined by the computing device is different from the door the user wants to select, the user can avoid operating the wrong door by averting their gaze. However, in Patent Document 1, if the user rejects the signal indicating that the door is ready for operation, the door will not unlock automatically, and the user must touch the door they want to open and manually unlock it. Therefore, it is desirable to improve the convenience of the device that controls vehicle doors.

[0006] In view of the above circumstances, this disclosure aims to provide a vehicle door control device, method, and computer-readable medium that can improve user convenience in vehicle door control. [Means for solving the problem]

[0007] To achieve the above objective, the Disclosure provides, in a first aspect, a vehicle door control device. The vehicle door control device includes a door determination means that performs a first door determination process to determine the door to be controlled based on an image of a user attempting to board a vehicle having multiple doors; a door control means that performs the operation of the determined door; and an inquiry means that inquires with the user whether or not the door can be operated. If the result of the inquiry indicates negligence, the door control means performs an operation to cancel the operation performed on the determined door and performs an operation on a door other than the determined door.

[0008] This disclosure provides, in a second aspect, a vehicle door control method. The vehicle door control method includes performing a first door determination process to determine the door to be controlled based on an image of a user attempting to board a vehicle having multiple doors, performing an operation on the determined door, inquiring with the user whether the door operation is permissible, and if the result of the inquiry indicates negligence, performing an operation to cancel the operation performed on the determined door, and performing an operation on a door other than the determined door.

[0009] This disclosure provides, in a third aspect, a computer-readable medium. The computer-readable medium stores a program that causes a processor to execute a process including: performing a first door determination process to determine the door to be controlled based on an image of a user attempting to board a vehicle having multiple doors; performing the operation of the determined door; querying the user whether the door operation is permissible; if the query result indicates negligence, performing an operation to cancel the operation performed on the determined door; and performing an operation on a door other than the determined door. [Effects of the Invention]

[0010] The vehicle door control device, method, and computer-readable medium relating to this disclosure can enhance user convenience in vehicle door control. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing a schematic configuration example of the vehicle door control device relating to this disclosure. [Figure 2] A block diagram showing a vehicle door control device according to one embodiment of the present disclosure. [Figure 3] A flowchart illustrating the operation procedure of the vehicle's control door system. [Figure 4] A block diagram showing the hardware configuration of the vehicle control system. [Modes for carrying out the invention]

[0012] Prior to describing embodiments of this disclosure, an overview of this disclosure will be provided. Figure 1 shows a schematic configuration example of a vehicle door control device according to this disclosure. The vehicle door control device 10 includes a door determination means 11, a door control means 12, and a query means 13.

[0013] The door determination means 11 performs a first door determination process based on an image of a user attempting to board the vehicle and determines which of the vehicle's multiple doors is to be controlled. The door control means 12 performs the operation of the door determined by the door determination means 11. The inquiry means 13 inquires with the user whether the door operation performed by the door control means 12 is permissible. If the inquiry result indicates negligence, the door control means 12 performs an operation to cancel the operation performed on the door determined in the first door determination process. The door control means 12 also performs an operation on a door different from the one determined in the first door determination process.

[0014] In this disclosure, if the query result obtained by the query means 13 indicates a negation, the door control means 12 performs an operation to cancel the operation performed on the door determined in the first door determination process, and performs an operation on a door different from the door determined in the first door determination process. In this disclosure, if the door determined as the door to be controlled in the first door determination process is different from the door that the user wants to operate, the door control means 12 performs an operation on a different door. In this way, even if the door determined as the door to be controlled in the first door determination process is different from the door that the user wants to operate, the user does not need to touch the vehicle. Therefore, the vehicle door control device 10 can improve user convenience in vehicle door control.

[0015] The embodiments of this disclosure will be described in detail below with reference to the drawings. Note that the following description and drawings have been omitted and simplified as appropriate for clarity of explanation. Furthermore, in each drawing, the same elements and similar elements are denoted by the same reference numerals, and redundant explanations have been omitted where necessary.

[0016] Figure 2 shows a vehicle door control device according to one embodiment of the present disclosure. The vehicle door control device 100 includes an occupant detection unit 101, a facial recognition unit 102, a door determination unit 103, a door control unit 104, and an inquiry unit 105. Physically, the vehicle door control device 100 may be configured as a device having, for example, one or more memories and one or more processors. At least some of the functions of each part within the vehicle door control device 100 may be realized by one or more processors operating in accordance with instructions read from one or more memories. The vehicle door control device 100 is mounted, for example, on a vehicle. The vehicle door control device 100 corresponds to the vehicle door control device 10 shown in Figure 1.

[0017] The occupant detection unit 101 acquires camera images from the camera 130. The camera 130 is mounted on the vehicle and captures images of the area around the vehicle. The camera 130 captures images of people attempting to get into the vehicle, i.e., occupants, around the vehicle. Occupants of a vehicle are also called users. The camera 130 may be embedded, for example, in the B-pillar of the vehicle, i.e., the pillar between the front door and the rear door.

[0018] Although only one camera 130 is shown in Figure 2, the number of cameras 130 is not limited to one. The occupant detection unit 101 may acquire camera images from each of multiple cameras 130. For example, the occupant detection unit 101 may acquire camera images from a camera that photographs the area on the left side of the vehicle, a camera that photographs the area on the right side of the vehicle, and a camera that photographs the area at the rear of the vehicle.

[0019] The occupant detection unit 101 uses camera images to detect users attempting to board the vehicle. For example, the occupant detection unit 101 extracts a face image or face region of a person near the vehicle from the camera images. If a face image or face region is extracted, the occupant detection unit 101 determines that a user has been detected.

[0020] The face authentication unit 102 authenticates the user detected by the passenger detection unit 101. In the present embodiment, the face authentication unit 102 authenticates the user by face authentication using the camera image of the camera 130. For example, the face authentication unit 102 stores the face image of a person permitted to board the vehicle or the feature amount extracted from the face image. The face authentication unit 102 compares the face image included in the camera image or the feature amount extracted from the face image with the registered face image or feature amount. The face authentication unit 102 calculates a collation score indicating the degree of similarity between the face image included in the camera image or the feature amount extracted from the face image and the registered face image or feature amount. When the collation score is greater than or equal to a predetermined threshold value, the face authentication unit 102 determines that the face authentication of the user has succeeded.

[0021] The door determination unit 103 performs a first door determination process for determining the door to be controlled based on the camera image. In the first door determination process, the door determination unit 103 detects at least one of the position of the user, the line of sight of the user, and the gesture of the user from the camera image, for example. The door determination unit 103 determines the door to be controlled based on at least one of the detected position of the user, the line of sight of the user, and the gesture of the user. The door determination unit 103 may determine the door to be controlled according to the combination of the detected position of the user, the line of sight of the user, and the gesture of the user.

[0022] When the door determination unit 103 determines the door to be controlled based on the position of the user, it detects the position of the user from the camera image. The door determination unit 103 determines, for example, whether the user is approaching the vehicle from the right side of the vehicle or the left side. The door determination unit 103 may further determine whether the user is moving toward the front seat door of the vehicle or the rear seat door. For example, the door determination unit 103 holds a table showing the correspondence between the position of the user and the door to be controlled. The door determination unit 103 may use the table to determine the door to be controlled according to the position of the user. Alternatively, the door determination unit 103 may determine the door closest to the position of the user as the door to be controlled.

[0023] When the door determination unit 103 determines the door to be controlled based on the user's gaze, it detects the user's eyes from the image of the face region in the camera image, i.e., the face image, and detects which direction the user's eyes are facing. Any known algorithm can be used for gaze detection. The vehicle has multiple regions set, each corresponding to one of the multiple doors 150. Based on the detected user gaze, the door determination unit 103 determines whether the user is looking at the region corresponding to a particular door. For example, if the user has been looking at the region corresponding to a particular door for a predetermined amount of time or longer, the door determination unit 103 may determine that particular door to be controlled.

[0024] When the door determination unit 103 determines the door to be controlled based on the user's gestures, it detects physical expressions such as posture, head movements, hand movements, and foot movements from the camera image. For example, the door determination unit 103 maintains a table showing the correspondence between gestures and the doors to be controlled. The door determination unit 103 refers to the table and determines whether the user's physical expression indicates a specific gesture associated with a particular door. If the user's physical expression indicates a specific gesture, the door determination unit 103 determines the door associated with that specific gesture as the door to be controlled. The door determination unit 103 may determine one or more doors 150 out of a plurality of doors 150 provided in the vehicle as the doors to be controlled. The door determination unit 103 corresponds to the door determination means 11 shown in Figure 1.

[0025] If the user has been successfully authenticated by the facial recognition unit 102, the door control unit 104 performs the door operation determined by the door determination unit 103. For example, the door control unit 104 unlocks the door determined by the door determination unit 103. In addition to unlocking the door, the door control unit 104 may also open the door. If the door is an electric sliding door, for example, the door control unit 104 sends a control signal to a controller such as an ECU (Electronic Control Unit) that controls the electric sliding door and controls the motor that drives the electric sliding door. If the door is a hinged door, for example, the door control unit 104 may open the door via some mechanism. The door control unit 104 corresponds to the door control means 12 shown in Figure 1.

[0026] The inquiry unit 105 inquires with the user whether the door determined by the door determination unit 103 can be operated. The inquiry unit 105 notifies the user of the door to be operated, for example, using voice and light, and asks the user whether the operation can be performed. The inquiry unit 105 may also notify the user of the door to be operated by having the door control unit 104 perform an operation to partially open the door determined by the door determination unit 103.

[0027] The inquiry unit 105 obtains the user's response to the inquiry as the result of the inquiry. The inquiry unit 105 obtains the result of the inquiry based on, for example, a camera image. The inquiry unit 105 may also detect the user's gesture from, for example, a camera image and obtain the result of the inquiry according to the detected gesture. For example, if the user nods their head up and down, the inquiry unit 105 obtains "affirmative" as the inquiry result. On the other hand, if the user shakes their head from side to side, the inquiry unit 105 obtains "negative" as the inquiry result. The inquiry unit 105 corresponds to the inquiry means 13 shown in Figure 1.

[0028] If the query obtained by the query unit 105 indicates a negative result, the door control unit 104 performs an operation to cancel the operation performed on the door determined to be controlled in the first door determination process. Alternatively, the door control unit 104 may perform an operation on a door different from the door determined to be controlled in the first door determination process. For example, the door control unit 104 may perform an operation to close a door that was left half-open in response to a query to the user and open a different door. If the query result indicates a positive result, the door control unit 104 may open a door that was left half-open in response to a query to the user.

[0029] If the query obtained by the query unit 105 indicates a negation, the door determination unit 103 may perform a second door determination process based on the camera image and determine the door to be controlled again. In the second door determination process, the door determination unit 103 detects, for example, the user's position, the user's gaze, and at least one of the user's gestures from the camera image. Based on the detected user's position, the user's gaze, and at least one of the user's gestures, the door determination unit 103 determines the target door again. In the second door determination process, the door determination unit 103 may determine a door different from the door determined in the first door determination process as the door to be controlled. The door control unit 104 performs the operation of the door determined as the door to be controlled in the second door determination process.

[0030] Here, the correspondence between the user's position, gaze, and gestures and the door to be controlled may differ between the first door detection process and the second door detection process. For example, the actions performed by the user to specify a door in the first door detection process may differ from the actions performed by the user to specify that door in the second door detection process. In the second door detection process, the door detection unit 103 may detect the user's gaze and gestures in a shorter time than in the first door detection process. The second door detection process is assumed to be performed when the user is closer to the vehicle than in the first door detection process. If the user's gaze and gestures are detected in a shorter time, the time the user has to wait until another door is operated can be reduced.

[0031] In the first door determination process, the door determination unit 103 may determine a first candidate door and a second candidate door as candidates for the door to be controlled, and may decide that the first candidate door is the door to be controlled. In that case, if the query result obtained by the query unit 105 indicates a negation, the door determination unit 103 may decide that the second candidate door is the door to be controlled without performing the second door determination process.

[0032] In this embodiment, the door determination unit 103 may perform the first door determination process after the distance between the user and the vehicle becomes within a first distance when the user approaches the vehicle. In other words, the door determination unit 103 may perform the first door determination process and determine the door to be controlled when the user approaches the vehicle to a first distance. Similarly, the facial recognition unit 102 may authenticate the user by facial recognition after the distance between the user and the vehicle becomes within a first distance. The determination of the door to be controlled by the door determination unit 103 and the facial recognition by the facial recognition unit 102 may be performed in parallel.

[0033] The door control unit 104 may perform door operation after the distance between the user and the vehicle has become shorter than the first distance, within a second distance. Alternatively, the inquiry unit 105 may notify the user of the door determined to be controlled in the first door determination process and inquire with the user whether or not to operate the door before the door control unit 104 performs the operation of the controlled door. In other words, the inquiry unit 105 may inquire with the user whether or not to operate the door when the distance between the user and the vehicle is longer than the second distance.

[0034] For example, the door determination unit 103 and the facial recognition unit 102 determine which door to control and perform facial recognition, respectively, when the user approaches within 5 meters of the vehicle. The inquiry unit 105 asks the user whether it is permissible to operate the door determined by the door determination unit 103. In response to the inquiry, the door control unit 104 may partially open the door determined by the door determination unit 103.

[0035] The door control unit 104 operates the door when the user approaches within 1 meter of the vehicle. For example, the door control unit 104 operates to open a partially open door. In this case, the door control unit 104 can complete the determination of the door to be controlled, facial recognition, and acquisition of the query results before the user approaches the vehicle, allowing the door to be operated only after the user has approached the vehicle sufficiently.

[0036] If the inquiry unit 105 obtains an inquiry result in which the user denies operating the door, and if the operation on the door determined in the first door determination process has already been performed, the door control unit 104 performs an operation to cancel the performed operation. For example, the door control unit 104 performs an operation to close the door that was left half-open in the inquiry. The door determination unit 103 performs a second door determination process to determine the door to be controlled. The door control unit 104 performs the operation on the door determined in the second door determination process.

[0037] Next, the operating procedure will be described. Figure 3 shows the operating procedure of the vehicle door control device 100. The operating procedure of the vehicle door control device 100 corresponds to the vehicle door control method. The occupant detection unit 101 detects a user around the vehicle based on the camera image of the camera 130 (step S1). The face recognition unit 102 performs face recognition on the user detected in step S1 (step S2). The door determination unit 103 performs a first door determination process and determines the door to be controlled (step S3). In step S3, the door determination unit 103 may detect the user's gaze or gesture and determine the door to be controlled based on the detected gaze or gesture.

[0038] The door control unit 104 performs the door operation determined in step S3 (step S4). The inquiry unit 105 asks the user whether the door operation determined in step S3 is permissible (step S5). For example, in step S4, the inquiry unit 105 may cause the door control unit 104 to perform an operation to partially open the door determined in step S3, and notify the user of the door determined in step S3. In step S5, the inquiry unit 105 may output a message such as "The driver's side door will be opened. Is that alright?" from the speaker, and ask the user whether the door operation is permissible.

[0039] The user performs an action to indicate a response to the inquiry. If the door the user wants to open is the door determined in step S3, the user performs an action to affirm the operation of the door. On the other hand, if the door the user wants to open is different from the door determined in step S3, the user performs an action to deny the operation of the door. The inquiry unit 105 determines whether the user has denied the operation of the door (step S6). If the user affirms the operation of the door, the process ends. If the user affirms the operation of the door, the door control unit 104 may perform an operation to open the partially open door.

[0040] If the user rejects the operation of a door, they perform an action to specify a different door. If the user rejects the operation of a door, the door determination unit 103 performs a second door determination process and determines the door to be controlled again (step S7). The action of the query result indicating rejection and the action of specifying a different door may be performed simultaneously. Also, if the user performs an action to specify a different door, in step S6, the query unit 105 may determine that the user rejected the operation of the door determined in the first door determination process. The door control unit 104 performs an action to cancel the operation performed in step S4 and performs the operation on the door determined in step S7 (step S8). For example, in step S8, the door control unit 104 may close the door that was left half-open in the query and open the door determined in step S7.

[0041] In this embodiment, when an occupant is detected, the door determination unit 103 performs a first door determination process to determine the door to be controlled. The door control unit 104 performs the operation of the door determined in the first door determination process. The inquiry unit 105 inquires with the user whether or not the operation of the door determined in the first door determination process is permitted. The door determination unit 103 performs a second door determination process to determine the door to be controlled again. If the result of the inquiry indicates negation, the door control unit 104 performs an operation to cancel the operation on the door determined in the first door determination process and performs the operation of the door determined in the second door determination process. In this way, even if the door determined as the door to be controlled in the first door determination process is different from the door that the user wants to operate, the vehicle door control device 100 can perform the operation of a different door in response to the user's response to the inquiry. In this embodiment, since the user can operate the door without touching the vehicle, user convenience in vehicle door control can be improved.

[0042] Next, the hardware configuration of the vehicle door control device 100 will be described. Figure 4 shows an example of the hardware configuration of the vehicle door control device 100. The vehicle door control device 100 has a processor (CPU: Central Processing Unit) 501, a ROM (read-only memory) 502, and a RAM (random access memory) 503. In the vehicle door control device 100, the processor 501, the ROM 502, and the RAM 503 are interconnected via a bus 504. The vehicle door control device 100 may include other circuits such as peripheral circuits, communication circuits, and interface circuits, although these are not shown in the figure.

[0043] ROM 502 is a non-volatile memory device. A semiconductor memory device, such as a relatively small-capacity flash memory, is used for ROM 502. ROM 502 stores the program executed by the processor 501.

[0044] The above program, when loaded into a computer, includes a set of instructions (or software code) for causing the computer to perform one or more of the functions described in the embodiments. The program may be stored in a non-temporary computer-readable medium or a physical storage medium. Examples, but not limited to, include RAM, ROM, flash memory, solid-state drive (SSD) or other memory technologies, Compact Disc (CD), digital versatile disc (DVD), Blu-ray® disc or other optical disc storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices. The program may be transmitted over a temporary computer-readable medium or a communication medium. Examples, but not limited to, include, a temporary computer-readable medium or a communication medium that includes electrically, optically, acoustically, or otherwise propagating signals.

[0045] RAM503 is a volatile memory device. Various semiconductor memory devices, such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory), can be used for RAM503. RAM503 can be used as an internal buffer for temporarily storing data.

[0046] The processor 501 loads the program stored in the ROM 502 into the RAM 503 and executes the program. By executing the program, the CPU 501 can realize the functions of each part within the vehicle door control device 100.

[0047] Furthermore, the vehicle door control device 100 does not necessarily have to be a single device. The vehicle door control device 100 may be composed of multiple physically separated devices. Also, the vehicle door control device 100 does not necessarily have to have all its functions mounted on the vehicle. For example, in the vehicle door control device 100, at least one of the door determination unit 103 and the facial recognition unit 102 may be located outside the vehicle. In that case, the functional unit located on the vehicle and the functional unit located outside the vehicle may transmit and receive information via a wireless communication network.

[0048] Although embodiments of this disclosure have been described in detail above, this disclosure is not limited to the embodiments described above, and modifications and changes to the embodiments described above, without departing from the spirit of this disclosure, are also included in this disclosure. [Explanation of symbols]

[0049] 10: Vehicle door control device 11: Door detection means 12: Door control means 13: Inquiry methods 100: Vehicle door control device 101: Crew detection unit 102: Facial Recognition Department 103: Door detection unit 104: Door Control Unit 105: Inquiry Department 130: Camera 150: Door 501: Processor 502:ROM 503:RAM 504: Bus

Claims

1. A door determination means performs a first door determination process that determines the door to be controlled based on an image taken of a user attempting to board a vehicle with multiple doors, A door control means for performing the determined door operation, The system includes an inquiry means for inquiring with the user about whether the door can be operated, If the result of the above inquiry indicates a negation, the door control means performs an operation to cancel the operation performed on the determined door, and performs an operation on a door other than the determined door. The inquiry means is a vehicle door control device that notifies the user of the door to be operated by causing the door control means to perform an operation to partially open the door determined in the first door determination process.

2. If the result of the inquiry indicates negation, the door determination means performs a second door determination process to determine a door different from the door determined in the first door determination process based on the image as the door to be controlled. The vehicle door control device according to claim 1, wherein the door control means performs the operation of the door determined in the second door determination process.

3. The vehicle door control device according to claim 2, wherein the door determination means detects at least one of the user's position, the user's gaze, and the user's gesture from the image in the first door determination process and the second door determination process, and determines the door to be controlled based on the detected at least one of the user's position, the user's gaze, and the user's gesture.

4. The vehicle door control device according to claim 3, wherein the correspondence between the user's position, the user's gaze, and at least one of the user's gestures in the first door determination process and the door to be controlled is different from the correspondence between the user's position, the user's gaze, and at least one of the user's gestures in the second door determination process and the door to be controlled.

5. The vehicle door control device according to any one of claims 1 to 3, wherein the query means obtains the result of the query based on the image.

6. If the result of the aforementioned inquiry indicates a positive result, the door control means opens the partially open door, as described in claim 1.

7. If the result of the aforementioned inquiry indicates a negation, the vehicle door control device according to claim 1, which performs the operation to close the partially open door.

8. A first door determination process is performed to determine the door to be controlled based on an image taken of a user attempting to board a vehicle with multiple doors. By performing the operation to partially open the aforementioned determined door, the user is notified of the door being operated. The user is asked whether the aforementioned door can be operated. If the result of the above inquiry indicates a negative result, the operation performed on the determined door shall be canceled. A vehicle door control method that includes performing an operation on a door different from the previously determined door.

9. A first door determination process is performed to determine the door to be controlled based on an image taken of a user attempting to board a vehicle with multiple doors. By performing the operation to partially open the aforementioned determined door, the user is notified of the door being operated. The user is asked whether the aforementioned door can be operated. If the result of the above inquiry indicates a negative result, the operation performed on the determined door shall be canceled. A program that causes a processor to perform a process that includes operating a door different from the previously determined door.

Images