Ride assistance system

Abstract

This system provides a passenger assistance system that can accurately and quickly grasp the occupant's intention to open the door, and control the door to open without delay at the necessary time. [Solution] The system is configured to detect an occupant approaching the vehicle, obtain the position and orientation of the detected occupant's hands, obtain the position of the door handle used to open and close the vehicle door, and then, if the conditions are met that the occupant's hand position is within a first range set relative to the door handle position and the occupant's hand orientation is within a second range set relative to the door handle position, control is performed to open the vehicle door.

Description

Technical Field

[0001] The present invention relates to a boarding assistance system that assists a user in boarding a vehicle.

Background Art

[0002] Conventionally, various systems for assisting an occupant to board a vehicle have been proposed. In particular, as assistance for automatically unlocking and opening a vehicle door when the occupant boards the vehicle on the system side, for example, Japanese Unexamined Patent Application Publication No. 2017-096056 proposes a technology for unlocking a door when the occupant touches a door handle. In addition, Japanese Unexamined Patent Application Publication No. 2016-061660 proposes a technology that enables detection that an occupant has approached a vehicle within a predetermined distance by communicating with a mobile terminal carried by the occupant, and unlocks a door when the occupant approaches within the predetermined distance.

Prior Art Literature

Patent Literature

[0003]

Patent Literature 1

Patent Literature 2

Summary of the Invention

Problem to be Solved by the Invention

[0004] Here, when the system automatically unlocks and opens the vehicle doors when boarding a vehicle, a crucial point is that the system accurately and as quickly as possible recognizes the occupant's intention to open the door. In this respect, the technology described in Patent Document 1 unlocks the door only when the occupant touches the door handle, which allows the system to accurately recognize the occupant's intention to open the door, but there is a problem in that the door unlocking is delayed. For example, the door may not be unlocked before the occupant pulls the door handle, potentially requiring the occupant to operate the door handle multiple times.

[0005] On the other hand, the technology described in Patent Document 2 unlocks the doors only when the occupant approaches the vehicle within a predetermined distance. While this allows for immediate unlocking of the doors, it has the problem of not being able to accurately grasp the occupant's intention to open the doors. For example, the doors would be unlocked even if the occupant simply walked past the vehicle without intending to open them.

[0006] The present invention was made to solve the aforementioned problems of the conventional system, and aims to provide a passenger assistance system that can accurately and as quickly as possible grasp the passenger's intention to open the door from the position and orientation of the passenger's hands, and that can control the opening of the door without delay until the necessary time. [Means for solving the problem]

[0007] To achieve the above objective, the passenger assistance system according to the present invention includes: hand information acquisition means for acquiring the position and orientation of the occupant's hands; handle position acquisition means for acquiring the position of the door handle operated to open and close the vehicle door; and door control means for performing control to open the vehicle door when the conditions are met that the position of the occupant's hands is within a first range set with respect to the position of the door handle and the orientation of the occupant's hands is within a second range set with respect to the position of the door handle. Furthermore, the "control for opening the door" may be a control for unlocking the door, a control for opening the door, or both. Furthermore, "occupant" is not limited to the driver (the occupant with the authority to unlock the doors), but may also include passengers. Moreover, it applies even if the occupant opens the doors from outside the vehicle. It is also applicable when the doors are opened from inside the vehicle. [Effects of the Invention]

[0008] According to the passenger assistance system of the present invention having the above configuration, by using multiple conditions consisting of different criteria such as the position and orientation of the passenger's hands as AND conditions for determining the passenger's intention, it becomes possible to accurately grasp the passenger's intention to open the door. Furthermore, it becomes possible to grasp the passenger's intention as early as possible, even before the passenger's hand touches the door handle. As a result, it becomes possible to control the opening of the door without delay until the necessary time. [Brief explanation of the drawing]

[0009] [Figure 1] This is a schematic diagram of the vehicle according to this embodiment. [Figure 2] This is a block diagram showing the configuration of the passenger assistance system according to this embodiment. [Figure 3] This is a flowchart of the passenger assistance processing program according to this embodiment. [Figure 4] This diagram illustrates a method for detecting the position of the crew member's hands. [Figure 5] This diagram illustrates a method for detecting the orientation of a crew member's hands. [Figure 6] This diagram illustrates a method for detecting the position of the crew member's feet. [Figure 7] This diagram illustrates a method for detecting the position of a door handle. [Figure 8] This is a diagram explaining the content of Condition 1. [Figure 9] This is a diagram explaining the content of Condition 2. [Figure 10] This figure illustrates the handle approximation line for condition 2. [Figure 11] This is a diagram explaining the content of Condition 3. [Modes for carrying out the invention]

[0010] Hereinafter, one embodiment of the passenger assistance system according to the present invention will be described in detail with reference to the drawings. First, the vehicle 2 equipped with the passenger assistance system 1 according to this embodiment will be described below. Figure 1 is a schematic diagram of the vehicle 2 according to this embodiment.

[0011] Here, Vehicle 2 may be, for example, an automobile powered by an internal combustion engine (internal combustion engine vehicle), an automobile powered by an electric motor (electric vehicle, fuel cell vehicle, etc.), or an automobile powered by both (hybrid vehicle). Furthermore, there is no restriction on the type of vehicle; it may be a regular passenger car, a large commercial truck, a bus, construction machinery, etc. Also, although the following explanation will refer to it as a four-wheeled vehicle, it may also be a two-wheeled or three-wheeled vehicle.

[0012] Furthermore, Vehicle 2 may be a vehicle capable of not only manual driving based on the user's driving operations, but also assisted driving through automated driving assistance, where the vehicle drives automatically without user input. Alternatively, it may be a vehicle capable of only assisted driving through automated driving assistance. On the other hand, Vehicle 2 is not necessarily limited to a vehicle capable of assisted driving through automated driving assistance as described above, and may be a vehicle capable of manual driving only.

[0013] Further, as shown in FIG. 1, a vehicle 2 that is a right-hand drive vehicle includes a driver's side front door 3R, a front passenger seat side front door 3L, a driver's side rear door 4R, a front passenger seat side rear door 4L, and a back door 5. Hereinafter, when the front doors 3R, 3L, the rear doors 4R, 4L, and the back door 5 are collectively described, they may be referred to as each door. Each door is, for example, a swing-type door. The vehicle 2 also includes door lock devices 6A to 6E that control locking of each door. The door lock devices 6A to 6E control locking of the front doors 3R, 3L, the rear doors 4R, 4L, and the back door 5 in this order. More specifically, the door lock devices 6A to 6E are devices that switch each door between a locked state in which the door is locked and an unlocked state in which the door is unlocked.

[0014] The configuration of the vehicle 2 shown in FIG. 1 is merely an example. For example, the vehicle 2 is not limited to a right-hand drive vehicle, and may be a left-hand drive vehicle. Further, each door is not limited to a swing-type door, and doors of other opening / closing methods such as a sliding door can be employed. In addition, the number of doors varies depending on the vehicle model.

[0015] Further, as shown in FIG. 1, in addition to the aforementioned doors and door lock devices 6A to 6E, the vehicle 2 includes a front camera 7, a rear camera 8, and side cameras 9A, 9B for capturing images around the vehicle, and a riding assistance ECU (Electronic Control Unit) 10 that performs various arithmetic processes based on input information. A riding assistance system 1 includes the riding assistance ECU 10 and each of the aforementioned components.

[0016] Hereinafter, each component included in the vehicle 2 other than the doors and the door lock devices 6A to 6E will be described. First, the front camera 7 is an imaging device including a camera using a solid-state image sensor such as a CCD, for example, and is installed, for example, above the front bumper of the vehicle 2 or on the back side of a rearview mirror with its optical axis directed forward in the traveling direction of the vehicle.

[0017] The rear camera 8 is an imaging device that also has a camera using a solid-state image sensor such as a CCD, and is mounted, for example, near the center above the license plate attached to the rear of the vehicle 2, with the optical axis facing the rear of the vehicle.

[0018] Furthermore, the side cameras 9A and 9B are imaging devices that also have cameras using solid-state image sensors such as CCDs, and are mounted, for example, on the left and right side mirrors of vehicle 2, with the optical axis facing the side of the vehicle.

[0019] The occupant assistance ECU 10 then performs image recognition processing on the images captured by the front camera 7, rear camera 8, and side cameras 9A and 9B to identify the position of the occupant's hands, hand orientation, and foot position as they approach the vehicle. In addition to identifying the position of the occupant's hands, hand orientation, and foot position, each camera can also be used to capture images of the area around the vehicle for display on a vehicle-mounted display, or as a sensor to detect obstacles around the vehicle.

[0020] Furthermore, in addition to the camera mentioned above, vehicle 2 may also be equipped with other sensors for detecting its surroundings, such as ultrasonic sensors, millimeter-wave sensors, laser sensors, etc. In addition to the camera that captures images outside the vehicle, it may also be equipped with a camera that captures images inside the vehicle.

[0021] On the other hand, the passenger assistance ECU 10 is an electronic control unit that performs various processes specifically to assist passengers in getting into vehicle 2. For example, it uses cameras to acquire the position of the passenger's hands, the orientation of their hands, and the position of their feet as they approach the vehicle, and reads the passenger's intention to open the door from this information and the position of the door handle. If the passenger's intention to open the door is read, it controls the door lock devices 6A to 6E to automatically release the door lock of the door the passenger is trying to open. In addition, when acquiring the position of the passenger's hands, the orientation of their hands, and the position of their feet as they approach the vehicle, it performs image recognition processing on the images captured by the aforementioned front camera 7, rear camera 8, and side cameras 9A and 9B. The passenger assistance ECU 10 is connected to the aforementioned door lock devices 6A to 6E, front camera 7, rear camera 8, and side cameras 9A and 9B via an in-vehicle network such as CAN. It is also connected to various sensors mounted on vehicle 2, such as a vehicle speed sensor, acceleration sensor, gyro sensor, steering sensor, shift position sensor, and ultrasonic sensor. The detailed configuration of the occupant assistance ECU10 will be described later.

[0022] In addition to the components shown in Figure 1, Vehicle 2 also has other basic components as Vehicle 2, but only the configuration related to the control of passenger assistance and the control related to said configuration will be explained.

[0023] Next, we will describe in detail the passenger assistance ECU 10, which is part of the passenger assistance system 1 provided by the vehicle 2 described above. Figure 2 is a block diagram showing the configuration of the passenger assistance system 1 according to this embodiment.

[0024] As shown in Figure 2, the passenger assistance ECU (Electronic Control Unit) 10 is an electronic control unit that controls the entire passenger assistance system 1, and includes a CPU 31 as a processing unit and control device, a RAM 32 which is used as working memory when the CPU 31 performs various calculations and stores image recognition results, a ROM 33 which stores control programs as well as passenger assistance processing programs (see Figure 3) described later, and a flash memory 34 which stores programs read from the ROM 33. The passenger assistance ECU 10 also has various means as processing algorithms. For example, the hand information acquisition means acquires the position and orientation of the occupant's hands. The steering wheel position acquisition means acquires the position of the door handle that is operated to open and close the vehicle doors. The door control means controls the opening of the vehicle doors when the position of the occupant's hands is within a first range set based on the position of the door handle and the orientation of the occupant's hands is within a second range set based on the position of the door handle. The foot information acquisition means acquires the position of the occupant's feet detected by the occupant detection means. The condition changing means changes at least one of the first and second ranges to a narrower range if the position of the occupant's feet is within the third range, while the position of the occupant's hands is not within the first range or the orientation of the occupant's hands is not within the second range, for a predetermined period of time or longer. The movement direction acquisition means acquires the direction of movement of the occupant when the occupant detected by the occupant detection means is moving. In other words, the passenger assistance ECU 10 is an example of a hand information acquisition means, a steering wheel position acquisition means, a door control means, a foot information acquisition means, a condition changing means, and a movement direction acquisition means.

[0025] Furthermore, the occupant assistance ECU 10 is also connected to a key radio wave transmission / reception unit 38 that transmits and receives radio waves with a smart key (registered trademark) 37, a type of electronic key held by the occupant, and various sensors 39 for detecting the behavior of the vehicle, such as a vehicle speed sensor, wheel speed sensor, acceleration sensor, gyro sensor, steering sensor, and shift position sensor.

[0026] On the other hand, the key radio wave transmitting / receiving unit 38 is a device for receiving radio waves emitted from the smart key 37 and transmitting radio waves to the smart key 37, and the occupant assistance ECU 10 can use the key radio wave transmitting / receiving unit 38 to detect when an occupant approaches the vehicle 2. Specifically, the key radio wave transmitting / receiving unit 38 of the vehicle 2 constantly emits weak radio waves into its surroundings to search for the target smart key (the smart key held by the occupant) 37. When the occupant holding the target smart key 37 approaches the car within a certain distance and the smart key 37 receives radio waves from the vehicle 2, the smart key 37 emits a response signal, and the key radio wave transmitting / receiving unit 38 catches the response signal from the smart key 37 and compares it with pre-registered authentication information. If a match is found, the unit detects that an occupant has approached the vehicle 2. Furthermore, if there are multiple antennas, the occupant's position can also be determined by triangulation. After detecting that an occupant has approached vehicle 2, the system uses cameras and other means to acquire information such as the position and orientation of the occupant's hands and feet, as described later. Based on this information and the position of the door handle, the system interprets the occupant's intention to open the door. If the system interprets the occupant's intention to open the door, it controls the door lock devices 6A-6E to automatically unlock the door the occupant intends to open. The smart key 37 also has separate buttons for operating the door lock devices 6A-6E and the door opening / closing devices. Therefore, after the smart key 37 has been verified, in addition to the automatic door unlocking described above, it is also possible to unlock and open the doors by pressing the buttons on the smart key 37.

[0027] Furthermore, the flash memory 34 includes a vehicle information DB 35 and a judgment information DB 36. The vehicle information DB 35 stores various information about the vehicle 2. For example, it stores the installation positions of cameras and sensors installed on the vehicle 2 (height from the ground, left-right position), detection axes (optical axis for cameras), overall length, vehicle width, wheelbase, minimum turning radius, and door handle positions. This information is entered in advance by the occupants or personnel from the vehicle manufacturer.

[0028] On the other hand, the judgment information DB36 stores threshold information for determining whether or not the occupant intends to open the doors of vehicle 2, based on the position of their hands, the orientation of their hands, the position of their feet, etc., as they approach the vehicle. The information in the judgment information DB36 may be pre-set by the vehicle manufacturer, for example, or the occupant may set it themselves after purchasing vehicle 2, taking usability into consideration.

[0029] Next, the passenger assistance processing program executed by the passenger assistance ECU 10 in the passenger assistance system 1 having the above configuration will be explained with reference to Figure 3. Figure 3 is a flowchart of the passenger assistance processing program according to this embodiment. Here, the passenger assistance processing program is executed, for example, when the ACC power (accessory power supply) of the vehicle 2 is turned off and the vehicle is parked, and all doors are locked. The program determines whether or not the occupant intends to open the doors of the vehicle 2 based on the position of the occupant's hands, the direction of their hands, the position of their feet, etc., as they approach the vehicle, and if it is determined that the occupant intends to open the doors, the program automatically unlocks the doors that the occupant is expected to open. However, the conditions for starting the execution of the passenger assistance processing program are not limited to the above conditions. For example, it may be executed when the ACC power of the vehicle 2 is on and an occupant has already boarded, in order to assist in the boarding of a new occupant, or it may be executed when some of the doors are unlocked. The program shown in the flowchart in Figure 3 below is stored in the RAM 32 and ROM 33 of the passenger assistance ECU 10 and executed by the CPU 31.

[0030] First, in step 1 (hereinafter abbreviated as S), the CPU 31 determines whether or not an occupant approaching the vehicle, that is, an occupant who is expected to board the vehicle, has been detected in the vicinity of the vehicle. The detection of the vehicle occupant can be performed, for example, using the key radio wave transmitting / receiving unit 38. As mentioned above, the key radio wave transmitting / receiving unit 38 catches the reply signal from the smart key 37 and compares it with pre-registered authentication information. If a match is found, it is detected that an occupant has approached the vehicle 2. In addition to the smart key 37, a smartphone used in a digital key system with similar functionality can also be used to detect an occupant, or another communication terminal capable of wireless communication with the vehicle 2 may be used. Furthermore, in S1, the key may not be matched; the system may detect that an occupant has approached the vehicle 2 as soon as the key radio wave transmitting / receiving unit 38 catches the reply signal from the smart key 37.

[0031] Furthermore, since the passenger assistance performed in the passenger assistance processing program is assistance related to unlocking the doors, the passengers who are eligible for assistance are basically those who possess the smart key 37, that is, passengers who have the authority to unlock the doors. Accordingly, in S1, it is detected whether or not a passenger who possesses the smart key 37 has approached. However, passengers who do not possess the smart key 37 may also be included in the detection target, and for their detection, for example, a camera or ultrasonic sensor may be used to detect the approach of passengers.

[0032] If it is determined that an occupant approaching the vehicle has been detected (S1:YES), the process proceeds to S2. Conversely, if it is determined that no occupant approaching the vehicle has been detected (S1:NO), the boarding assistance processing program is terminated.

[0033] In S2, the CPU 31 activates the front camera 7, the rear camera 8, and the side cameras 9A and 9B. However, if the occupant's position has been determined in advance, only the cameras whose imaging range includes the occupant's position may be activated.

[0034] Next, in S3, the CPU 31 performs image recognition processing on the real-time images captured by the front camera 7, rear camera 8, and side cameras 9A and 9B. Based on the results of the image recognition processing, it obtains the position of the occupants' hands, the orientation of their hands, and the position of their feet, which were detected in S1 (S4-S6). If the occupants' positions have been determined in advance, the image recognition processing may be performed only on the images captured by the cameras whose imaging range includes the occupants' positions.

[0035] First, regarding the process of acquiring the position of the occupant's hands in S4, the CPU 31 executes a pre-stored skeleton detection program to detect the occupant's skeleton contained in the captured image. The skeleton detection program uses, for example, a posture estimation AI to estimate and detect feature points such as the person's face, waist, and joints (wrists, ankles, knees, elbows, shoulders, and neck) from above the clothing, and the occupant's skeleton can be detected by connecting these feature points with line segments. For example, Figure 4 shows an example of skeleton detection performed on an image 41 of an occupant contained in an captured image 40. As shown in Figure 4, in the image 41 of the occupant, the face, waist, and joints (wrists, ankles, knees, elbows, shoulders, and neck) are identified as feature points 42, and the occupant's skeleton is identified by line segments 43 connecting them. Then, the CPU 31 acquires the position coordinates P (hereinafter referred to as hand coordinates) of the feature point 42 corresponding to the wrist among the occupant's skeleton detected using the captured image as the position of the occupant's hand. Note that the positions of both the right and left hands may be acquired, or only the position of the hand closest to the door handle may be acquired. The hand coordinates P are defined in a 2D coordinate system set for the captured image, but they may also be defined in a 3D coordinate system in real space. Furthermore, if the position of the fingertips can be determined by skeletal detection, the position of the fingertips, rather than the position of the wrist, may be used as the position of the occupant's hand.

[0036] Next, regarding the process of acquiring the orientation of the occupant's hands in S5, the CPU 31 detects the occupant's skeleton by executing a pre-stored skeleton detection program, similar to S4. Then, as shown in Figure 5, the CPU 31 uses the captured image to acquire a line L (hereinafter referred to as the elbow-hand extension line) as the orientation of the occupant's hands, which is an extension of the line connecting the feature point 42 corresponding to the elbow and the feature point 42 corresponding to the wrist, towards the fingertips. It is also possible to acquire the orientation of both the right and left hands, or only the orientation of the hand closest to the door handle. The elbow-hand extension line L is identified in a 2D coordinate system set for the captured image, but it may also be identified in a 3D coordinate system in real space. Furthermore, if the position of the fingertips can also be identified by skeleton detection, the line connecting the elbow position to the fingertips position may be acquired as the elbow-hand extension line L.

[0037] Next, regarding the process of acquiring the position of the occupant's feet in S6, the CPU 31 detects the occupant's skeleton by executing a pre-stored skeleton detection program, similar to S4. Then, as shown in Figure 6, the CPU 31 uses the captured image to acquire the position coordinate Q (hereinafter referred to as the foot coordinate) of the feature point 42 corresponding to the ankle among the occupant's skeleton, as the position of the occupant's foot. Note that the positions of both the right and left feet may be acquired, or only the position of the foot closest to the door handle may be acquired. The foot coordinate Q is a coordinate identified in a 2D coordinate system set for the captured image, but it may also be identified in a 3D coordinate system in real space. Furthermore, if the position of the toes can also be identified by skeleton detection, the position of the toes may be acquired as the position of the occupant's foot instead of the ankle.

[0038] Next, in S7, the CPU 31 performs image recognition processing on the real-time images captured by the front camera 7, rear camera 8, and side cameras 9A and 9B to obtain the position of the door handle 45 included in the captured images. For example, as shown in Figure 7, the CPU 31 identifies the door handle 45 included in the captured image 40 using image recognition processing using known feature points or image recognition processing using pattern matching, and obtains the position coordinates R (hereinafter referred to as door handle coordinates) of the identified door handle 45 as the position of the door handle 45. Note that the door handle coordinates R are coordinates identified in a two-dimensional coordinate system set for the captured image, but they may also be identified in a three-dimensional coordinate system in real space. Here, as shown in Figure 1, the vehicle 2 of this embodiment is equipped with a driver's side front door 3R, a passenger side front door 3L, a driver's side rear door 4R, a passenger side rear door 4L, and a back door 5, and a door handle 45 that is operated to open and close the door is provided for each door. In S7, all door handles 45 equipped on the vehicle 2 are targeted. Furthermore, the door handle coordinates R can also be determined without image recognition processing. For example, the positions where the door handles are installed on vehicle 2 can be stored in vehicle information DB35 as vehicle information beforehand, and the coordinates where the door handles are expected to be located within the captured image can be calculated as the door handle coordinates R using this information along with the optical axis and imaging range of the camera.

[0039] Subsequently, in S8, the CPU 31 determines whether the occupant detected in S1 has the intention to open the door of vehicle 2 (which can also be said to be the intention to get into vehicle 2). Specifically, it is determined that the occupant has the intention to open the door of vehicle 2 if all of the following conditions 1 to 3 are met. [Condition 1] As shown in Figure 8, the hand coordinate P, which indicates the position of the hand, is within an area 46 (hereinafter referred to as the first range) with radius W set relative to the position of the door handle 45. The first range 46 is a circle with radius W centered on the door handle coordinate R. Condition 1, in other words, is a condition that the distance from the hand position to the position of the door handle 45 is less than or equal to a predetermined distance W. The distance W can be set as appropriate, but for example, it is set to the distance at which the average time from the occupant's hand, which is located at a distance W from the door handle, until the occupant operates the door handle is 400 ms. Note that 400 ms is the time required from the time the passenger assistance ECU 10 decides to release the door lock until it actually controls the door lock devices 6A~6E to release the door lock. [Condition 2] Condition 1 is that the elbow extension line L, which indicates the direction of the hand as shown in Figure 9, is within a range (hereinafter referred to as the second range) set based on the position of the door handle 45. Condition 2 is, in other words, that the direction of the occupant's hand is pointing towards the door handle 45. The second range is indicated by the handle approximation line V, which is set vertically through the door handle coordinate R as shown in Figure 10, and is a line of predetermined length D centered on the door handle coordinate R. When the elbow extension line L and the handle approximation line V intersect, the elbow extension line L is considered to be within the second range, that is, the direction of the occupant's hand is pointing towards the door handle 45. [Condition 3] As shown in Figure 11, the condition is that the foot coordinate Q, which indicates the position of the foot, is within an area 48 (hereinafter referred to as the third range) with radius H set relative to the position of the door handle 45. The third range 48 is a circle with radius H centered at point J (see Figure 10) where a line drawn vertically downward from the door handle coordinate R intersects the road surface. Condition 3, in other words, is a condition that the distance from the position of the foot to the position of the door handle 45 is less than or equal to a predetermined distance H. Note that the distance H can be set as appropriate, but for example, it is set to 1m.

[0040] Then, if it is determined that all of the above conditions 1 to 3 are met (S8: YES), it is determined that the occupant detected in S1 intends to open the door of vehicle 2. In particular, the passenger assistance system 1 of this embodiment makes it possible to more accurately determine whether or not the occupant intends to open the door of vehicle 2 by judging multiple conditions consisting of different criteria such as the position of the occupant's hands, the direction of their hands, and the position of their feet as an AND condition. The process then proceeds to S9.

[0041] However, for condition 2 above, the conditions may also be further added that the occupant's hand orientation remains within the second range, and the occupant is moving in a direction approaching the door handle. For example, the CPU 31 obtains the direction of the occupant's movement when the occupant detected in S1 is moving. Specifically, the CPU 31 obtains the direction of the occupant's movement by comparing the occupant's position between the preceding and succeeding frames of the camera's captured images. Then, it compares the occupant's direction of movement with the door handle coordinates R to determine whether or not the occupant is moving in a direction approaching the door handle.

[0042] In S9, CPU31 identifies the door to be unlocked. Specifically, the door to be unlocked is the door corresponding to the door handle that serves as the basis for the first to third range when it is determined that the above conditions 1 to 3 are met. In other words, it is the door that corresponds to the door handle that is close to the position of the occupant's hands or feet and that the occupant's hands are pointing towards. Note that the doors to be unlocked are not limited to the doors that are opened when getting in (e.g., front doors 3R, 3L, rear doors 4R, 4L), but also include the doors that are opened when loading luggage (e.g., the back door 5).

[0043] Subsequently, in S10, the CPU 31 operates the door lock devices 6A to 6E corresponding to the door identified in S9, and unlocks the corresponding door. In addition to unlocking the door, the system may also automatically open the door afterward. Furthermore, the key verification of the smart key 37 held by the occupant may be performed when the occupant approaches the vehicle 2 in S1, or when the door is unlocked in S10.

[0044] As a result, occupants can automatically unlock a door simply by moving near the door they want to open and reaching for the door handle, without having to operate the smart key or door handle. In other words, even if the door of vehicle 2 is locked, the lock is already automatically released when the occupant reaches for the door handle and grasps it, so no operation is required to unlock the door. The door can be opened simply by operating the door handle to open it.

[0045] On the other hand, if it is determined that at least one of the above conditions 1 to 3 is not met (S8: NO), the process proceeds to S11.

[0046] In S11, the CPU 31 determines whether the condition 3 has been met for a predetermined time (e.g., 15 seconds) or longer. That is, it determines whether the position of the occupant's feet is within the third range, while the position of the occupant's hands is not within the first range or the orientation of the occupant's hands is not within the second range, and this condition has continued for a predetermined time or longer.

[0047] If it is determined that the conditions in condition 3 have been met for a specified period of time or longer, it is presumed that the occupants are near the door but are not opening it and are performing other tasks. For example, they may be chatting, making phone calls, or using their smartphones. If they are performing such tasks, it is presumed that the occupants do not intend to open the door of vehicle 2, at least at this time.

[0048] However, it is possible that the occupant may then make a hand movement, and although the occupant does not intend to open the door, conditions 1 and 2 may be met by chance. In such cases, it is desirable to avoid unlocking the door. Therefore, if it is determined that the state satisfying condition 3 has continued for a predetermined time or longer (S11: YES), in S12 the CPU 31 narrows the range in which each of conditions 1 and 2 is satisfied (i.e., tightens the conditions).

[0049] Specifically, as a method to narrow the range that satisfies each of conditions 1 and 2, the CPU 31 shortens (for example, by 20%) the lengths of the distance W (Figure 8) that defines the first range and the distance D (Figure 10) that defines the second range. This changes the first and second ranges to narrower ranges. Note that both the first and second ranges may be narrowed, or only one of them may be narrowed. As a result, it is possible to prevent the door lock from being released when the occupant has no intention of opening the door. On the other hand, even if the first and second ranges are narrowed by the process in S12, the process of releasing the door lock when conditions 1 to 3 are met will continue to be performed, so it is also possible to release the door lock if the occupant who was standing still and performing other tasks later decides to open the door.

[0050] In addition to unlocking the doors using the above-mentioned passenger assistance program, occupants can, of course, also manually unlock the doors as before.

[0051] As described in detail above, according to the passenger assistance system 1 and the computer program executed by the passenger assistance system 1 according to this embodiment, an occupant approaching the vehicle is detected (S1), the position and orientation of the detected occupant's hands are obtained (S4, S5), the position of the door handle to be operated to open and close the vehicle door is obtained (S7), and then, if the conditions are met that the position of the occupant's hands is within a first range set based on the position of the door handle and the orientation of the occupant's hands is within a second range set based on the position of the door handle, control is performed to open the vehicle door (S8~S10). Therefore, by using multiple conditions consisting of different criteria, such as the position and orientation of the occupant's hands, as an AND condition for determining the occupant's intention, it becomes possible to accurately grasp the occupant's intention to open the door. Furthermore, it becomes possible to grasp the occupant's intention as early as possible, even before the occupant's hands come into contact with the door handle. As a result, it becomes possible to perform control to open the door without delay until the necessary time. Furthermore, the system acquires the position of the occupant's feet as they approach the vehicle (S6), and if the occupant's hand position is within a first range, the orientation of the occupant's hand is within a second range, and the position of the occupant's feet is within a third range set based on the position of the door handle, the system controls the opening of the vehicle door (S8-S10). By using multiple conditions consisting of different criteria—the occupant's hand position, hand orientation, and foot position—as an AND condition for determining the occupant's intention, it becomes possible to accurately grasp the occupant's intention to open the door. Furthermore, if the position of the occupant's feet is within the third range, but the position of the occupant's hands is not within the first range or the orientation of the occupant's hands is not within the second range, this condition persists for a predetermined period of time or longer (S11: YES), then at least one of the first and second ranges is changed to a narrower range (S12). This prevents the door from being opened when, for example, the occupant is standing still near the door performing other tasks and has no intention of opening the door. On the other hand, even if the first and second ranges become narrower, the door will still be opened if the conditions are met, so it is possible to open the door if the occupant who was standing still performing other tasks later decides to open the door. Furthermore, if the occupant is moving, the direction of the occupant's movement is obtained (S3), and if the conditions are met that the occupant's hand position is within the first range and the orientation of the occupant's hand is within the second range, and the occupant moves in a direction approaching the door handle, control is performed to open the vehicle door (S10). This makes it possible to more accurately determine whether or not the occupant intends to open the door.

[0052] [Note] The embodiments described above also disclose the following inventions. In the following description, the names and expressions of corresponding components in the embodiments, as well as the reference numerals used in the drawings, are indicated in parentheses for reference. However, the components of each invention are not limited to these indications.

[0053] (Invention A) As the orientation of the occupant's hand, the extension of the line from the occupant's elbow to the fingertips (L) is obtained. The system includes a handle approximation line acquisition means (10) that acquires a line passing vertically through the position of the door handle (45) as a handle approximation line (V), The door control means (10) is a passenger assistance system (1) that determines that the orientation of the occupant's hand is within the second range when the extension line and the handle approximation line intersect.

[0054] According to this method, it is possible to accurately determine whether the occupant's hand is pointing towards the door handle, that is, whether the occupant intends to open the door, from the intersection of two straight lines: the extension line and the handle approximation line.

[0055] It should be noted that the present invention is not limited to the embodiments described above, and various improvements and modifications are possible without departing from the spirit of the invention. For example, in this embodiment, when the occupant's intention to open the door is confirmed, the control to open the vehicle door is performed by releasing the door lock (S10). However, this could also be a control to open the door (open and close the door), or both.

[0056] Furthermore, in this embodiment, when the key radio wave transmitting / receiving unit 38 detects that an occupant possessing the smart key 37 (an occupant authorized to unlock the doors) is approaching the vehicle 2 (S1:YES), the position and orientation of the approaching occupant's hands are detected to confirm the occupant's intention to open the doors. However, this can also be performed on occupants other than the one possessing the smart key 37. For example, after an occupant already possessing the smart key 37 approaches the vehicle 2 and unlocks the doors of the vehicle 2, the processing from S3 onwards may be performed on other occupants other than the one possessing the smart key 37 (hereinafter referred to as "passengers"). In that case, processing S1 is unnecessary. Then, when the intention of a passenger to open the door is confirmed (S8:YES), it is possible to perform control to automatically open the door in which the passenger is presumed to be entering (S9, S10). Furthermore, although this embodiment uses the example of a case where an occupant is outside the vehicle and opens the vehicle door from outside, the method can also be applied to cases where an occupant is inside the vehicle and opens the vehicle door from inside, by detecting the occupant's behavior using an in-vehicle camera and targeting the door handle inside the vehicle.

[0057] Furthermore, in this embodiment, if it is determined in S8 that all of the above conditions 1 to 3 are met (S8: YES), it is determined that the occupant intends to open the door of vehicle 2. However, it is not necessarily required that all of conditions 1 to 3 be met; some of the conditions may be met. For example, if only conditions 1 and 2 are met, it may be determined that the occupant intends to open the door of vehicle 2, and the door may be unlocked.

[0058] Furthermore, in this embodiment, the doors that are subject to unlocking by the above-mentioned boarding assistance processing program are not limited to doors that are opened when boarding (e.g., front doors 3R, 3L, rear doors 4R, 4L), but also include doors that are opened when loading luggage (e.g., back door 5). However, it is also possible to target only the doors that are opened when boarding. Alternatively, it is also possible to target only a specific door (e.g., the driver's side door).

[0059] Furthermore, in this embodiment, the passenger assistance ECU 10 of the passenger assistance system 1 executes the processing of the passenger assistance processing program (Figure 3), but the execution entity can be changed as appropriate. For example, it may be configured so that the vehicle control ECU, the control unit of the navigation system, or other in-vehicle devices execute the processing. [Explanation of Symbols]

[0060] 1…Riding assistance system, 2…Vehicle, 3R,3L…Front door, 4R,4L…Rear door, 5…Back door, 6A~6E…Door lock device, 10…Riding assistance ECU (Example of hand information acquisition means, steering wheel position acquisition means, door control means, foot information acquisition means, condition change means, movement direction acquisition means), 31…CPU, 45…Door handle, 46…First range, 48…Third range, V…Steering wheel approximation line (Second range)

Claims

1. A means for acquiring hand information to obtain the position and orientation of the crew member's hands, A handle position acquisition means for acquiring the position of the door handle operated to open and close the vehicle door, A passenger assistance system comprising: a door control means that controls the opening of the vehicle door when the position of the occupant's hand is within a first range set with respect to the position of the door handle, and the orientation of the occupant's hand is within a second range set with respect to the position of the door handle.

2. The system includes foot information acquisition means for acquiring the position of the occupant's feet, The passenger assistance system according to claim 1, wherein the door control means controls the opening of the vehicle door when the position of the occupant's hand is within the first range and the orientation of the occupant's hand is within the second range, and furthermore, the position of the occupant's foot is within the third range set with respect to the position of the door handle.

3. The passenger assistance system according to claim 2, further comprising a condition changing means for changing at least one of the first and second ranges to a narrower range if the position of the passenger's feet is within the third range, while the position of the passenger's hands is not within the first range or the orientation of the passenger's hands is not within the second range, for a predetermined period of time or longer.

4. When the occupant is moving, the system has a means for acquiring the direction of movement of the occupant, The passenger assistance system according to any one of claims 1 to 3, wherein the door control means controls the opening of the vehicle door when the conditions are met that the position of the occupant's hand is within the first range and the orientation of the occupant's hand is within the second range, and the occupant moves in a direction approaching the position of the door handle.

Images