Vehicle system and vehicle control method

Abstract

A vehicle system and vehicle control method are provided. Even when a pedestrian is present, the system can appropriately detect when a person is alighting from the vehicle, thereby enabling appropriate vehicle control and improving convenience. In the vehicle system (100), the alighting detection unit (17) performs a first detection process and a second detection process. In the first detection process, it detects whether a first condition is met based on an image captured by the person detection unit (16) that the person is present around the door (3a) of the vehicle (1). In the second detection process, it detects whether a second condition is met that the detected person is leaving the vehicle (1). If both the first and second conditions are met, the detected person is identified as a person alighting from the vehicle (1). The vehicle system (100) includes a vehicle control unit (15), which performs vehicle control during alighting when the alighting detection unit (17) detects that the detected person is a person alighting from the vehicle (1).

Description

Technical Field

[0001] This invention relates to vehicle systems and vehicle control methods. Background Technology

[0002] In a control system for controlling a vehicle, a structure is disclosed that includes the following components: a person detection unit that detects people based on a captured image composed of dynamic images obtained by capturing images of the outside of the vehicle, and an exit detection unit that detects people outside the vehicle as exiting persons by reducing the image area of ​​the detected people from a value above a predetermined value (for example, see Patent Document 1).

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2023-148786 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] However, the previous structure may have mistakenly detected passersby as people getting off the vehicle. Therefore, it is not possible to accurately identify people getting off the vehicle when passersby are present, which may lead to inappropriate vehicle control when people get off the vehicle and reduce convenience.

[0008] The present invention was made in view of the above background, and its object is to properly detect when a person gets off the vehicle even in the presence of a pedestrian, thereby properly controlling the vehicle and improving convenience.

[0009] Methods for solving problems

[0010] As one aspect of this disclosure, a vehicle system is provided, comprising: a person detection unit that detects people based on a captured image composed of dynamic images obtained by photographing the exterior of the vehicle; and an alighting detection unit that detects whether the detected person is a person alighting from a target vehicle. The alighting detection unit performs a first detection process and a second detection process. In the first detection process, it detects whether a first condition is met based on the captured image, indicating that the detected person is present near a target door of the target vehicle. In the second detection process, it detects whether a second condition is met, indicating that the detected person is leaving the target vehicle. If both the first and second conditions are met, the alighting detection unit detects that the detected person is a person alighting from the target vehicle. The vehicle system also includes a vehicle control unit that performs vehicle control during alighting when the alighting detection unit detects that the detected person is a person alighting from the target vehicle.

[0011] As another aspect of this disclosure, a vehicle control method executed by a computer is provided, wherein the vehicle control method includes: a person detection step, wherein a person is detected by a person detection unit based on a captured image composed of dynamic images obtained by capturing images of the exterior of the vehicle; an alighting detection step, wherein the alighting detection unit detects whether the detected person is alighting from a target vehicle; and a vehicle control step, wherein if the detected person is alighting from the target vehicle, the vehicle control unit performs vehicle control during alighting, wherein the alighting detection step includes a first detection process and a second detection process, wherein the first detection process detects whether a first condition is met based on the captured image, indicating that the detected person is present around the target door of the target vehicle, and the second detection process detects whether a second condition is met, indicating that the detected person is leaving the target vehicle, and if both the first and second conditions are met, the detected person is alighting from the target vehicle.

[0012] Invention Effects

[0013] According to the present invention, even in the presence of pedestrians, the disembarkation of a person from the target vehicle can be properly detected, thereby enabling proper vehicle control and improving convenience. Attached Figure Description

[0014] Figure 1 It is a structural diagram of a vehicle equipped with a vehicle system.

[0015] Figure 2 This is a diagram showing the shooting range of the camera on the left.

[0016] Figure 3 This is a block diagram showing the vehicle's control system.

[0017] Figure 4 This is an example of a captured image.

[0018] Figure 5 This is a diagram used to illustrate the second detection process.

[0019] Figure 6 This is a flowchart illustrating the operation of the vehicle control device.

[0020] Explanation of reference numerals in the attached figures

[0021] 1…Vehicle, 1a…Body, 1b…Front pillar, 1c…Middle pillar, 1d…Rear pillar, 1e…Side beam, 2a…Driver's seat (front seat), 2b…Passenger's seat (front seat), 2c~2e…Rear seat (rear seat), 3a…Left door (object door), 3b…Right door, 4a~4d…Door lock mechanism, 12…Information acquisition unit, 13…Operation input unit, 15…Vehicle control unit, 15a…Door control unit, 16…Person detection unit, 17…Disembarkation detection unit, 17a…Person tracking unit, 17b…First detection processing unit, 17c…Second detection processing unit, 41a, 41b…Camera, 100…Vehicle system, Ar1…Object area, U1~U2…User, U1a…Disembarking user, M1…Passerby, P1, P2…Image captured, S…Car compartment, V…Other vehicles, W…Frame, SR…Designated area. Detailed Implementation

[0022] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

[0023] Figure 1 This diagram illustrates the structure of a vehicle 1 equipped with a vehicle system 100 according to an embodiment of the present invention. The vehicle system 100 comprises a vehicle control device 10 installed in the vehicle 1 and door lock mechanisms 4a to 4d installed in the vehicle 1 and electrically connected to the vehicle control device 10. In the following description, unless otherwise specified, directions such as front-back, left-right, and up-down refer to directions relative to the vehicle body 1a.

[0024] [1. Vehicle Structure]

[0025] Reference Figure 1 The structure of vehicle 1 will be described.

[0026] Vehicle 1 is a four-wheeled automobile, having a body 1a and the body 1a having multiple opening and closing bodies that allow the passenger compartment S to open to the outside of the vehicle. The aforementioned body 1a has a passenger space for passengers to ride in, namely the passenger compartment S.

[0027] The vehicle compartment S is equipped with a driver's seat 2a, a front passenger seat 2b, a rear left seat 2c, a rear center seat 2d, and a rear right seat 2e. Figure 1 The diagram shows a situation where user U1, as the driver, is sitting in the driver's seat 2a, and user U2, as a passenger, is sitting in the front passenger seat 2b.

[0028] Vehicle 1 has a body 1a consisting of a front pillar 1b, a middle pillar 1c, a rear pillar 1d, a side beam 1e, and a roof panel 1f surrounding the passenger compartment S. However, the structure of vehicle 1, including body 1a, can be appropriately modified.

[0029] Vehicle 1 has a left door 3a, a right door 3b, a rear left door 3c, and a rear right door 3d as multiple opening and closing bodies. The left door 3a opens when the user U1 in the driver's seat 2a moves out of the vehicle and when the user U1 sits in the driver's seat 2a from outside the vehicle. The right door 3b opens when the user U2 in the passenger seat 2b moves out of the vehicle and when the user U2 sits in the passenger seat 2b from outside the vehicle. Furthermore, the rear left door 3c and the rear right door 3d open when the users in the rear seats 2c-2e move out of the vehicle and when the users sit in the rear seats 2c-2e from outside the vehicle. In this embodiment, each of the doors 3a-3d is a hinged door.

[0030] Vehicle 1 is equipped with door lock mechanisms 4a to 4d for locking / unlocking each door 3a to 3d and door sensors 5a to 5d for detecting the opening and closing of each door 3a to 3d.

[0031] exist Figure 1 In the diagram, a double-dotted line indicates that the left door 3a is open, showing the driver (user U1) who has just moved outside the car after opening the left door 3a, i.e., the user U1a who has gotten out of the car.

[0032] Vehicle 1 includes a steering wheel 31, a gear selector 32, an ignition switch 33, and a touch panel 34, serving as the operating system used by the user U1 seated in the driver's seat 2a for driving and other purposes. The gear selector 32 is an operating device for selecting gears and driving modes, and is configured as a gear lever or mode switch depending on the specifications of vehicle 1. In addition, vehicle 1 includes a microphone 35 for collecting voices within the passenger compartment S and a speaker 36 for outputting voices into the passenger compartment S.

[0033] Furthermore, vehicle 1 is equipped with cameras 41a and 41b that capture images of the exterior of the vehicle.

[0034] Camera 41a is mounted on the left center pillar 1c and is an exterior camera capable of capturing images of user U1, i.e., user U1a who is the driver, exiting the vehicle by opening the left door 3a. Camera 41a can also capture images of user U1a leaving the vehicle 1.

[0035] In this embodiment, vehicle 1 is a left-hand drive vehicle, which can use the left-hand camera 41a to capture the driver getting off the vehicle.

[0036] The right-side camera 41b is positioned symmetrically to the left-side camera 41a, and can capture the driver (user U1a) exiting the vehicle when it is a right-hand drive vehicle. In a left-hand drive vehicle, the right-side camera 41b can capture the user U2 exiting from the passenger seat 2b. Aside from being symmetrically positioned to the left and right of camera 41a, the right-side camera 41b has the same function as camera 41a; therefore, descriptions related to the right-side camera 41b are omitted.

[0037] Reference Figure 2 The shooting range of the left-side camera 41a will be explained.

[0038] like Figure 2 As shown, camera 41a is capable of capturing... Figure 2 The person within the object area Ar1 shown is positioned with the camera 41a set to the angle, horizontal view, and vertical view.

[0039] In this embodiment, the target area Ar1 is set to be such that, in addition to being able to photograph the front seat user U1a who exits the vehicle by opening the left door 3a, it is also able to photograph the rear seat user who exits the vehicle by opening the rear left door 3c, i.e., the user who exits from the rear seats 2c to 2e on the left side of the vehicle. Furthermore, by using a wide-angle lens in the lens of camera 41a, it is possible to photograph... Figure 2 The large object area Ar1 shown is occupied by people. Therefore, in addition to capturing the user U1 who gets off from the left door 3a, the camera 41a can also capture the user who gets off from the rear left door 3c.

[0040] For example, with vehicle 1 as a reference, camera 41a captures images of people within a fan-shaped object area Ar1 with a horizontal angle of 90° to 150° and a vertical angle of 90° to 150°, at least 1.6m away from the vehicle. This allows camera 41a to capture images of all users exiting the vehicle on the left side and to continue capturing images until the user has moved approximately 1.6m away from vehicle 1.

[0041] Since the camera 41a is mounted on the center pillar 1c, it can also capture images of the open left door 3a, which is located on the opposite side of the camera 41a, across from the alighting user U1a. In other words, the camera 41a can capture images of at least the open left door 3a and the alighting user U1a located between the left door 3a and the camera 41a.

[0042] Figure 2 The multiple routes RT1 to RT5 shown are examples of pedestrian movement routes on the left side of vehicle 1. These routes RT1 to RT5 pass through the object area Ar1, so pedestrians passing through routes RT1 to RT5 are likely to be captured by camera 41a with a high probability.

[0043] Camera 41a is configured to capture at least the upper body, including the face, of a person within the target area Ar1. For example, by positioning camera 41a horizontally or slightly downwards around the upper perimeter of the center pillar 1c, it captures at least the upper body, including the face of the alighting user U1a and passersby. By capturing the upper body, including the face, the detection accuracy of the alighting user U1a is improved. Furthermore, camera 41a can capture a roughly full-body image even when the alighting user U1a has moved a certain distance from the vehicle 1.

[0044] Camera 41a is a color or monochrome motion image camera that captures moving images. However, as long as camera 41a can capture images of the open left door 3a and the user U1a who is getting off the vehicle located between the left door 3a and camera 41a, its specific specifications, position, and shooting range can be appropriately changed.

[0045] For example, instead of the center pillar 1c, the camera 41a can be placed in a peripheral component located around the center pillar 1c. The peripheral component is a structure such as the center pillar 1c or the side beam 1e. Alternatively, the camera 41a can be placed inside the vehicle, near the center pillar 1c, to capture images of the exterior from inside the vehicle. Furthermore, the number of cameras 41a is not limited to one; multiple cameras can be used.

[0046] [2. Structure of vehicle control device 10]

[0047] Figure 3 This is a block diagram showing the control system of vehicle 1. The control system of vehicle 1 includes a vehicle control device 10 and vehicle-mounted components such as cameras 41a and 41b installed on vehicle 1, which are connected to the vehicle control device 10. The vehicle control device 10 can also be described as an on-board device mounted on vehicle 1.

[0048] The vehicle control unit 10 includes a processor 11 and a memory 21. The processor 11 is a computer composed of a CPU (Central Processing Unit), an MCU (Micro Controller Unit), or an MPU (Micro Processor Unit).

[0049] Memory 21 is a rewritable, non-volatile storage device that stores the program 22 executed by processor 11 and the data processed by processor 11. Memory 21 may be constructed, for example, as a semiconductor storage device such as flash memory (ROM) or SSD (Solid State Disk), or a magnetic storage device. Memory 21 may also include RAM (Random Access Memory) forming a working area for temporarily storing the program 22 and data. In addition, vehicle control device 10 may sometimes be constructed as an integrated circuit (IC) that integrates processor 11 and memory 21.

[0050] The vehicle control unit 10 is connected to cameras 41a and 41b, sensor group 51, ignition switch 33, touch panel 34, voice input unit 52, voice output unit 53, and door lock mechanism 4a to 4d.

[0051] Under the control of the vehicle control device 10, cameras 41a and 41b acquire images composed of dynamic images obtained by photographing the outside of the vehicle, and output the image data of the captured images to the vehicle control device 10.

[0052] Sensor group 51 consists of multiple sensors that acquire information from various parts of vehicle 1 and output the acquired information to vehicle control device 10. Sensor group 51 includes a speed sensor 51a that detects the driving speed of vehicle 1, a selector sensor 51b that detects the operating position of gear selector 32, and door sensors 5a-5d that detect the opening, closing, locking / unlocking of each door 3a-3d.

[0053] Ignition switch 33 is a switch that allows user U1, as the driver, to turn on the ignition when the vehicle is in use and to turn off the ignition when the vehicle is no longer in use. Ignition switch 33 can also be called a start switch or power switch.

[0054] The touch panel 34 is equipped with a display panel for displaying various images and a touch sensor for detecting touch operations on the display panel, and outputs user instructions detected by touch operations of users such as U1 to the vehicle control unit 10.

[0055] The voice input unit 52 includes a microphone 35 and circuitry for amplifying and converting the voice input via the microphone 35 to digital, and outputs the voice data input via the microphone 35 to the vehicle control device 10. The voice input unit 52 is used, for example, to implement voice control, in which the user U1 gives various instructions via voice.

[0056] The voice output unit 53 includes a speaker 36 and a drive circuit for driving the speaker 36, and outputs various voices to the vehicle compartment S under the control of the vehicle control device 10.

[0057] The door lock mechanisms 4a to 4d are controlled by the vehicle control device 10 to lock / unlock each door 3a to 3d.

[0058] The processor 11 reads and executes the program 22 stored in the memory 21, and functions as the information acquisition unit 12, operation input unit 13, display control unit 14, vehicle control unit 15, person detection unit 16, and disembarkation detection unit 17.

[0059] The information acquisition unit 12 acquires images captured by cameras 41a and 41b and information detected by sensor group 51.

[0060] Furthermore, the information acquisition unit 12 may also have the function of acquiring various information from a server connected to the Internet via a communication unit (not shown). In this case, for example, program 22 can be updated by acquiring update data of program 22 from a server connected to the Internet.

[0061] The operation input unit 13 inputs the user U1's operations and user instructions via the ignition switch 33, the touch panel 34, and the voice input unit 52.

[0062] The display control unit 14 controls the image displayed on the display panel of the touch panel 34 based on user instructions input via the operation input unit 13.

[0063] The vehicle control unit 15 performs control of various parts of the vehicle 1. In this embodiment, the vehicle control unit 15 includes a door control unit 15a. The door control unit 15a switches the locking / unlocking of each door 3a to 3d by outputting signals that control the operation of the door lock mechanisms 4a to 4d.

[0064] The human detection unit 16 performs processing to detect people within the captured image based on the image captured by the camera 41a.

[0065] Figure 4 This is a diagram showing an example of an image captured by camera 41a.

[0066] Figure 4 The image P1 shown was taken immediately after the driver, i.e., user U1, opened the left door 3a and got out of the car, showing user U1a getting out and the left door 3a. On the other hand, Figure 4 The captured image P2 shows the situation where the user U1a who got off the vehicle is not present, and there is a passerby within the shooting range (equivalent to the range of the object area Ar1).

[0067] In the captured images P1 and P2, there is also a side beam 1e, one of the peripheral components of the door opening for closing the left door 3a, and the presence of other vehicles V parked around the vehicle 1.

[0068] Reference Figure 4 The explanation for the human testing department 16 is as follows.

[0069] The human detection unit 16 detects people in the captured image by performing object detection processing using known image recognition technology. More specifically, the human detection unit 16 detects the presence or absence of people based on the captured image composed of dynamic images by using image recognition technology that detects facial and body features and / or artificial intelligence technology that uses deep learning.

[0070] In addition, the person detection unit 16 generates a rectangular box W (see reference) in each frame of the captured image to surround the region of the detected person. Figure 4 The box W, also known as the bounding box, is a minimal rectangle that encloses the detected person and can be used to represent the location and size of the detected person.

[0071] Furthermore, in addition to human characteristics, the human detection unit 16 performs the following processing: it detects vehicle doors using image recognition technology that detects features of vehicle doors and / or artificial intelligence technology using deep learning. Thus, it is able to detect the presence or absence of the left door 3a within the captured image composed of dynamic images.

[0072] The following situations exist: the image captured by camera 41a shows the driver, i.e., user U1a, who has just gotten out of the car (image P1), and the pedestrian M1 (…). Figure 4 The image shows the situation of a user getting off the bus through the rear left door 3c, and the image shows the situation of a user getting off the bus through the rear left door 3c. Therefore, in the image P1, a bounding box W is generated that surrounds the user U1a who is getting off the bus, and in the image P2, a bounding box W is generated that surrounds the pedestrian M1.

[0073] like Figure 4 As shown, in the captured image P1, the alighting user U1a and the left door 3a are located in the lower right region, and the alighting user U1a overlaps with the left door 3a. Therefore, by performing the detection processing of the person and the door in the lower right region where the alighting user U1a and the left door 3a are envisioned, the person detection unit 16 can reduce the computational load required for detecting the alighting user U1a and the left door 3a. Furthermore, it can prevent the detection of a user alighting from the rear left door 3c.

[0074] As shown in image P2, a passerby M1 is sometimes reflected in the lower right area. Therefore, the person present in the lower right area of ​​the captured image is not necessarily the user U1a who got off the bus.

[0075] In this embodiment, as described later, a person overlapping with a door in the captured image is detected as a user getting off the vehicle, U1a, thus distinguishing between pedestrian M1 and user U1a getting off the vehicle. Furthermore, the person detection unit 16 can also perform detection processing for both people and vehicle doors across the entire captured image area.

[0076] exist Figure 4 The other vehicles V shown in the captured images P1 and P2 are not the targets of the human detection unit 16, so their presence will not affect the detection of the alighting user U1a. However, if it is necessary to detect other vehicles V, the human detection unit 16 can also detect them.

[0077] Figure 3 The vehicle disembarkation detection unit 17 shown performs the following processing: confirming the disembarkation intention of the driver (user U1) of vehicle 1. The vehicle disembarkation detection unit 17 includes a person tracking unit 17a, a first detection processing unit 17b, and a second detection processing unit 17c.

[0078] The human tracking unit 17a has the function of tracking (tracking) a person as the detection object based on the detection results of a person in consecutive frames of a captured image composed of dynamic images from the camera 41a. In the tracking process, a bounding box W surrounding the same person is generated for each frame.

[0079] The first detection processing unit 17b performs a first detection process, in which it detects whether the person detected by the person detection unit 16 is the alighting user U1a. As the first detection process, it uses known image recognition technology to detect whether the detected person is the alighting user U1a based on the captured image by detecting whether a first condition is met. The first condition is that the detected person is located at a position overlapping with the vehicle door.

[0080] like Figure 4 As shown, the frame W surrounding the detected alighting user U1 includes at least a portion of the lower left and right regions of the captured image P1, i.e., the designated region SR (in this embodiment, the lower right region SR). When this frame W overlaps with the left door 3a, it is determined that the first condition is met. Thus, it is possible to detect the alighting user U1a who has just gotten off the vehicle from the driver's seat 2a with high accuracy, and to prevent false detection of users getting off the vehicle from the rear seat and pedestrians M1.

[0081] Here, the defined area SR is set as the area that does not overlap with the frame W surrounding the detected pedestrian M1. In this structure, the defined area SR is set as the lower right region that has a margin mx between it and the width end of the left door 3a and a margin my between it and the upper end of the left door 3a.

[0082] The margin mx is set to a value that ensures that even when part of the pedestrian M1 extends out from the left door 3a, the pedestrian M1 (e.g., the frame W surrounding the pedestrian M1) does not overlap with the specified area SR, i.e., the pedestrian M1 will not be mistakenly identified as the alighting user U1a.

[0083] In addition, the margin my is set to a value that ensures that even if the frame W surrounding pedestrian M1 deviates, pedestrian M1 (e.g., the frame W surrounding pedestrian M1) does not overlap with the specified area SR, that is, it will not misidentify pedestrian M1 as a user getting off the bus U1a.

[0084] The second detection processing unit 17c performs a second detection processing based on the tracking results, in which it detects whether a second condition is met, namely, whether a person is leaving the vehicle 1.

[0085] Reference Figure 5 The second detection process will now be explained. In the second detection process, the alighting user U1a detected by the first detection processing unit 17b is tracked, and a process is performed to detect whether the following condition, which is an example of a second condition, is met: whether the frame W of the detection area of ​​the alighting user U1a shrinks over time. Figure 5 The symbol 't' in the text represents the time axis.

[0086] like Figure 5 As shown, box W is a minimum rectangular box that encloses the detected person, and therefore the size of box W shrinks as the disembarking user U1a moves away from vehicle 1. The shrinkage of box W can be detected based on at least one of the following: area, height, width, or diagonal.

[0087] In the second detection process, the reduction in size of the disembarking user U1a is detected as a reduction in the bounding box W. If the second condition is met, it is determined that the disembarking user U1a has left vehicle 1. Based on this structure, it is possible to detect with high accuracy that the disembarking user U1a has left vehicle 1 based on the captured image.

[0088] As described above, since camera 41a uses a wide-angle lens that captures a large area in the horizontal direction, in other words, by using a wide-angle lens type camera capable of capturing users getting out of the vehicle from both the front and rear seats, it is possible to capture the user U1a getting out of the vehicle with a wider field of view. As a result, the smaller the size of the user U1a is as they move further away from vehicle 1, the more emphasized this effect becomes, and the accuracy of detecting whether the user U1a is moving away from vehicle 1 is improved. Therefore, it is possible to detect the user U1a leaving vehicle 1 with high accuracy and in a short time.

[0089] In this embodiment, by Figure 2Within the target area Ar1 shown, the system detects when user U1a leaves vehicle 1, thereby detecting the departure of user U1a from vehicle 1 within a period of approximately 1.6m from vehicle 1.

[0090] In addition, such as Figure 4 As illustrated, the bounding box W surrounding the alighting user U1a in image P1 is nearly square, while the bounding box W surrounding the pedestrian M1 in image P2 is also elongated vertically because the pedestrian M1 is captured in a more elongated vertical direction. This difference in bounding box shape improves the detection accuracy of whether the pedestrian U1a is alighting user in the first detection process.

[0091] For example, as a specific first detection method, the following method is implemented: in addition to detecting whether at least a portion of the frame W surrounding the detected person is a specified area SR, the difference between the width and height of the frame W surrounding the detected person is within a specified value range that can be estimated to be a person getting off the vehicle. In other words, the size of the frame W surrounding the person getting off the vehicle is detected within a range that can be regarded as a square, thereby improving the detection accuracy of the user U1a getting off the vehicle.

[0092] Furthermore, as a specific second detection method, the following method is implemented: in addition to detecting whether at least a portion of the bounding box W surrounding the detected person is a specified area SR, it is also detected whether the width of the bounding box W surrounding the detected person is within a specified value range that can be estimated to be a person getting off the vehicle, and whether the height of the bounding box W surrounding the detected person is within a specified value range that can be estimated to be a person getting off the vehicle, thereby improving the detection accuracy of the user U1a getting off the vehicle.

[0093] Furthermore, these detection methods can be modified. For example, they are not limited to detecting whether the width of the bounding box W surrounding the detected person is within a specified range that allows estimation of whether the person is getting off the vehicle, nor are they limited to detecting whether the height of the bounding box W surrounding the detected person is within a specified range that allows estimation of whether the person is getting off the vehicle; either one can be omitted. Additionally, the detection process of detecting whether at least a portion of the bounding box W surrounding the detected person is a specified region SR can also be omitted.

[0094] In addition, Figure 5 In the example, for ease of explanation, we take the case where the entire body of user U1, who is disembarking user U1a, is included in the captured image to illustrate the detection of user U1a leaving vehicle 1. However, this is not a limitation; even if only the upper body of user U1 is captured in the image, the detection of user U1a leaving vehicle 1 can still be achieved by reducing the detection area for user U1 (e.g., reducing the width of the bounding box W).

[0095] [3. Operation of vehicle control device 10]

[0096] Next, according to Figure 6 The flowchart shown illustrates the actions of the vehicle control device 10 when executing the disembarkation sequence. Additionally, Figure 6 The flowchart shown illustrates the process performed when the user U1 or others have set the disembarkation schedule, or when a scheduled disembarkation schedule is being reserved. Furthermore, the setting and reservation methods for the disembarkation schedule can be implemented using appropriate methods, such as via the touch panel 34.

[0097] First, the vehicle control unit 10 begins the process of detecting a person using the person detection unit 16, based on the image captured by the camera 41a (step S1). If a person is detected (step S1: Yes), the vehicle control unit 10 checks whether the left door 3a has been detected by the person detection unit 16 and whether a first condition is met, which is that the person detected by the first detection processing unit 17b is present at a position overlapping with the detected left door 3a (step S2).

[0098] The processing in steps S1 and S2 is equivalent to object detection processing, which detects the object consisting of a person and the left door 3a. This object detection processing is used to detect whether a user U1a is alighting. The vehicle control device 10 monitors whether a user U1a is alighting by repeatedly executing the object detection processing at a predetermined cycle. The predetermined cycle only needs to be set to an appropriate time range that allows for the rapid and proper detection of the user U1a alighting.

[0099] If a user U1a is detected getting off the vehicle (step S2: Yes), the vehicle control device 10 starts tracking the detected user U1a by the human tracking unit 17a, generates a frame W surrounding the detected user U1a, and thereby monitors the detection area of ​​the user U1a (step S3).

[0100] The vehicle control unit 10 detects, via the second detection processing unit 17c, whether the condition that the frame W shrinks over time, as a second condition, is met (step S4). The processing of steps S3 and S4 is equivalent to the processing of detecting the alighting intention of the alighting user U1a.

[0101] When the second condition is met, that is, when the user U1a leaves the vehicle 1, the intention of the user U1a to get off is detected. This process is repeated at a predetermined cycle. When the intention to get off is detected more than k times (k is an integer greater than or equal to 2) (step S5: Yes), the vehicle control device 10 determines the intention to get off through the second detection processing unit 17c.

[0102] Upon confirming the intention to alight, the vehicle control device 10, as a means of vehicle control during alighting, locks each door 3a to 3d via the door control unit 15a (step S6). Thus, when the user U1a, as the driver, alights and leaves the vehicle 1, each door 3a to 3d can be automatically locked.

[0103] Furthermore, the vehicle control in step S6 is not limited to locking control of doors 3a to 3d; it can also include at least one of the following: locking control of doors 3a to 3d, controlling the closing of doors and windows, and controlling the closing of doors when the door closing function is available. That is, as the vehicle control in step S6, any vehicle control preferably implemented when getting off the vehicle can be applied.

[0104] In this way, vehicle control is initiated after multiple attempts to disembark have been detected, thus preventing situations where vehicle control is initiated when the driver immediately returns to the vehicle after disembarking. Furthermore, in this embodiment, it is possible to capture... Figure 2 The position of camera 41a is set according to the way people are within the target area Ar1, so multiple attempts to get off the vehicle can be easily detected until the user U1a exits the target area Ar1. Furthermore, the time interval between getting off the vehicle and the number of times the getting off intention is confirmed (k times) can be set appropriately.

[0105] like Figure 6 As shown, if a negative result is obtained in any of the above steps S1, S2, S4, and S5, the vehicle control device 10 proceeds to the processing of step S1. Therefore, in the case where a user other than the driver exits the vehicle, and via... Figure 2 When pedestrians M1 on routes RT1 to RT5 pass through the target area Ar1, it is possible to avoid situations where vehicle control is required when disembarking.

[0106] As explained above, the vehicle exit detection unit 17 of this embodiment performs a first detection process and a second detection process. In the first detection process, based on the image captured by the camera 41a that captures images of the left side of the vehicle 1, it detects whether a first condition is met—that a detected person is present around the left door 3a of the vehicle 1. In the second detection process, it detects whether a second condition is met—that a detected person has left the vehicle 1. Furthermore, if both the first and second conditions are met, the vehicle control unit 15 performs vehicle control during the exit process.

[0107] By detecting the driver's exit from vehicle 1 in this way, even in the presence of pedestrians, it is possible to accurately detect the driver's exit from vehicle 1. This allows for proper vehicle control during exit procedures, improving convenience.

[0108] Furthermore, the exit detection unit 17 tracks the detected person, and determines that the second condition is met if the detection area (frame W) of that person shrinks over time. Thus, it is possible to detect the driver exiting vehicle 1 based on the tracking of people present around the left door 3a and the shrinking of the detection area. Therefore, even in the presence of pedestrians, it is possible to more accurately detect the driver exiting vehicle 1.

[0109] Furthermore, the first condition includes the detection of a person overlapping with the left door 3a. Therefore, it is possible to detect a driver exiting the vehicle with high accuracy based on an image captured where the driver is overlapping with the left door 3a. Moreover, since an image captured from the inside of the open left door 3a is used, it is easier to distinguish between pedestrians and drivers, further improving the accuracy of driver detection.

[0110] Furthermore, the first condition also includes that at least a portion of the detection area of ​​the detected person is a predetermined region SR in the lower part of the captured image. Therefore, it is possible to detect a disembarking driver with high accuracy based on an image containing at least a portion of the disembarking driver within the predetermined region in the lower part of the captured image. Consequently, it is easier to distinguish between disembarking drivers and pedestrians, further improving detection accuracy.

[0111] In addition, the image captured is of the driver who got out of the vehicle in the lower designated area SR of the captured image, while the images of pedestrians around the vehicle 1 are captured outside the designated area SR, thus improving the detection accuracy of the driver who got out of the vehicle.

[0112] Furthermore, the exit detection unit 17 determines a person to be a pedestrian if the detected person does not overlap with the left door 3a in the captured image, and / or if the detection area (frame W) of the detected person is not present in the lower designated area SR of the captured image. This makes it easier to distinguish between drivers exiting the vehicle and pedestrians, thus improving detection accuracy.

[0113] Furthermore, the captured image is taken by a camera 41a installed at or around the center pillar 1c on the driver's seat 2a side. Within the captured image, if a person detected by the person detection unit 16 is detected at a position overlapping with the left door 3a, which is the front seat side door, the exit detection unit 17 determines that the person is exiting the vehicle. Therefore, it is possible to detect a person exiting the vehicle from the driver's seat 2a with high accuracy.

[0114] Furthermore, the first condition includes: the detected person overlaps with the left door 3a in the captured image, and at least a portion of the detection area of ​​the detected person is a predetermined region SR in the lower part of the captured image. The predetermined region SR is the area to the left or right of the lower part of the captured image (right in this embodiment). Therefore, the detection accuracy of people getting off the vehicle is improved.

[0115] Furthermore, the captured image is an image including the peripheral components of the door opening closed by the left door 3a, namely the central pillar 1c. The first condition includes either the detected person overlapping with the left door 3a in the captured image, or the detected person overlapping with the aforementioned peripheral components in the captured image. Thus, it is possible to capture images of people who have just alighted from the door opening, and to quickly detect people getting off the vehicle.

[0116] Furthermore, the peripheral components of the door opening shown in the captured image are not limited to the side beam 1e, but could also be, for example, the front pillar 1b and / or the middle pillar 1c.

[0117] Furthermore, the first condition includes the fact that the width of the detection region (box W) of the detected person is within a specified range that allows for the estimation of whether the person is getting off the vehicle. Therefore, based on the width of the box W in the captured image, it is possible to detect people getting off the vehicle with high accuracy.

[0118] Furthermore, the first condition includes the height of the bounding box W surrounding the detected person being within a specified range that allows for the estimation of whether the person is getting off the vehicle. Therefore, it is possible to detect people getting off the vehicle with high accuracy based on the height of the bounding box W in the captured image.

[0119] Furthermore, the first condition includes the fact that the difference between the width and height of the bounding box W of the detected person is within a specified range that allows for the estimation of whether the person is getting off the vehicle. Therefore, it is possible to detect people getting off the vehicle with high precision based on the difference between the width and height of the bounding box W in the captured image.

[0120] In addition, the designated area SR is set at a position that is separated from the width end and the upper end of the left door 3a of the vehicle 1. Therefore, the designated area SR can be set at a position that overlaps with the detection area of ​​people getting off the vehicle and is unlikely to overlap with the detection area of ​​pedestrians, which can reduce the situation of pedestrians being mistaken for passengers getting off the vehicle.

[0121] In addition, the vehicle control unit 15 includes a door control unit 15a for controlling the locking of doors 3a to 3d, including the left door 3a, as vehicle control during alighting. Thus, when a person is detected alighting, doors 3a to 3d can be automatically locked.

[0122] Furthermore, in the case of a right-hand drive vehicle, if the driver is detected getting out of the vehicle, or in the case of a left-hand drive vehicle, if a person is detected getting out of the passenger seat 2b, the same processing can be performed based on the image captured by the right camera 41b.

[0123] For example, the exit detection unit 17 performs a first detection process and a second detection process. In the first detection process, based on the image captured by the camera 41b on the right side of the vehicle 1, it detects whether a first condition is met—that a detected person exists around the right door 3b of the vehicle 1. In the second detection process, it tracks the detected person and detects whether a second condition is met—that the detection area (frame W) of the person shrinks over time. Then, if the first and second conditions are met, the vehicle control unit 15 performs vehicle control during exit. Thus, even in the presence of pedestrians, it is possible to properly detect whether the driver is exiting the vehicle 1 or whether a person is exiting the passenger seat 2b. Therefore, it is possible to properly control the vehicle during exit, improving convenience.

[0124] In this case, the first condition includes situations where the detected person overlaps with the right door 3b in the captured image and at least a portion of the detection area of ​​the detected person is a predetermined area in the lower part of the captured image, where the predetermined area SR is set to the area to the left or right of the lower part of the captured image. Therefore, it is possible to detect the driver getting out of the vehicle with high precision in the case of a right-hand drive vehicle. Furthermore, in the case of a left-hand drive vehicle, it is possible to detect a person getting out of the passenger seat 2b with high precision.

[0125] Figure 1 The vehicle 1 shown is an example of the vehicle to which this disclosure is made, and the left door 3a and / or the right door 3b are examples of the doors to which this disclosure is made.

[0126] Furthermore, the processing performed by the personnel detection unit 16 is an example of the personnel detection steps of this disclosure, the processing performed by the vehicle disembarkation detection unit 17 is an example of the vehicle disembarkation detection steps of this disclosure, and the processing performed by the vehicle control unit 15 is an example of the vehicle control steps of this disclosure.

[0127] [4. Other Implementation Methods]

[0128] The above-described embodiments are merely one embodiment of the present invention, and can be arbitrarily modified and applied without departing from the spirit of the present invention.

[0129] For example, in the above embodiment, the locking control of doors 3a to 3d is described as vehicle control when getting off the vehicle, but it is not limited to this. For example, it is also possible to perform the following in addition to the locking control of doors 3a to 3d: control of closing doors and windows of doors 3a to 3d, or door closing control of closing open doors of doors 3a to 3d when a door closing function is available.

[0130] In addition, as Figure 3The structures shown can be widely applied to software-implemented structures, hardware-implemented structures, and structures implemented by combining software and hardware. Furthermore, the method of dividing the processing units and the processing order in the flowchart are not limited to the examples shown and can be appropriately modified.

[0131] Furthermore, regarding the application of this invention... Figure 1 The vehicle system 100 and vehicle control method of the four-wheeled vehicle shown have been described, but the invention is not limited thereto and can also be applied to the vehicle systems and vehicle control methods of other vehicles.

[0132] Furthermore, while the case where the program 22 implementing the vehicle control method of the present invention is recorded on the vehicle control device 10 has been described, the program 22 can also be obtained from an external device via communication. Alternatively, the program 22 can be recorded on a recording medium that can be read by a computer. The recording medium can be a magnetic, optical, or semiconductor storage device. Specifically, examples include removable or fixed recording media such as floppy disks, various optical discs, optical disks, flash memory, and card-type recording media. Alternatively, the recording medium can be a non-volatile storage device such as RAM, ROM, or HDD, which is an internal storage device included in an image display device.

[0133] [5. Structures supported by the above embodiments]

[0134] The above implementation is a specific example of the following structure.

[0135] (Structure 1) A vehicle system comprising: a person detection unit that detects a person based on a captured image composed of dynamic images obtained by capturing images of the exterior of the vehicle; and an alighting detection unit that detects whether the detected person is a person alighting from a target vehicle, wherein the alighting detection unit performs a first detection process and a second detection process, wherein in the first detection process, it detects whether a first condition is met based on the captured image, indicating that the detected person is present around a target door of the target vehicle, and in the second detection process, it detects whether a second condition is met, indicating that the detected person is leaving the target vehicle, wherein if the first condition and the second condition are met, the alighting detection unit detects that the detected person is a person alighting from the target vehicle, and the vehicle system has a vehicle control unit that performs vehicle control during alighting when the alighting detection unit detects that the detected person is a person alighting from the target vehicle.

[0136] According to this structure, even in the presence of pedestrians, it can properly detect when people get off the vehicle, thereby enabling appropriate vehicle control and improving convenience.

[0137] (Structure 2) According to the vehicle system of Structure 1, the disembarkation detection unit tracks the detected person, and determines that the second condition is met when the detection area of ​​the person shrinks over time.

[0138] Based on this structure, it is possible to detect when a person gets off a vehicle, provided that the tracking of people present around the object's door is achieved and the detection area is narrowed. Therefore, even in the presence of passersby, it is possible to more accurately detect when a person gets off a vehicle.

[0139] (Structure 3) The vehicle system according to claim 1, wherein the first condition includes the detection that the person overlaps with the door in the captured image.

[0140] Based on this structure, it is possible to detect the driver getting off the vehicle with high precision by capturing an image of the passenger getting off the vehicle with the door overlapping.

[0141] (Structure 4) The vehicle system according to Structure 1 or 2, wherein the first condition includes the case that at least a portion of the detected area of ​​the person is a predetermined area in the lower part of the captured image.

[0142] According to this structure, passengers disembarking can be detected with higher precision based on images captured in a predetermined area at the bottom of the captured image, where at least a portion of the disembarking passenger is reflected.

[0143] (Structure 5) According to the vehicle system of Structure 4, wherein the captured image is an image of a person getting off the vehicle being captured within the designated area of ​​the captured image, and an image of a passerby located around the vehicle being captured outside the designated area.

[0144] Based on this structure, the detection accuracy of disembarking passengers is improved.

[0145] (Structure 6) According to the vehicle system of Structure 3, the disembarkation detection unit determines that the person is a pedestrian if the detected person does not overlap with the door in the captured image and / or if there is no detection area for the detected person in the lower part of the captured image.

[0146] Based on this structure, it is easy to distinguish between passengers getting off the train and passersby, thus improving detection accuracy.

[0147] (Structure 7) A vehicle system according to any one of Structures 1 to 6, wherein the captured image is an image captured by an external camera installed at or around the center pillar located at the front-to-rear center of the target vehicle, and the exit detection unit determines that the person exiting the vehicle is the person who is detected by the person detection unit at a position overlapping with the door on the front seat side in the captured image.

[0148] Based on this structure, it is possible to detect with high precision when a person gets out of the front seat.

[0149] (Structure 8) According to the vehicle system of Structure 7, wherein the first condition includes the case that the detected person overlaps with the door in the captured image and at least a portion of the detection area of ​​the detected person is a predetermined area in the lower part of the captured image, the predetermined area being the area to the left or right of the lower part of the captured image.

[0150] Based on this structure, it is possible to detect with high precision when a person gets out of the front seat.

[0151] (Structure 9) A vehicle system according to any one of Structures 1 to 8, wherein the captured image is an image containing the peripheral components of a door opening closed by the door, and the first condition includes a situation that satisfies any one of the following: the detected person overlaps with the door in the captured image, and the detected person overlaps with the peripheral components in the captured image.

[0152] Based on this structure, it is possible to photograph people who have just gotten off the vehicle through the door opening, and to quickly detect the people getting off the vehicle.

[0153] (Structure 10) A vehicle system according to any one of Structures 1 to 9, wherein the first condition includes the width of the detection area of ​​the detected person being within a predetermined value range that allows estimation of whether the person is getting off the vehicle.

[0154] Based on this structure, it is possible to detect people getting off the vehicle with high precision based on the width of the detection area in the captured image.

[0155] (Structure 11) A vehicle system according to any one of Structures 1 to 10, wherein the first condition includes the height of the detection area of ​​the detected person being within a predetermined value range that allows estimation of whether the person is getting off the vehicle.

[0156] Based on this structure, it is possible to detect people getting off the vehicle with high precision based on the height of the detection area in the captured image.

[0157] (Structure 12) A vehicle system according to any one of Structures 1 to 11, wherein the first condition includes the difference between the width and height of the detection area of ​​the detected person being within a specified value range that can be used to estimate that the person is getting off the vehicle.

[0158] Based on this structure, it is possible to detect people getting off the vehicle with high precision by using the difference between the width and height of the detection area in the captured image.

[0159] (Structure 13) According to the vehicle system of Structure 4 or 5, the designated area is set at a position separated from the width direction end and upper end of the object door of the object vehicle.

[0160] According to this structure, a designated area can be set at a position that overlaps with the detection area of ​​people getting off the vehicle but is less likely to overlap with the detection area of ​​passersby, thereby reducing the situation of mistakenly identifying passersby as passengers getting off the vehicle.

[0161] (Structure 14) A vehicle system according to any one of Structures 1 to 13, wherein the vehicle control unit includes a door control unit that performs locking control of doors, including the object door, as vehicle control during alighting.

[0162] According to this structure, the door can be automatically locked when a person getting off the vehicle is detected.

[0163] (Structure 15) A vehicle control method executed by a computer, wherein the vehicle control method includes: a person detection step, wherein a person is detected by a person detection unit based on a captured image composed of dynamic images obtained by capturing images of the exterior of a vehicle; an alighting detection step, wherein the alighting detection unit detects whether the detected person is a person alighting from a target vehicle; and a vehicle control step, wherein if the detected person is a person alighting from the target vehicle, the vehicle control unit performs vehicle control during alighting, wherein in the alighting detection step, a first detection process and a second detection process are performed, wherein in the first detection process, a first condition is met based on the captured image to determine whether the detected person is present around the target door of the target vehicle, and in the second detection process, a second condition is met to determine whether the detected person is leaving the target vehicle, and if the first condition and the second condition are met, the detected person is a person alighting from the target vehicle.

[0164] According to this structure, even in the presence of pedestrians, it can properly detect when people get off the vehicle, enabling appropriate vehicle control and improving convenience.

Claims

1. A vehicle system, wherein, The vehicle system includes: A human detection unit that detects people based on images composed of dynamic images obtained from outside the vehicle; and The vehicle disembarkation detection unit checks whether the person detected has disembarked from the target vehicle. in, The vehicle alighting detection unit performs a first detection process and a second detection process. In the first detection process, based on the captured image, it detects whether a first condition is met: the detected person is present near the door of the target vehicle. In the second detection process, it detects whether a second condition is met: the detected person is leaving the target vehicle. If both the first and second conditions are met, the vehicle alighting detection unit determines that the detected person is a person alighting from the target vehicle. The vehicle system has a vehicle control unit that performs vehicle control when the disembarkation detection unit detects that the person being disembarked is a person getting off the target vehicle.

2. The vehicle system according to claim 1, wherein, The disembarkation detection unit tracks the detected person, and determines that the second condition is met when the detection area of ​​the person shrinks over time.

3. The vehicle system according to claim 1, wherein, The first condition includes the detected situation where the person overlaps with the door in the captured image.

4. The vehicle system according to claim 1, wherein, The first condition includes cases where at least a portion of the detected area of ​​the person is a defined area in the lower part of the captured image.

5. The vehicle system according to claim 4, wherein, The captured images are images of people getting off the vehicle within the designated area, while images of passersby around the vehicle are captured outside the designated area.

6. The vehicle system according to claim 3, wherein, If the detected person does not overlap with the door in the captured image and / or if there is no detection area for the detected person in the lower part of the captured image, the vehicle disembarkation detection unit determines that the person is a pedestrian.

7. The vehicle system according to claim 1, wherein, The captured images are taken by an external camera installed at or around the center pillar at the front or rear center of the vehicle. If the person detected by the person detection unit is located in the captured image at a position overlapping with the door on the front seat side, the exit detection unit determines that the person is the one exiting the vehicle.

8. The vehicle system according to claim 7, wherein, The first condition includes cases where the detected person overlaps with the door within the captured image, and at least a portion of the detection area of ​​the detected person is a predetermined area in the lower part of the captured image. The designated area is the area on the lower left and right sides of the captured image.

9. The vehicle system according to claim 1, wherein, The captured image is an image that includes the surrounding components of the door opening that has been closed by the door. The first condition includes a situation where either the detected person overlaps with the door in the captured image, or the detected person overlaps with the surrounding components in the captured image.

10. The vehicle system according to claim 1, wherein, The first condition includes the width of the detection area of ​​the detected person being within a specified range that allows for the estimation that the person is getting off the vehicle.

11. The vehicle system according to claim 1, wherein, The first condition includes the height of the detection area of ​​the detected person being within a specified range that allows for the estimation that the person is getting off the vehicle.

12. The vehicle system according to claim 1, wherein, The first condition includes the difference between the width and height of the detected area of ​​the person being identified being within a specified range that allows for the estimation that the person is getting off the vehicle.

13. The vehicle system according to claim 4, wherein, The designated area is located at a position separate from the width end and upper end of the object door of the object vehicle.

14. The vehicle system according to claim 1, wherein, The vehicle control unit includes a door control unit that performs locking control of doors, including the target door, as vehicle control during disembarkation.

15. A vehicle control method, executed by a computer, wherein, The vehicle control method includes: The human detection step involves a human detection unit that detects people based on images composed of dynamic images obtained from the outside of the vehicle. The disembarkation detection step involves using a disembarkation detection unit to determine whether the detected person is a person disembarking from the target vehicle; and In the vehicle control step, if the detected person is someone alighting from the target vehicle, the vehicle control unit performs vehicle control during the alighting process. In the vehicle disembarkation detection step, a first detection process and a second detection process are performed. In the first detection process, based on the captured image, it is detected whether a first condition is met that the detected person is present around the object door of the object vehicle. In the second detection process, it is detected whether a second condition is met that the detected person is leaving the object vehicle. If both the first and second conditions are met, the detected person is identified as someone who is disembarking from the object vehicle.

Images