Abandonment detection device and abandonment detection method

Abstract

To provide an abandoned vehicle detection device and an abandoned vehicle detection method capable of detecting the presence or absence of a passenger left in a vehicle while reducing the processing load of a captured image with a simple device configuration. [Solution] The system includes multiple sets of camera units 50A-50C, each including one camera 11A-11C that captures images of the interior of a vehicle 1A, and multiple mirrors 12A-12C that are each positioned within the capture area of ​​these cameras 11A-11C and that capture the blind spot ranges of the cameras 11A-11C within the interior of the vehicle; an analysis unit 25A that corresponds to the area of ​​each mirror 12A-12C in the images captured by all cameras 11A-11C in all camera units 50A-50C and that sequentially analyzes the images reflected in the mirrors when it is determined that no passengers are captured in the images captured by all cameras 11A-11C; and a second judgment unit 27 that determines whether or not a passenger is captured in the analyzed images.

Description

[Technical Field]

[0001] The present invention relates to an abandonment detection device and an abandonment detection method. [Background technology]

[0002] In recent years, the situation where a passenger (a person other than the driver) is left unattended in a vehicle, i.e., passenger abandonment in a vehicle, has become a problem. This type of abandonment is often caused by the driver getting out of the vehicle without realizing that a passenger is still in the vehicle, or by the driver being aware that a passenger is still in the vehicle but taking some time for some reason to get back in after getting out of the vehicle, raising concerns about the health of the passenger who is left behind. For this reason, a technology has been known that detects passenger abandonment based on video captured by a camera mounted in the vehicle (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-57088 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional configurations, for example, passengers in the front seats of a vehicle are captured in the camera's video, but passengers in the rear seats may not be captured in the video if they are in the camera's blind spot hidden by the front seats. While it is conceivable to install multiple cameras in a vehicle to reduce the blind spots of each camera, this type of configuration complicates the device configuration and increases the processing load of the captured video, leaving room for improvement in this regard.

[0005] The present invention has been made in consideration of the above, and aims to provide an abandonment detection device and abandonment detection method that can detect whether or not a passenger is left behind in the interior of a vehicle while reducing the processing load of captured images with a simple device configuration. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems and achieve the object, the abandoned object detection device of the present invention comprises multiple sets of camera units including one camera that photographs the interior of the vehicle and multiple mirrors that are positioned within the camera's shooting area and that reflect the camera's blind spot within the interior of the vehicle; a vehicle disembarkation detection unit that detects the driver disembarking; a first judgment unit that determines whether or not any vehicle occupants are visible in the images captured by all cameras in all camera units after the driver disembarking is detected; an analysis unit that corresponds to the area of ​​each mirror in the images captured by all cameras and sequentially analyzes the images reflected in the mirrors when it is determined that no occupants are visible in the images captured by all cameras in all camera units; and a second judgment unit that determines whether or not any occupants are visible in the analyzed images.

[0007] In addition, the abandonment detection method of the present invention is a method for detecting whether an occupant has been left behind in the vehicle using multiple sets of camera units including one camera that photographs the interior of the vehicle and multiple mirrors that are positioned within the camera's shooting area and that reflect the camera's blind spot within the vehicle, and has a computer execute the following steps: detecting whether the driver has gotten out of the vehicle; determining whether any occupants of the vehicle are visible in the images captured by all cameras in all camera units after the driver has gotten out; if it is determined that no occupants are visible in the images captured by all cameras in all camera units, sequentially analyzing the images reflected in the mirrors in the areas corresponding to each mirror in the images captured by all cameras; and determining whether any occupants are visible in the analyzed images. [Effects of the Invention]

[0008] According to the present invention, it is possible to detect whether or not a passenger remains in the vehicle cabin while reducing the processing load of captured images with a simple device configuration. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating an example of the configuration of an abandonment detection device according to the first embodiment. [Figure 2] FIG. 2 is a schematic diagram showing an example of the configuration of a vehicle equipped with an abandoned vehicle detection device. [Figure 3] FIG. 3 is a schematic diagram showing an example of an image of the interior of a vehicle captured by the camera of the abandoned vehicle detection device. [Figure 4] FIG. 4 is a flowchart showing the procedure of the operation of the abandonment detection device according to the first embodiment. [Figure 5] FIG. 5 is a diagram showing an example of a corrected image in which distortion of the image reflected in the mirror has been corrected. [Figure 6] FIG. 6 is a block diagram illustrating an example of the configuration of an abandonment detection device according to the second embodiment. [Figure 7] FIG. 7 is a schematic diagram showing a configuration example of a vehicle equipped with an abandoned vehicle detection device according to the second embodiment. [Figure 8] FIG. 8 is a flowchart showing the procedure of the operation of the abandonment detection device according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the following embodiments.

[0011] [First embodiment] FIG. 1 is a block diagram showing an example of the configuration of an abandonment detection device according to a first embodiment. FIG. 2 is a schematic diagram showing an example of the configuration of a vehicle equipped with an abandonment detection device. FIG. 3 is a schematic diagram showing an example of an image of the interior of the vehicle captured by a camera of the abandonment detection device. The abandonment detection device 10 is mounted on the vehicle and detects whether or not a passenger is present in the vehicle interior (whether or not the passenger has been abandoned) after the driver leaves the vehicle (driver's seat). Furthermore, when the abandonment detection device detects the presence (abandonment) of a passenger (occupant), it notifies the driver or a predetermined contact (police station or fire station) to that effect.

[0012] In this embodiment, as an example of the passenger abandonment detection device detecting passenger abandonment, a case where the driver leaves the vehicle without noticing that there is a passenger inside the vehicle will be described. However, the present invention may also be applied to a case where the driver is aware that there is a passenger inside the vehicle but temporarily leaves the vehicle and does not return for a predetermined period of time. Furthermore, the passenger may be, for example, a small child, but may also be an infant, an elderly person, a person requiring care, or even an animal such as a dog or a cat. Furthermore, the vehicle may be, for example, a regular car, but may also include a public bus or a shuttle bus.

[0013] As shown in FIG. 1, the abandoned object detection device 10 includes a camera 11, a mirror 12, a communication unit 14, and a control unit (abandonment detection control device) 20.

[0014] Camera 11 is a camera that photographs the interior of the vehicle. Camera 11 is mounted on the vehicle and photographs the interior of the vehicle, for example, from the front to the rear in the direction of travel of the vehicle. Camera 11 is disposed in a position that allows it to photograph at least the faces of passengers, including the driver, seated in all seats 2 (front seat 2A and rear seat 2B) of vehicle 1. In this embodiment, camera 11 is disposed above the vehicle windshield 3 or around the rearview mirror (not shown) as shown in FIG. 2, and photographs the interior of the vehicle. Camera 11 has a fixed or nearly fixed photographing range and photographing direction.

[0015] Camera 11 is a camera capable of capturing near-infrared light, for example, and is a camera that detects and visualizes electromagnetic waves in the infrared wavelength range, primarily targeting the near-infrared range (wavelengths of 0.8 μm to 1.5 μm). Camera 11 can be used as a nighttime surveillance camera that captures images of the interior of vehicle 1 even at night when there is little visible light received, by using, for example, infrared LED lighting as a light source. Camera 11 is configured to be activated even when vehicle 1 (engine) is stopped, and is activated, for example, when the driver gets off the vehicle, to capture images of the interior of vehicle 1. Camera 11 may also be configured to capture images of the interior of vehicle 1 at all times, or for a predetermined period after the engine is stopped. In this case, it is possible to detect from the image that the driver has exited the vehicle. Camera 11 outputs the captured image to image acquisition unit 22 (FIG. 1) of control unit 20.

[0016] As shown in Figures 2 and 3, mirror 12 is disposed within the imaging area (field of view) of camera 11 and projects a virtual image of an object in a blind spot 6 (Figure 2) of camera 11 within the interior of vehicle 1. Specifically, as shown in Figure 2, mirror 12 is fixed to the upper part of rear window 4 or ceiling 5 of vehicle 1 and disposed within the imaging area of ​​camera 11. Mirror 12 is attached so that its position can be adjusted, for example, via a drive unit (not shown). Therefore, by operating this drive unit, the relative position of mirror 12 with respect to camera 11 can be corrected (corrected).

[0017] As described above, camera 11 is capable of capturing images of at least the faces of passengers (including the driver) seated in all seats 2 of vehicle 1. In this case, for example, the space between front seat 2A and rear seat 2B is hidden by the backrest of front seat 2A, forming blind spot 6 for camera 11. Mirror 12 is disposed behind the backrest of front seat 2A, i.e., facing blind spot 6 for camera 11, and can project a virtual image of an object in blind spot 6 within the imaging area of ​​camera 11.

[0018] Furthermore, mirror 12 is a curved mirror that is curved in both the width and height directions, with the center portion protruding forward (toward camera 11) beyond the edges. By using this type of curved mirror, a virtual image of a wide range in the width and height directions of the interior of vehicle 1 can be reflected on mirror 12. Meanwhile, since mirror 12 is disposed within the shooting area of ​​camera 11 as described above, the captured image includes a real image of the interior of the vehicle and a virtual image reflected on mirror 12. For this reason, in this embodiment, as shown in FIG. 3, mirror 12 is configured to include a plurality of (four in FIG. 3) reflecting portions 13 that indicate the outline (outer edge) of mirror 12 and reflect light (e.g., infrared light) emitted from camera 11 back to camera 11.

[0019] Each of the multiple reflecting portions 13 is formed, for example, by a plane mirror, and is disposed on the edge of the mirror 12. Therefore, the outline of the mirror 12 is formed by connecting adjacent reflecting portions 13. Furthermore, each of the multiple reflecting portions 13 is provided so that its angle with respect to the mirror 12 can be adjusted. According to this configuration, after the mirror 12 is attached to the vehicle and the relative position of the mirror 12 with respect to the camera 11 is adjusted, the position of each reflecting portion 13 is adjusted, so that each reflecting portion 13 reflects light emitted from the camera 11 back to the camera 11. Therefore, the area of ​​the mirror 12 can be defined based on the position of the reflecting portion 13. The position of the reflecting portion 13 can be adjusted manually by the driver or passenger, or may be adjusted using the drive unit for adjusting the position of the mirror 12 described above.

[0020] In this embodiment, as shown in FIG. 3 , the mirror 12 is configured to include a QR code (registered trademark) 13A (information code portion) superimposed on at least one of the multiple reflecting portions 13. This QR code 13A has mirror information including at least curvature information of the mirror 12, and this mirror information may further include information regarding the size of the mirror 12 and the number of reflecting portions 13. With this configuration, the QR code 13A superimposed on the reflecting portion 13 can be acquired through the camera 11, making it possible to easily acquire information such as curvature information of the mirror 12. In this embodiment, the QR code 13A is superimposed on at least one reflecting portion 13. However, for example, the QR code 13A may be provided in place of at least one of the multiple reflecting portions 13.

[0021] The communication unit 14 communicates with devices outside the vehicle via the Internet, a mobile phone line, or the like. For example, when the communication unit 14 detects that a passenger has been left inside the vehicle, it sends a notification to the driver's pre-registered mobile phone. Furthermore, for example, when the communication unit 14 detects that a passenger has been left inside the vehicle and the detection of the passenger not being left remains even after a predetermined time has elapsed (the driver has not returned to the vehicle), it makes an emergency call to the police or fire department.

[0022] The control unit 20 is an arithmetic processing device configured with, for example, a CPU (Central Processing Unit). As shown in FIG. 1, the control unit 20 includes a dismount detection unit 21, an image acquisition unit 22, a mirror information acquisition unit 23, a mirror position correction unit (position correction unit) 24, an analysis unit 25, a first determination unit 26, a second determination unit 27, a warning unit 28, and a communication control unit 29, all of which are connected via a bus BS. The control unit 20 implements these components and executes these processes by reading and executing programs stored in a storage unit (not shown). The control unit 20 may execute these processes using one CPU, or may include multiple CPUs that execute these processes in parallel.

[0023] The alighting detection unit 21 detects that the driver has alighted and left the vehicle. The alighting detection unit 21 detects the driver alighting, for example, based on image analysis of the above-mentioned camera 11 that captures the interior of the vehicle 1, or based on the detection results of a pressure sensor (not shown) that is provided in the driver's seat and determines whether the driver is seated. The alighting detection unit 21 may also detect the driver alighting when the driver's door is opened and then closed again while the engine (drive unit) is stopped, or may detect the driver alighting by using a weight sensor (not shown) that monitors the weight of the vehicle to detect a reduction in the vehicle weight as the driver's door is opened and closed.

[0024] The image acquisition unit 22 acquires captured image information of the interior of the vehicle. The image acquisition unit 22 acquires the captured image information output by the camera 11 all the time, or for a predetermined period after the engine is stopped, or after it is detected that the driver has exited the vehicle.

[0025] The mirror information acquisition unit 23 acquires mirror information corresponding to the QR code 13A photographed by the camera 11. The mirror information acquisition unit 23 acquires at least curvature information of the mirror 12 as the mirror information.

[0026] The mirror position correction unit 24 corrects the relative position of the mirror 12 with respect to the camera 11 based on the image captured by the camera 11. Specifically, the mirror position correction unit 24 corrects the relative position of the mirror 12 with respect to the camera 11 based on the position of the reflecting portion 13 provided on the mirror 12 so that at least the entire mirror 12 is included in the captured image. In this case, for example, the mounting positions of the mirror 12 or the camera 11 may be automatically corrected, or a message may be issued to prompt the driver or passenger to correct these mounting positions. Furthermore, in this embodiment, the mirror position correction unit 24 is configured to correct the position of the mirror 12 with respect to the camera 11 as a position correction unit, but it may also be configured to correct the position of the camera 11 with respect to the mirror 12, for example.

[0027] The analysis unit 25 analyzes the image reflected on the mirror 12, which corresponds to the region of the mirror 12 in the acquired captured image information. As described above, the mirror 12 is formed by bending with a predetermined curvature. Therefore, the image (virtual image) of the object reflected on the mirror 12 is distorted with respect to the real image of the object due to the curvature of the mirror 12. Therefore, the analysis unit 25 extracts partial image information corresponding to the region of the mirror 12 in the acquired captured image information, corrects the distortion of this partial image information, and generates corrected image information.

[0028] Specifically, the analysis unit 25 identifies the outline of the mirror 12 based on the positions of the multiple reflectors 13 provided on the mirror 12, and extracts partial image information from the acquired captured image information that corresponds to the area of ​​the mirror 12. Furthermore, the analysis unit 25 corrects distortion of the partial image information based on curvature information included in the acquired mirror information, to generate corrected image information.

[0029] After detecting that the driver has exited the vehicle, the first determination unit 26 determines whether or not a passenger in the vehicle 1 is captured in the image captured by the camera 11. The first determination unit 26 performs face recognition processing on the image captured by the camera 11 (captured image information) and determines whether or not the face of the passenger in the vehicle 1 is detected. Here, "the passenger is not captured" includes a case where the passenger is captured in the mirror 12 but the face cannot be detected. When the first determination unit 26 detects the passenger's face, it is determined that the passenger has been left behind in the vehicle 1. A known method can be used to detect a person's face from the captured image, and there is no limitation.

[0030] The second determination unit 27 determines whether or not a passenger is reflected in the analyzed image of the mirror 12. The second determination unit 27 performs a physique recognition process on the image (corrected image information) corresponding to the analyzed area of ​​the mirror 12. Because the mirror 12 reflects the blind spot range within the vehicle 1, the passenger's face is usually not frequently reflected in the mirror 12. Therefore, the second determination unit 27 determines whether or not characteristic points of the passenger's physique (for example, the shape of the back of the head, back, or waist) are detected. If the second determination unit 27 detects characteristic points of the passenger's physique, it is determined that the passenger has been left behind in the vehicle 1. Well-known methods can also be used as a method for detecting this type of physique.

[0031] As described above, in this embodiment, the control unit 20 performs a two-stage determination process using the first determination unit 26 and the second determination unit 27. Therefore, if the first determination unit 26 determines that a passenger is captured in the video, the analysis unit 25 does not analyze the video, and the second determination unit 27 does not make a determination. This reduces the processing load on the captured video, while achieving highly accurate abandoned vehicle detection. Note that, although the analysis unit 25 and the second determination unit 27 are configured separately in this embodiment, the analysis unit 25 and the second determination unit 27 may be integrated using a generation AI (Artificial Intelligence) that has machine-learned these physique information as event data.

[0032] When it is determined that a passenger has been left behind inside the vehicle 1, the warning unit 28 outputs a warning to the driver of the vehicle 1. More specifically, when the first determination unit 26 or the second determination unit 27 determines that a passenger is captured in the target video, the warning unit 28 outputs a warning to the driver of the vehicle. The warning may be an audio output from a speaker (not shown) of the vehicle.

[0033] The communication control unit 29 controls the operation of the communication unit 14. Specifically, when the warning unit 28 outputs a warning to the driver of the vehicle, the communication control unit 29 controls the operation of the communication unit 14 to notify the driver of the vehicle 1 of the fact that the vehicle has been abandoned via his / her mobile phone. This notification may be, for example, a text notification via email, but is preferably an audio notification via telephone. This allows the driver of the vehicle 1 to be notified of the fact that the vehicle has been abandoned via his / her mobile phone. Furthermore, if the driver has not returned to the vehicle 1 (for example, the vehicle door has not been opened) within a predetermined time (for example, five minutes) after notifying the driver's mobile phone, the communication control unit 29 controls the operation of the communication unit 14 to make an emergency call to the police or fire department. This prevents harm to the health of passengers and ensures their safety. The communication control unit 29 may simultaneously notify the driver's mobile phone and make an emergency call to the police or fire department.

[0034] Next, the operation of the abandonment detection device 10 according to the first embodiment will be described. Fig. 4 is a flowchart showing the procedure of the operation of the abandonment detection device according to the first embodiment. Fig. 5 is a diagram showing an example of a corrected image in which distortion of the image reflected in the mirror has been corrected.

[0035] The control unit 20 acquires captured image information of the interior of the vehicle 1 (step S1). The control unit 20 acquires the captured image information output by the camera 11 via the image acquisition unit 22.

[0036] The control unit 20 detects that the driver has dismounted (step S2). In this embodiment, the control unit 20 detects that the driver has dismounted by the dismounting detection unit 21 based on the detection result of at least the face of the driver through video analysis of the captured video information acquired by the video acquisition unit 22.

[0037] Next, the control unit 20 determines whether or not a passenger is captured in the captured image of the interior of the vehicle 1 (step S3). More specifically, the control unit 20 causes the first determination unit 26 to execute face recognition processing on the acquired captured image information, and determines whether or not the face of a passenger of the vehicle 1 is detected, that is, whether or not the passenger is captured in the captured image. In this determination, if it is determined that a passenger is captured in the captured image (step S3; Yes), the control unit 20 proceeds to step S7.

[0038] On the other hand, if it is determined that the passenger is not captured in the captured image (step S3; No), the control unit 20 acquires mirror information (step S4). The control unit 20 acquires mirror information corresponding to the QR code 13A from the captured image information of the interior of the vehicle 1 using the mirror information acquisition unit 23. This mirror information includes at least curvature information of the mirror 12.

[0039] Next, the control unit 20 analyzes the image reflected in the mirror area in the captured image (step S5). More specifically, the control unit 20 analyzes the image reflected on the mirror 12, which corresponds to the area of ​​the mirror 12 in the acquired captured image information, using the analysis unit 25. In this embodiment, the mirror 12 is bent with a predetermined curvature. Therefore, the image (virtual image) of the object reflected on the mirror 12 is distorted relative to the real image of the object due to the curvature of the mirror 12. The control unit 20 identifies the outline of the mirror 12 based on the positions of the multiple reflectors 13 provided on the mirror 12, and extracts partial image information 30 corresponding to the area of ​​the mirror 12 in the acquired captured image information, as shown in FIG. 5. The control unit 20 also generates corrected image information 31 by correcting the distortion of the partial image information 30 based on the curvature information included in the acquired mirror information, using the analysis unit 25.

[0040] Next, the control unit 20 determines whether a passenger is reflected in the image corresponding to the area of ​​the mirror 12 (step S6). More specifically, the control unit 20 determines whether a passenger is reflected in the analyzed image of the mirror 12 by the second determination unit 27. The second determination unit 27 performs a physique recognition process on the image (corrected image information) corresponding to the analyzed area of ​​the mirror 12. Because the mirror 12 reflects blind spots within the vehicle 1, the passenger's face is usually not frequently reflected in the mirror 12. For this reason, the second determination unit 27 detects feature points of the passenger's physique (for example, the shape of the back of the head, back, or waist), that is, determines whether the passenger is reflected in the corrected image. Note that the second determination unit 27 may perform a face recognition process in addition to the physique recognition process. In this determination, if the passenger is not reflected in the image corresponding to the area of ​​the mirror 12 (step S6; No), the passenger left behind in the vehicle 1 has not been found, and the process ends. On the other hand, if a passenger is reflected in the image corresponding to the area of ​​the mirror 12 (step S6; Yes), the control unit 20 shifts the process to step S7.

[0041] If a passenger is visible in the image in steps S3 and S6, the passenger has been left behind in the vehicle 1, and the control unit 20 outputs a warning to the driver's mobile phone (step S7). Specifically, the control unit 20 causes the warning unit 28 to output a warning to the driver of the vehicle 1 that a passenger has been left behind in the vehicle 1. Then, the control unit 20 controls the operation of the communication unit 14 by the communication control unit 29 to notify the driver of the vehicle 1 that the passenger has been left behind to the mobile phone of the driver.

[0042] After notifying the driver's mobile phone, the control unit 20 determines whether a predetermined time has elapsed (step S8). This predetermined time can be set appropriately (for example, 5 minutes) as a grace period until the driver returns to the vehicle. If the predetermined time has not elapsed (step S8; No), the control unit 20 repeatedly executes the process of step S8 until the predetermined time has elapsed. On the other hand, if the predetermined time has elapsed (step S8; Yes), the control unit 20 determines whether the driver has returned to the vehicle (step S9). The determination that the driver has returned to the vehicle 1 can be made, for example, by detecting the opening of the door of the vehicle 1.

[0043] If the driver has returned to the vehicle (step S9; Yes), the control unit 20 ends the process. On the other hand, if the driver has not returned to the vehicle (step S9; No), the control unit 20 controls the operation of the communication unit 14 to make an emergency call to the police station or fire station (step S10). More specifically, the control unit 20 controls the operation of the communication unit 14 by the communication control unit 29 to make an emergency call to the police station or fire station, and ends the process.

[0044] The abandoned object detection device 10 according to this embodiment includes a camera 11 that captures images of the interior of the vehicle 1, a mirror 12 that is positioned within the image capture area of ​​the camera 11 and that captures the blind spot 6 of the camera 11 within the interior, a vehicle alighting detection unit 21 that detects when the driver alights, a first determination unit 26 that determines whether a passenger in the vehicle 1 is captured in the image captured by the camera 11 after the driver alights, an analysis unit 25 that analyzes the image captured in the mirror 12 in the captured image when it is determined that the passenger is not captured in the captured image, and a second determination unit 27 that determines whether the passenger is captured in the analyzed image. With this configuration, for example, the interior of the vehicle 1 can be captured including the blind spot 6 using one camera 11 and one mirror 12, thereby achieving a simple device configuration. Furthermore, according to this configuration, a two-stage judgment process is performed by the first judgment unit 26 and the second judgment unit 27, so that the processing load of the captured image can be reduced while accurately detecting whether or not there is a passenger left in the vehicle.

[0045] Furthermore, in the abandoned object detection device 10 of this embodiment, the mirror 12 has a plurality of reflecting portions 13 that indicate the outer edge of the mirror 12 and reflect light emitted from the camera 11 toward the camera 11, so that the position, size, and range of the area of ​​the mirror 12 can be appropriately determined based on the positions of these reflecting portions 13.

[0046] Furthermore, in the abandoned object detection device 10 of this embodiment, the mirror 12 is provided with a QR code 13A indicating mirror information including curvature information of the mirror 12, in place of at least one reflective section 13 or superimposed on the reflective section 13, so that the QR code 13A can be acquired through the camera 11, and curvature information of the mirror 12, etc. can be easily acquired.

[0047] Furthermore, the abandoned object detection device 10 of this embodiment is equipped with a mirror position correction unit 24 that corrects the relative position of the mirror 12 with respect to the camera 11 using the reflecting portion 13 as a landmark, and therefore, based on the position of the reflecting portion 13 provided on the mirror 12, the position of the mirror 12 with respect to the camera 11 can be appropriately corrected so that at least the entire mirror 12 is included in the captured image.

[0048] [Second embodiment] FIG. 6 is a block diagram showing an example of the configuration of an abandonment detection device according to the second embodiment. FIG. 7 is a schematic diagram showing an example of the configuration of a vehicle equipped with an abandonment detection device according to the second embodiment. The same components as those in the abandonment detection device 10 according to the first embodiment are assigned the same reference numerals, and descriptions thereof will be omitted. In this second embodiment, the abandonment detection device is equipped in a large vehicle such as a bus, and detects whether or not a passenger is present in the vehicle (whether or not the passenger has been abandoned) after the driver leaves the vehicle (driver's seat). Furthermore, when the abandonment detection device detects the presence (abandonment) of a passenger (occupant), it notifies the driver or a predetermined contact point (police station or fire station) to that effect.

[0049] As shown in FIG. 6, the abandonment detection device 10A includes a plurality of (three in this embodiment) camera units 50A, 50B, and 50C, and a control unit 20A.

[0050] Camera unit 50A includes one camera 11A and multiple (three in this embodiment) mirrors 12A arranged within the image capturing range of camera 11A. Camera unit 50B and camera unit 50C also have the same configuration. That is, camera unit 50B includes one camera 11B and multiple (three in this embodiment) mirrors 12B arranged within the image capturing range of camera 11B. Camera unit 50C includes one camera 11C and multiple (three in this embodiment) mirrors 12C arranged within the image capturing range of camera 11C. The configurations of cameras 11A to 11C and mirrors 12A to 12C are the same as those of camera 11 and mirror 12 described above.

[0051] As shown in FIG. 7, these camera units 50A-50C are provided in each area of ​​the interior of a large vehicle 1A such as a bus. In the example of FIG. 7, the vehicle 1A has nine rows of seats 2 from front to rear. For example, camera unit 50A is arranged corresponding to seats 2 in rows 1-3, camera unit 50B is arranged corresponding to seats 2 in rows 4-6, and camera unit 50C is arranged corresponding to seats 2 in rows 7-9. In this configuration, the number of cameras is minimized and each camera unit 50A-50C detects an abandoned passenger in its corresponding seat (area), thereby simplifying the device configuration. Note that the imaging range of camera 11A includes the driver's seat, and the image acquisition unit 22 acquires captured image information from camera 11A, and the disembarkation detection unit 21 detects the driver's disembarkation based on the detection result of at least the driver's face through image analysis. Therefore, even if the driver looks around at each row of seats 2 to inspect the interior of the vehicle, the person will not be mistakenly identified as an abandoned passenger.

[0052] In camera units 50A-50C, cameras 11A-11C are disposed on one side of the interior surface of vehicle 1A, and mirrors 12A-12C are disposed on the other side of the interior surface. Specifically, as shown in FIG. 7, camera 11A of camera unit 50A is disposed on the upper part (or ceiling surface) of left interior surface 7 corresponding to seat 2 in the first row of vehicle 1A, and three mirrors 12A are disposed on the upper part (or ceiling surface) of right interior surface 8 corresponding to seats 2 in the first to third rows of vehicle 1A, respectively. In this case, the left-right direction refers to the direction defined when facing forward inside vehicle 1A. With this configuration, the blind spot of camera 11A is projected as a virtual image on mirror 12A of each row, making it possible to reliably determine whether or not a passenger is present in seat 2 of each row using the captured image.

[0053] In this embodiment, the cameras 11A to 11C and the mirrors 12A to 12C are alternately arranged in each of the camera units 50A to 50C. Specifically, the camera 11A and the mirror 12A of the camera unit 50A are arranged as described above, the camera 11B of the camera unit 50B is arranged on the upper part (or the ceiling surface) of the right-side interior surface 8 corresponding to the seat 2 in the fourth row of the vehicle 1A, as shown in FIG. 7, and the three mirrors 12B are arranged on the upper part (or the ceiling surface) of the left-side interior surface 7 corresponding to the seats 2 in the fourth to sixth rows of the vehicle 1A. Similarly, the camera 11C of the camera unit 50C is arranged on the upper part (or the ceiling surface) of the left-side interior surface 7 corresponding to the seat 2 in the seventh row of the vehicle 1A, and the three mirrors 12C are arranged on the upper part (or the ceiling surface) of the right-side interior surface 8 corresponding to the seats 2 in the seventh to ninth rows of the vehicle 1A.

[0054] In this way, the cameras 11A to 11C and the mirrors 12A to 12C for each of the camera units 50A to 50C are arranged alternately along the inner surface of the vehicle 1A, so that the camera units 50A to 50C can be arranged efficiently so that the target area of ​​each of the camera units 50A to 50C can be visually understood. Note that the arrangement of the camera units is not limited to this, and can be changed as appropriate as long as the combination of cameras and mirrors reduces the blind spots of the cameras.

[0055] As shown in FIG. 6, the control unit 20A includes a disembarkation detection unit 21, an image acquisition unit 22, a mirror information acquisition unit 23, a mirror position correction unit (position correction unit) 24, an analysis unit 25A, a first judgment unit 26, a second judgment unit 27, a warning unit 28, and a communication control unit 29, which are all connected to each other via a bus BS.

[0056] The analysis unit 25A sequentially analyzes the images reflected on the mirrors 12A to 12C, which correspond to the regions of the mirrors 12A to 12C in the captured image information of the cameras 11A to 11C. Specifically, the analysis unit 25A identifies the contours of the mirrors 12A to 12C for each of the camera units 50A to 50C based on the positions of the reflectors 13 provided on the mirrors 12A to 12C, and sequentially extracts partial image information corresponding to the regions of the mirrors 12A to 12C in the acquired captured image information. The analysis unit 25A also sequentially corrects distortion of the partial image information based on curvature information included in the acquired mirror information. In this embodiment, in order to reduce the processing load, the analysis unit 25A does not simultaneously analyze the images reflected on the mirrors 12A to 12C, but performs these analyses sequentially.

[0057] Next, the operation of the abandonment detection device 10A according to the second embodiment will be described below. Fig. 8 is a flowchart showing the procedure of the operation of the abandonment detection device according to the second embodiment.

[0058] The control unit 20A acquires captured image information of the interior of the vehicle 1A for each of the camera units 50A to 50C (step S11). The control unit 20A acquires, via the image acquisition unit 22, the captured image information output by each of the cameras 11 in each of the camera units 50A to 50C.

[0059] The control unit 20A detects that the driver has dismounted (step S12). In this embodiment, the control unit 20A detects that the driver has dismounted based on the detection result of at least the face of the driver by the dismount detection unit 21 through video analysis of the captured video information acquired by the video acquisition unit 22.

[0060] Next, the control unit 20A determines whether or not the passenger is not captured in any of the captured images of the interior of the vehicle 1A (step S13). More specifically, the control unit 20A causes the first determination unit 26 to execute face recognition processing on the captured image information acquired from each of the cameras 11A to 11C, and determines whether or not the face of the passenger of the vehicle 1A is detected, that is, whether or not the passenger is not captured in any of the captured images. If it is determined that the passenger is not captured in any of the captured images (step S13; Yes), the control unit 20A acquires mirror information for each of the mirrors 12A to 12C of the camera units 50A to 50C (step S14). The control unit 20A then causes the mirror information acquisition unit 23 to acquire mirror information for each of the mirrors 12A to 12C corresponding to the QR code 13A from the captured image information of the interior of the vehicle 1A. This mirror information includes at least curvature information for each of the mirrors 12A to 12C.

[0061] On the other hand, if it is determined that a passenger appears in at least one captured image (step S13; No), the control unit 20A shifts the process to step S17.

[0062] Next, control unit 20A sequentially analyzes the images reflected in the areas of each mirror in the captured image (step S15). More specifically, control unit 20A sequentially analyzes the images reflected on mirrors 12A-12C that correspond to the areas of mirrors 12A-12C in the captured image information of each camera 11A-11C using analysis unit 25A. As described above, mirrors 12A-12C are each formed by bending with a predetermined curvature. Therefore, the images (virtual images) of the object reflected on mirrors 12A-12C are distorted relative to the real images of the object due to the curvature of mirrors 12A-12C. Control unit 20A identifies the contours of mirrors 12A-12C based on the positions of multiple reflectors 13 provided on mirrors 12A-12C, using analysis unit 25A, and sequentially extracts partial image information 30 corresponding to the areas of mirrors 12A-12C in the captured image information acquired. Furthermore, the control unit 20A causes the analysis unit 25A to sequentially generate corrected image information 31 in which distortion of the partial image information 30 is corrected based on the curvature information included in the acquired mirror information.

[0063] Next, the control unit 20A determines whether or not a passenger is reflected in the image corresponding to the area of ​​at least one of the mirrors 12A to 12C (step S16). More specifically, the control unit 20A sequentially determines whether or not a passenger is reflected in the analyzed image of each of the mirrors 12A to 12C using the second determination unit 27. In this case, it is sufficient to sequentially determine whether or not a passenger is reflected in the image in the order of analysis. The second determination unit 27 sequentially performs a physique recognition process on the image (corrected image information) corresponding to the analyzed area of ​​the mirrors 12A to 12C. The second determination unit 27 detects feature points of the passenger's physique (for example, the shape of the back of the head, back, or waist), that is, determines whether or not the passenger is reflected in at least one corrected image. Note that the second determination unit 27 may perform a face recognition process in addition to the physique recognition process. In this determination, if the passenger is not reflected in the images corresponding to the areas of all mirrors 12A to 12C (step S16; No), the passenger left behind in vehicle 1A has not been found, and the process ends. On the other hand, if the passenger is reflected in the images corresponding to the areas of at least one of mirrors 12A to 12C (step S16; Yes), control unit 20A proceeds to step S17.

[0064] If a passenger is visible in the image in steps S13 and S16, the passenger has been left behind in vehicle 1A, and control unit 20A outputs a warning to the driver's mobile phone (step S17). Specifically, control unit 20A causes warning unit 28 to output a warning to the driver of vehicle 1A that a passenger has been left behind in vehicle 1A. Then, control unit 20A controls communication control unit 29 to operate communication unit 14 to notify the driver of vehicle 1A of the fact that the passenger has been left behind to the mobile phone of the driver of vehicle 1A.

[0065] After notifying the driver's mobile phone, the control unit 20A determines whether a predetermined time has elapsed (step S18). This predetermined time can be set appropriately (for example, 5 minutes) as a grace period until the driver returns to the vehicle. If the predetermined time has not elapsed (step S18; No), the control unit 20A repeatedly executes the process of step S18 until the predetermined time has elapsed. On the other hand, if the predetermined time has elapsed (step S18; Yes), the control unit 20A determines whether the driver has returned to the vehicle (step S19). The determination that the driver has returned to the vehicle 1A can be made, for example, by detecting the opening of the door of the vehicle 1A.

[0066] If the driver has returned to the vehicle (step S19; Yes), the control unit 20A ends the process. On the other hand, if the driver has not returned to the vehicle (step S19; No), the control unit 20A controls the operation of the communication unit 14 to make an emergency call to the police station or fire station (step S20). More specifically, the control unit 20A controls the operation of the communication unit 14 by the communication control unit 29 to make an emergency call to the police station or fire station, and ends the process.

[0067] The abandoned vehicle detection device 10A according to this embodiment includes a plurality of sets of camera units 50A to 50C, each set including one camera 11A to 11C that captures an image of the interior of the vehicle 1A, and a plurality of mirrors 12A to 12C that are respectively arranged within the image capturing areas of the cameras 11A to 11C and that capture the blind spot ranges of the cameras 11A to 11C in the interior of the vehicle, a vehicle alighting detection unit 21 that detects the driver alighting, and a vehicle abandonment detection unit 22 that, after the driver alighting is detected, monitors all of the cameras 11A to 11C. The system further includes a first determination unit 26 that determines whether a passenger of vehicle 1A is visible in the images captured by cameras 11A-11C of all camera units 50A-50C; an analysis unit 25A that sequentially analyzes the images captured by the mirrors 12A-12C corresponding to the areas of the mirrors 12A-12C in the images captured by all cameras 11A-11C when it is determined that a passenger is not visible in the images captured by all cameras 11A-11C of all camera units 50A-50C; and a second determination unit 27 that determines whether a passenger is visible in the analyzed images. With this configuration, for example, by providing a camera unit 50A including one camera 11A and multiple mirrors 12A, it is possible to capture images of a portion of the interior of vehicle 1A, including blind spots, thereby achieving a simple device configuration. Furthermore, by providing multiple camera units 50A-50C of this type, it is possible to detect whether an object has been left in the seat (area) corresponding to each camera unit 50A-50C, even in a large vehicle such as a bus, with the number of cameras minimized. Furthermore, according to this configuration, a two-stage judgment process is performed by the first judgment unit 26 and the second judgment unit 27, so that the processing load of the captured image can be reduced while accurately detecting whether or not there is a passenger left in the vehicle.

[0068] Furthermore, in this embodiment, camera unit 50A has camera 11A positioned on one side of the inner surface of vehicle 1A and mirror 12A positioned on the other side of the inner surface, so that the blind spot of camera 11A is projected as a virtual image on mirror 12A of each row, and therefore it is possible to reliably determine whether or not there is a passenger in seat 2 of each row using the captured image.

[0069] In addition, in this embodiment, the cameras 11A to 11C and the mirrors 12A to 12C are arranged alternately along the inner surface of the vehicle 1A for each of the camera units 50A to 50C, so that the camera units 50A to 50C can be arranged efficiently so that the target area of ​​each camera unit 50A to 50C can be visually understood.

[0070] Although the embodiments of the present invention have been described above, the embodiments are not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of what is called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0071] In this embodiment, the passenger is specifically described as a preschool child, but the present invention is not limited to this and applies to all age groups. Furthermore, the passenger does not have to be a person, and may be a pet animal such as a dog or cat, as long as it can be detected by a camera.

[0072] Furthermore, each component of the control unit (abandonment detection control device) 20, 20A according to this embodiment is a functional concept and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of each device is not limited to that shown in the figure, and all or part of each device may be functionally or physically distributed or integrated in any unit depending on the processing load and usage status of each device. The configuration of the control unit 20, 20A is realized, for example, as software by a program loaded into memory. In the above embodiment, these functional blocks have been described as being realized by cooperation between these hardware and software. That is, these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof. [Explanation of symbols]

[0073] 1, 1A vehicle 2 seats 3. Windshield 4 Rear window 5 Ceiling 6 Blind Spot Range 7 Inside left side 8 Right inner surface 10, 10A Abandonment detection device 11, 11A, 11B, 11C Camera 12, 12A, 12B, 12C mirrors 13 Reflector 13A QR code (information code section) 14 Communications Department 20, 20A control section 21 Exit detection unit 22 Video acquisition unit 23 Mirror information acquisition unit 24 Mirror position correction unit (position correction unit) 25, 25A analysis section 26 1st Judgment Section 27 Second Judgment Section 50A, 50B, 50C camera units

Claims

1. a plurality of sets of camera units each including a camera that photographs the interior of the vehicle and a plurality of mirrors that are arranged within the photographing area of ​​the camera and that reflect the blind spot range of the camera within the interior of the vehicle; a vehicle exit detection unit that detects when the driver exits the vehicle; a first determination unit that determines whether or not an occupant of the vehicle is captured in images captured by all of the cameras in all of the camera units after the driver has exited the vehicle; an analysis unit that, when it is determined that the occupant is not captured in the captured images of all the cameras in all the camera units, sequentially analyzes the images captured in the mirrors, the analysis unit corresponding to the areas of the mirrors in the captured images of all the cameras; a second determination unit that determines whether the occupant is captured in the analyzed video; An abandoned vehicle detection device comprising:

2. 2. The abandoned vehicle detection device according to claim 1, wherein the camera unit has the camera disposed on one side of an inner surface of the vehicle and the mirror disposed on the other side of the inner surface.

3. 3. The abandoned vehicle detection device according to claim 1, wherein the cameras and the mirrors are arranged alternately along an inner surface of the vehicle for each of the camera units.

4. 3. The abandoned object detection device according to claim 1, wherein the mirror includes a plurality of reflecting portions that indicate an outer edge of the mirror and that reflect light emitted from the camera toward the camera.

5. A method for detecting whether an occupant has been left behind in a vehicle cabin using a plurality of camera units including one camera that photographs the interior of the vehicle cabin and a plurality of mirrors that are arranged within a photographing area of ​​the camera and that reflect the blind spot range of the camera in the cabin, Detecting that the driver has exited the vehicle; a step of determining whether or not an occupant of the vehicle is captured in images captured by all of the cameras in all of the camera units after detecting that the driver has exited the vehicle; when it is determined that the occupant is not captured in the captured images of all the cameras in all the camera units, sequentially analyzing the images captured in the mirrors that correspond to the areas of the mirrors in the captured images of all the cameras; determining whether the occupant is present in the analyzed video; A method for detecting abandonment by having a computer execute the above.

Images