Abandoned object detection device and abandoned object detection method

Abstract

This invention provides a vehicle abandonment detection device and method that can detect the presence or absence of passengers left inside a vehicle while reducing the processing load on captured video with a simple device configuration. [Solution] The system includes a camera 11 that photographs the interior of the vehicle 1, a mirror 12 positioned within the shooting area of ​​the camera 11 that reflects the blind spot area 6 of the camera 11 inside the vehicle, a passenger exit detection unit 21 that detects when the driver exits the vehicle, a first determination unit 26 that determines whether or not a passenger of the vehicle 1 is visible in the image captured by the camera 11 after the driver has been detected to have exited the vehicle, an analysis unit 25 that analyzes the image corresponding to the area of ​​the mirror 12 in the captured image and the image reflected in the mirror 12 if it is determined that no passenger is visible in the captured image, and a second determination unit 27 that determines whether or not a passenger is visible in the analyzed image.

Description

Technical Field

[0001] The present invention relates to an abandonment detection device and an abandonment detection method.

Background Art

[0002] In recent years, a situation where a passenger (a passenger other than the driver) is left unattended in the vehicle interior, so-called abandonment of a passenger in the vehicle, has become a problem. This type of abandonment occurs when the driver gets off the vehicle without noticing that there is a passenger remaining in the vehicle, or when, despite the driver recognizing that there is a passenger remaining in the vehicle, it takes some time for some reason until the driver gets back on the vehicle after getting off. There are many cases caused by this, and there is also concern about the impact on the health of the abandoned passenger. For this reason, conventionally, a technique for detecting abandonment of a passenger based on the photographed image of a camera mounted in the vehicle interior is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a conventional configuration, for example, a passenger in the front seat of a vehicle is reflected in the photographed image of a camera, but a passenger in the rear seat may not be reflected in the photographed image when located in a blind spot of the camera hidden by the front seat. Here, it is also conceivable to arrange a plurality of cameras in the vehicle to reduce the blind spots of each camera. However, in this type of configuration, the device configuration becomes complicated and the processing load of the photographed image increases, so there is room for improvement in this regard.

[0005] The present invention has been made in view of the above, and aims to provide an abandoned vehicle detection device and an abandoned vehicle detection method that can detect the presence or absence of a passenger left inside a vehicle while reducing the processing load of captured video with a simple device configuration. [Means for solving the problem]

[0006] To solve the above-mentioned problems and achieve the objective, the abandoned vehicle detection device according to the present invention comprises a camera that photographs the interior of a vehicle, a mirror positioned within the camera's shooting area that reflects the camera's blind spot in the interior, a vehicle exit detection unit that detects the driver's exit, a first determination unit that determines whether or not a vehicle occupant is visible in the camera's captured image after the driver's exit has been detected, an analysis unit that analyzes the image corresponding to the mirror's area in the captured image and the image reflected in the mirror if it is determined that no occupant is visible in the captured image, and a second determination unit that determines whether or not an occupant is visible in the analyzed image.

[0007] Furthermore, the abandoned vehicle detection method according to the present invention is an abandoned vehicle detection method that uses a camera that photographs the interior of a vehicle and a mirror placed within the camera's shooting area that reflects the camera's blind spot in the interior to detect whether an occupant has been left behind in the interior, and involves having a computer perform the following steps: detecting that the driver has left the vehicle; determining whether or not an occupant of the vehicle is visible in the camera's captured image after the driver has left the vehicle; if it is determined that no occupant is visible in the captured image, analyzing the image corresponding to the area of ​​the mirror in the captured image and the image reflected in the mirror; and determining whether or not an occupant is visible in the analyzed image. [Effects of the Invention]

[0008] According to the present invention, a simple device configuration can be used to reduce the processing load on captured video while simultaneously detecting whether or not there are passengers remaining inside the vehicle. [Brief explanation of the drawing]

[0009] [Figure 1]Figure 1 is a block diagram showing an example configuration of an abandoned object detection device according to the first embodiment. [Figure 2] Figure 2 is a schematic diagram showing an example of the configuration of a vehicle equipped with an abandoned vehicle detection device. [Figure 3] Figure 3 is a schematic diagram showing an example of a camera from an abandoned vehicle detection device capturing images of the vehicle's interior. [Figure 4] Figure 4 is a flowchart showing the operation procedure of the abandoned object detection device according to the first embodiment. [Figure 5] Figure 5 shows an example of a corrected image in which the distortion of the image reflected in the mirror has been corrected. [Figure 6] Figure 6 is a block diagram showing an example configuration of an abandoned object detection device according to the second embodiment. [Figure 7] Figure 7 is a schematic diagram showing an example of the configuration of a vehicle equipped with an abandoned vehicle detection device according to the second embodiment. [Figure 8] Figure 8 is a flowchart showing the operation procedure of the abandoned object detection device according to the second embodiment. [Modes for carrying out the invention]

[0010] Embodiments of the present invention will be described in detail below with reference to the drawings. However, the present invention is not limited to the embodiments described below.

[0011] [First Embodiment] Figure 1 is a block diagram showing an example configuration of the abandoned vehicle detection device according to the first embodiment. Figure 2 is a schematic diagram showing an example configuration of a vehicle equipped with the abandoned vehicle detection device. Figure 3 is a schematic diagram showing an example of the abandoned vehicle detection device's camera capturing the interior of the vehicle. The abandoned vehicle detection device 10 is installed in a vehicle and detects whether or not there are passengers inside the vehicle after the driver leaves the vehicle (driver's seat) (whether or not they have been abandoned). Furthermore, when the abandoned vehicle detection device detects the presence of a passenger (occupant) (abandonment), it notifies the driver or a designated contact (police station or fire station).

[0012] In this embodiment, as an example of how the abandoned passenger detection device detects an abandoned passenger, we describe a case where the driver leaves the vehicle without realizing there is a passenger inside. However, it may also be applied to cases where the driver is aware there is a passenger inside the vehicle and then temporarily leaves the vehicle and does not return for a predetermined period of time or longer. Furthermore, the passenger may be, for example, a toddler, but could also be an infant, an elderly person, a person requiring care, or even an animal such as a dog or cat. Also, the vehicle may be, for example, a regular passenger car, but could also include public transportation such as route buses or shuttle buses.

[0013] As shown in Figure 1, the abandoned object detection device 10 includes a camera 11, a mirror 12, a communication unit 14, and a control unit (abandoned object detection control device) 20.

[0014] Camera 11 is a camera that photographs the interior of the vehicle. Camera 11 is mounted on the vehicle and photographs the interior of the vehicle, for example, from the front to the rear in the direction of travel of the vehicle. Camera 11 is positioned to photograph at least the faces of passengers, including the driver, who are seated in all seats 2 (front seats 2A and rear seats 2B) of the vehicle 1. In this embodiment, as shown in Figure 2, camera 11 is positioned above the vehicle's windshield 3 or around the rearview mirror (not shown) to photograph the interior of the vehicle. Camera 11 has a fixed or nearly fixed shooting range and shooting direction.

[0015] The camera 11 is, for example, a camera capable of photographing near-infrared light, and is a camera that mainly detects and visualizes electromagnetic waves in the infrared wavelength region (wavelength of 0.8 μm or more and 1.5 μm or less). The camera 11 can be used as a night surveillance camera that photographs the interior of the vehicle 1 even at night when there is little reception of visible light, for example, by including infrared LED illumination as a light source. The camera 11 is configured to be able to be activated even when the vehicle 1 (engine) is stopped, and is activated, for example, triggered by the driver getting off the vehicle, and photographs the interior of the vehicle 1. Also, the camera 11 may be configured to photograph the interior of the vehicle 1 constantly or until a predetermined period after the engine stops. In this case, it is possible to detect by video that the driver has gotten off the vehicle. The camera 11 outputs the photographed video to the video acquisition unit 22 (FIG. 1) of the control unit 20.

[0016] As shown in FIGS. 2 and 3, the mirror 12 is disposed within the imaging area (angle of view) of the camera 11 and reflects a virtual image of an object in the dead angle range 6 (FIG. 2) of the camera 11 in the interior of the vehicle 1. Specifically, as shown in FIG. 2, the mirror 12 is fixed to the upper part of the rear glass 4 or the ceiling part 5 of the vehicle 1 and disposed within the imaging area of the camera 11. The mirror 12 is attached so that the position of the mirror 12 can be adjusted, for example, via a drive unit (not shown). Therefore, by operating this drive unit, the relative position of the mirror 12 with respect to the camera 11 can be corrected.

[0017] As described above, the camera 11 is capable of photographing at least the faces of the passengers (including the driver) sitting in all the seats 2 of the vehicle 1. In this case, for example, the space between the front seat 2A and the rear seat 2B is hidden by the backrest of the front seat 2A, and thus becomes the dead angle range 6 of the camera 11. The mirror 12 is disposed opposite to the back side of the backrest of the front seat 2A, that is, in the dead angle range 6 of the camera 11, and can reflect a virtual image of an object in the dead angle range 6 into the imaging area of the camera 11.

[0018] Further, the mirror 12 is a curved mirror that is curved in the width direction and the height direction, and the central portion protrudes forward (toward the camera 11 side) more than the edge portion. By using this type of curved mirror, virtual images in a wide range in the width direction and the height direction inside the vehicle 1 can be reflected on the mirror 12. On the other hand, since the mirror 12 is arranged within the imaging region of the camera 11 as described above, the captured video includes a real image of the interior of the vehicle and a virtual image reflected on the mirror 12. Therefore, in the present embodiment, as shown in FIG. 3, the mirror 12 includes a plurality (four in FIG. 3) of reflecting portions 13 that show the contour (outer edge) of the mirror 12 and reflect the light (for example, infrared light) emitted from the camera 11 back to the camera 11.

[0019] The plurality of reflecting portions 13 are each formed of, for example, a plane mirror, and these reflecting portions 13 are each arranged at the edge portion of the mirror 12. Therefore, by connecting adjacent reflecting portions 13 to each other, the contour of the mirror 12 is formed. Further, the plurality of reflecting portions 13 are each provided so that the angle with respect to the mirror 12 can be adjusted. According to this configuration, after the mirror 12 is attached to the vehicle and the relative position of the mirror 12 with respect to the camera 11 is adjusted, by adjusting the positions of the respective reflecting portions 13, each reflecting portion 13 reflects the light emitted from the camera 11 back to the camera 11. Therefore, the region of the mirror 12 can be defined based on the positions of the reflecting portions 13. The adjustment of the positions of the reflecting portions 13 can be manually performed by the driver or a passenger, or may be performed using the drive unit for adjusting the position of the mirror 12 described above.

[0020] Furthermore, in this embodiment, as shown in Figure 3, the mirror 12 is configured to have a QR code (registered trademark) 13A (information code portion) superimposed on at least one of the multiple reflective portions 13. This QR code 13A contains mirror information that includes at least curvature information of the mirror 12, and this mirror information may further include information regarding the size of the mirror 12 and the number of reflective portions 13. With this configuration, the QR code 13A superimposed on the reflective portion 13 can be acquired through the camera 11, and the curvature information of the mirror 12 can be easily acquired. In addition, although this embodiment has been described as a configuration in which the QR code 13A is superimposed on at least one reflective portion 13, for example, it may be provided in place of at least one of the multiple reflective portions 13.

[0021] The communication unit 14 communicates with devices outside the vehicle via the internet or a mobile phone network. For example, if the communication unit 14 detects that a passenger has been left behind in the vehicle, it sends a notification to the driver's mobile phone, which has been registered in advance. Furthermore, if the communication unit 14 detects that a passenger has been left behind in the vehicle and a predetermined time has elapsed, the detection of the passenger being left behind has not been resolved (the driver has not returned to the vehicle), it sends an emergency call to the police station or fire station.

[0022] The control unit 20 is a processing unit composed of, for example, a CPU (Central Processing Unit). As shown in Figure 1, the control unit 20 includes a disembarkation detection unit 21, a video acquisition unit 22, a mirror information acquisition unit 23, a mirror position correction unit (position correction unit) 24, an analysis unit 25, a first determination unit 26, a second determination unit 27, a warning unit 28, and a communication control unit 29, all of which are connected via a bus BS. The control unit 20 realizes these configurations and performs these processes by reading and executing a program stored in a memory unit (not shown). The control unit 20 may perform these processes with a single CPU, or it may have multiple CPUs and perform these processes in parallel with the multiple CPUs.

[0023] The disembarkation detection unit 21 detects when the driver has disembarked and left the vehicle. The disembarkation detection unit 21 detects the driver's disembarkation based, for example, on video analysis of the camera 11 that takes pictures of the interior of the vehicle 1, or on the detection result of a pressure sensor (not shown) installed in the driver's seat that determines whether the driver is seated or not. Alternatively, the disembarkation detection unit 21 may detect the driver's disembarkation when the driver's door is opened and then closed again while the engine (drive unit) is stopped, or it may detect the driver's disembarkation by using a weight sensor (not shown) that monitors the weight of the vehicle and detecting a reduction in the vehicle's weight as the driver's door opens and closes.

[0024] The video acquisition unit 22 acquires video information of the interior of the vehicle. The video acquisition unit 22 acquires video information output by the camera 11 at all times, or for a predetermined period after the engine has stopped, or after the driver has exited the vehicle.

[0025] The mirror information acquisition unit 23 acquires mirror information corresponding to the QR code 13A captured by the camera 11. The mirror information acquisition unit 23 acquires at least curvature information of the mirror 12 as mirror information.

[0026] The mirror position correction unit 24 corrects the relative position of the mirror 12 with respect to the camera 11 based on the image captured by the camera 11. Specifically, the mirror position correction unit 24 corrects the relative position of the mirror 12 with respect to the camera 11 based on the position of the reflective part 13 provided on the mirror 12, so that at least the entire mirror 12 is included in the captured image. In this case, for example, the system may be configured to automatically correct the mounting position of the mirror 12 or the camera 11, or it may be configured to issue a message to prompt the driver or passenger to correct these mounting positions. Furthermore, in this embodiment, the mirror position correction unit 24 is configured to correct the position of the mirror 12 with respect to the camera 11, but for example, it may be configured to correct the position of the camera 11 with respect to the mirror 12.

[0027] The analysis unit 25 corresponds to the area of ​​the mirror 12 in the acquired captured video information and analyzes the image reflected in the mirror 12. As described above, the mirror 12 is formed by bending it with a predetermined curvature. Therefore, the image (virtual image) of the object reflected in the mirror 12 is distorted relative to the real image of the object due to the curvature of the mirror 12. For this reason, the analysis unit 25 extracts partial video information corresponding to the area of ​​the mirror 12 in the acquired captured video information, corrects the distortion of this partial video information, and obtains corrected video information.

[0028] Specifically, the analysis unit 25 identifies the contour of the mirror 12 based on the positions of the multiple reflective parts 13 provided on the mirror 12 and extracts partial image information corresponding to the area of ​​the mirror 12 in the acquired captured image information. Furthermore, the analysis unit 25 corrects the distortion of the partial image information based on the curvature information contained in the acquired mirror information and converts it into corrected image information.

[0029] The first determination unit 26 determines, after detecting that the driver has exited the vehicle, whether or not a passenger in vehicle 1 is visible in the video footage captured by camera 11. The first determination unit 26 performs face recognition processing on the video footage (video information) captured by camera 11 and determines whether or not the face of a passenger in vehicle 1 is detected. Here, "not showing a passenger" includes cases where a passenger is reflected in mirror 12 but their face cannot be detected. If the first determination unit 26 detects a passenger's face, it is determined that a passenger has been left behind in vehicle 1. The method for detecting a person's face from the video footage can be any well-known method and is not limited to that.

[0030] The second determination unit 27 determines whether or not a passenger is reflected in the analyzed image of the mirror 12. The second determination unit 27 performs body shape recognition processing on the image (corrected image information) corresponding to the area of ​​the analyzed mirror 12. Since the mirror 12 reflects the blind spots inside the vehicle 1, the frequency with which a passenger's face is reflected in this mirror 12 is usually low. Therefore, the second determination unit 27 determines whether or not characteristic points of the passenger's body shape (for example, the shape of the back of the head, back, or waist) are detected. If the second determination unit 27 detects characteristic points of the passenger's body shape, it is determined that a passenger has been left behind inside the vehicle 1. Known methods can also be used for detecting this type of body shape.

[0031] As described above, in this embodiment, the control unit 20 performs a two-stage determination process using the first determination unit 26 and the second determination unit 27. Therefore, if the first determination unit 26 determines that a passenger is visible, the analysis unit 25 does not analyze the video, and the second determination unit 27 does not perform the determination. This reduces the processing load on the captured video while achieving accurate detection of a person being left behind. In this embodiment, the analysis unit 25 and the second determination unit 27 are configured separately, but the analysis unit 25 and the second determination unit 27 may be integrated using a generative AI (Artificial Intelligence) that has learned from this body size information as event data.

[0032] The warning unit 28 outputs a warning to the driver of vehicle 1 when it determines that a passenger has been left behind inside vehicle 1. More specifically, the warning unit 28 outputs a warning to the driver of the vehicle when the first determination unit 26 or the second determination unit 27 determines that a passenger is visible in the target video. The warning may also be output as an audio signal from the vehicle's speaker (not shown).

[0033] The communication control unit 29 controls the operation of the communication unit 14. Specifically, when the warning unit 28 outputs a warning to the vehicle driver, the communication control unit 29 controls the operation of the communication unit 14 to notify the driver of vehicle 1's mobile phone that they have been left behind. This notification may be a text notification via email, for example, but it is preferable to be a voice notification via telephone. This allows the driver of vehicle 1 to be notified via their mobile phone that they have been left behind. Furthermore, if the driver has not returned to vehicle 1 after a predetermined time (for example, 5 minutes) has elapsed since the notification was sent to the driver's mobile phone (for example, the vehicle doors are not opened), the communication control unit 29 controls the operation of the communication unit 14 to make an emergency call to the police station or fire station. This prevents harm to the health of passengers and ensures their safety. The communication control unit 29 may send a notification to the driver's mobile phone and make an emergency call to the police station or fire station simultaneously.

[0034] Next, the operation of the abandoned object detection device 10 according to the first embodiment will be described. Figure 4 is a flowchart showing the procedure of operation of the abandoned object detection device according to the first embodiment. Figure 5 is a diagram showing an example of a corrected image in which the distortion of the image reflected in the mirror has been corrected.

[0035] The control unit 20 acquires video information of the interior of the vehicle 1 (step S1). The control unit 20 acquires the video information output by the camera 11 using the video acquisition unit 22.

[0036] Next, the control unit 20 detects the driver's exit (step S2). In this embodiment, the control unit 20 detects the driver's exit based on the driver's exit detection result obtained by the video acquisition unit 22 through video analysis of the captured video information, using the exit detection unit 21.

[0037] Next, the control unit 20 determines whether or not a passenger is visible in the captured video footage of the interior of vehicle 1 (step S3). More specifically, the control unit 20 uses the first determination unit 26 to perform face recognition processing on the acquired captured video information and determines whether or not the face of a passenger in vehicle 1 is detected, that is, whether or not a passenger is visible in the captured video. If it is determined in this determination that a passenger is visible in the captured video (step S3; Yes), the control unit 20 proceeds to step S7.

[0038] On the other hand, if it is determined that no passengers are visible in the captured video (Step S3; No), the control unit 20 acquires mirror information (Step S4). The control unit 20 uses the mirror information acquisition unit 23 to acquire mirror information corresponding to the QR code 13A from the captured video information of the interior of the vehicle 1. This mirror information includes at least the curvature information of the mirror 12.

[0039] Next, the control unit 20 analyzes the image reflected in the mirror region of the captured video (step S5). More specifically, the control unit 20 uses the analysis unit 25 to analyze the image reflected in the mirror 12, which corresponds to the region of the mirror 12 in the acquired captured video information. In this embodiment, the mirror 12 is formed by bending it with a predetermined curvature. Therefore, the image (virtual image) of the object reflected in the mirror 12 is distorted relative to the real image of the object due to the curvature of the mirror 12. The control unit 20 uses the analysis unit 25 to identify the contour of the mirror 12 based on the positions of the multiple reflective parts 13 provided on the mirror 12, and extracts partial video information 30 corresponding to the region of the mirror 12 in the acquired captured video information, as shown in Figure 5. The control unit 20 also uses the analysis unit 25 to generate corrected video information 31, which corrects the distortion of the partial video information 30 based on the curvature information included in the acquired mirror information.

[0040] Next, the control unit 20 determines whether or not a passenger is reflected in the image corresponding to the area of ​​the mirror 12 (step S6). More specifically, the control unit 20 uses the second determination unit 27 to determine whether or not a passenger is reflected in the analyzed image of the mirror 12. The second determination unit 27 performs body shape recognition processing on the image corresponding to the area of ​​the mirror 12 (corrected image information) that has been analyzed. Since the mirror 12 reflects the blind spots inside the vehicle 1, the frequency with which a passenger's face is reflected in the mirror 12 is usually low. For this reason, the second determination unit 27 determines whether or not characteristic points of the passenger's physique (for example, the shape of the back of the head, back, and waist) are detected, that is, whether or not a passenger is reflected in the corrected image. The second determination unit 27 may also perform face recognition processing in addition to body shape recognition processing. In this determination, if the passenger is not reflected in the image corresponding to the area of ​​the mirror 12 (step S6; No), the passenger left behind in the vehicle 1 is not found, and the process ends. On the other hand, if a passenger is reflected in the image corresponding to the area of ​​mirror 12 (step S6; Yes), the control unit 20 proceeds to step S7.

[0041] In steps S3 and S6, if a passenger is visible, it means the passenger has been left behind in vehicle 1, so the control unit 20 outputs a warning and notifies the driver's mobile phone (step S7). Specifically, the control unit 20 uses the warning unit 28 to output a warning to the driver of vehicle 1 that a passenger has been left behind in vehicle 1. Then, the control unit 20 uses the communication control unit 29 to control the operation of the communication unit 14 to notify the driver of vehicle 1's mobile phone that the passenger has been left behind.

[0042] The control unit 20 determines whether a predetermined time has elapsed after notifying the driver's mobile phone (step S8). This predetermined time can be set appropriately (for example, 5 minutes) as a grace period for the driver to return to the vehicle. If the predetermined time has not elapsed (step S8; No), the control unit 20 repeatedly executes the process in step S8 until the predetermined time has elapsed. On the other hand, if the predetermined time has elapsed (step S8; Yes), the control unit 20 determines whether the driver has returned to the vehicle (step S9). The determination that the driver has returned to vehicle 1 can be made, for example, by detecting the opening of the door of vehicle 1.

[0043] If the driver returns to the vehicle (Step S9; Yes), the control unit 20 terminates processing. On the other hand, if the driver does not return to the vehicle (Step S9; No), the control unit 20 controls the operation of the communication unit 14 to make an emergency call to the police station or fire station (Step S10). More specifically, the control unit 20, with the control of the communication control unit 29, controls the operation of the communication unit 14 to make an emergency call to the police station or fire station and terminates processing.

[0044] The abandoned vehicle detection device 10 according to this embodiment includes a camera 11 that photographs the interior of a vehicle 1, a mirror 12 positioned within the shooting area of ​​the camera 11 and reflecting the blind spot area 6 of the camera 11 inside the vehicle, a passenger exit detection unit 21 that detects the driver exiting the vehicle, a first determination unit 26 that determines whether or not a passenger in the vehicle 1 is visible in the image captured by the camera 11 after the driver has been detected to have exited the vehicle, an analysis unit 25 that analyzes the image corresponding to the area of ​​the mirror 12 in the captured image and the image reflected in the mirror 12 if it is determined that no passenger is visible in the captured image, and a second determination unit 27 that determines whether or not a passenger is visible in the analyzed image. With this configuration, for example, it is possible to photograph the blind spot area 6 inside the vehicle 1 using only one camera 11 and one mirror 12, thus enabling a simple device configuration. Furthermore, this configuration allows for two-stage determination processing by the first determination unit 26 and the second determination unit 27, thereby reducing the processing load on the captured video while accurately detecting whether or not there are passengers left inside the vehicle.

[0045] Furthermore, in the abandoned object detection device 10 according to this embodiment, the mirror 12 has an outer edge and is equipped with a plurality of reflective parts 13 that reflect light emitted from the camera 11 back towards the camera 11. Therefore, the position, size, and range of the area of ​​the mirror 12 can be appropriately defined based on the positions of these reflective parts 13.

[0046] Furthermore, in the abandoned object detection device 10 according to this embodiment, the mirror 12 is provided with a QR code 13A that indicates mirror information including curvature information of the mirror 12, either in place of or superimposed on at least one reflective portion 13. Therefore, the QR code 13A can be acquired through the camera 11, and the curvature information of the mirror 12 can be easily acquired.

[0047] Furthermore, the abandoned object detection device 10 according to this embodiment includes a mirror position correction unit 24 that corrects the relative position of the mirror 12 with respect to the camera 11 using the reflective portion 13 as a guide. Therefore, based on the position of the reflective portion 13 provided on the mirror 12, the position of the mirror 12 with respect to the camera 11 can be appropriately corrected so that at least the entire mirror 12 is included in the captured image.

[0048] [Second Embodiment] Figure 6 is a block diagram showing an example configuration of the abandoned vehicle detection device according to the second embodiment. Figure 7 is a schematic diagram showing an example configuration of a vehicle equipped with the abandoned vehicle detection device according to the second embodiment. Components identical to those in the abandoned vehicle detection device 10 according to the first embodiment are denoted by the same reference numerals and their explanations are omitted. In this second embodiment, the abandoned vehicle detection device is installed in a large vehicle such as a bus and detects whether or not there are passengers inside the vehicle after the driver has left the vehicle (driver's seat) (whether or not they have been abandoned). Furthermore, when the abandoned vehicle detection device detects the presence of a passenger (occupant) (abandonment), it notifies the driver or a designated contact (police station or fire station).

[0049] As shown in Figure 6, the abandoned object detection device 10A comprises a plurality of (three in this embodiment) camera units 50A, 50B, and 50C, and a control unit 20A.

[0050] Camera unit 50A comprises one camera 11A and a plurality (three in this embodiment) of mirrors 12A positioned within the shooting range of camera 11A. Camera units 50B and 50C have similar configurations. Specifically, camera unit 50B comprises one camera 11B and a plurality (three in this embodiment) of mirrors 12B positioned within the shooting range of camera 11B. Similarly, camera unit 50C comprises one camera 11C and a plurality (three in this embodiment) of mirrors 12C positioned within the shooting range of camera 11C. The configurations of these cameras 11A to 11C and mirrors 12A to 12C are equivalent to those of the camera 11 and mirrors 12 described above.

[0051] These camera units 50A to 50C are installed in each area of ​​the interior of a large vehicle 1A, such as a bus, as shown in Figure 7. In the example in Figure 7, the vehicle 1A has nine rows of seats 2 from front to back. For example, camera unit 50A is positioned to correspond to seats 2 from rows 1 to 3, camera unit 50B is positioned to correspond to seats 2 from rows 4 to 6, and camera unit 50C is positioned to correspond to seats 2 from rows 7 to 9. With this configuration, the number of cameras is minimized, and each camera unit 50A to 50C detects being left behind in their respective corresponding seats (areas), thus simplifying the device configuration. The camera 11A's shooting range includes the driver's seat, and the video acquisition unit 22 acquires the captured video information from camera 11A. The disembarkation detection unit 21 then analyzes the video to detect the driver's disembarkation based on the detection result of at least the driver's face. Therefore, even if the driver looks around at the seats 2 in each row for an interior inspection, they will not be mistakenly identified as a passenger who has been left behind.

[0052] Camera units 50A to 50C each have cameras 11A to 11C positioned on one inner surface of vehicle 1A, and mirrors 12A to 12C positioned on the other inner surface. Specifically, as shown in Figure 7, camera 11A of camera unit 50A is positioned on the upper part (or ceiling) of the left inner surface 7 corresponding to the first row of seats 2 of vehicle 1A, and the three mirrors 12A are positioned on the upper part (or ceiling) of the right inner surface 8 corresponding to the first to third rows of seats 2 of vehicle 1A. In this case, the left-right direction refers to the direction defined when facing forward within vehicle 1A. In this configuration, the blind spot range of camera 11A is projected as a virtual image onto the mirrors 12A of each row, so it is possible to reliably determine whether or not there is a passenger in each row of seats 2 using the captured video.

[0053] Furthermore, in this embodiment, cameras 11A to 11C and mirrors 12A to 12C are arranged alternately for each camera unit 50A to 50C. Specifically, the camera 11A and mirror 12A of camera unit 50A are arranged as described above, the camera 11B of camera unit 50B is positioned on the upper part (or ceiling) of the right inner surface 8 corresponding to the fourth row seat 2 of vehicle 1A, as shown in Figure 7, and the three mirrors 12B are positioned on the upper part (or ceiling) of the left inner surface 7 corresponding to the fourth to sixth row seats 2 of vehicle 1A. Similarly, the camera 11C of camera unit 50C is positioned on the upper part (or ceiling) of the left inner surface 7 corresponding to the seventh row seat 2 of vehicle 1A, and the three mirrors 12C are positioned on the upper part (or ceiling) of the right inner surface 8 corresponding to the seventh to ninth row seats 2 of vehicle 1A.

[0054] In this way, since cameras 11A to 11C and mirrors 12A to 12C are alternately arranged along the inner surface of vehicle 1A for each camera unit 50A to 50C, the camera units 50A to 50C can be efficiently arranged so that the target area of ​​each camera unit 50A to 50C can be visually understood. Note that the arrangement configuration of the camera units is not limited to this, and can be changed as appropriate as long as the combination of cameras and mirrors suppresses the blind spots of the cameras.

[0055] As shown in Figure 6, the control unit 20A includes a disembarkation detection unit 21, an image acquisition unit 22, a mirror information acquisition unit 23, a mirror position correction unit (position correction unit) 24, an analysis unit 25A, a first determination unit 26, a second determination unit 27, a warning unit 28, and a communication control unit 29, all of which are connected via the bus BS.

[0056] The analysis unit 25A corresponds to the region of each mirror 12A to 12C in the captured video information of each camera 11A to 11C, and sequentially analyzes the images reflected in the mirrors 12A to 12C. Specifically, for each camera unit 50A to 50C, the analysis unit 25A identifies the contour of the mirrors 12A to 12C based on the positions of the multiple reflective parts 13 provided on each mirror 12A to 12C, and sequentially extracts partial video information corresponding to the region of the mirrors 12A to 12C in the acquired captured video information. Furthermore, the analysis unit 25A sequentially corrects the distortion of the partial video information based on the curvature information contained in the acquired mirror information. In this embodiment, in order to reduce the processing load, the analysis unit 25A does not analyze the images reflected in the mirrors 12A to 12C simultaneously, but performs these analyses sequentially.

[0057] Next, the operation of the abandoned object detection device 10A according to the second embodiment will be described. Figure 8 is a flowchart showing the operation procedure of the abandoned object detection device according to the second embodiment.

[0058] The control unit 20A acquires captured video information of the interior of the vehicle 1A for each camera unit 50A to 50C (step S11). The control unit 20A acquires captured video information output by each camera 11 in each camera unit 50A to 50C using the video acquisition unit 22.

[0059] The control unit 20A detects the driver's exit (step S12). In this embodiment, the control unit 20A detects the driver's exit based on the driver's exit detection result obtained by the video acquisition unit 22 through video analysis of captured video information, using the exit detection unit 21.

[0060] Next, the control unit 20A determines whether or not passengers are visible in all of the captured images of the interior of vehicle 1A (step S13). More specifically, the control unit 20A uses the first determination unit 26 to perform face recognition processing on multiple captured image data acquired from each camera 11A to 11C and determines whether or not the faces of passengers in vehicle 1A are detected, that is, whether or not passengers are visible in all of the captured images. If, in this determination, it is determined that passengers are not visible in all of the captured images (step S13; Yes), the control unit 20A acquires mirror information for each of the mirrors 12A to 12C of the camera units 50A to 50C (step S14). The control unit 20A uses the mirror information acquisition unit 23 to acquire mirror information for each of the mirrors 12A to 12C corresponding to the QR code 13A from the captured image data of the interior of vehicle 1A. This mirror information includes curvature information for at least each of the mirrors 12A to 12C.

[0061] On the other hand, if it is determined that a passenger is visible in at least one of the captured images (step S13; No), the control unit 20A proceeds to step S17.

[0062] Next, the control unit 20A sequentially analyzes the images reflected in the region of each mirror in the captured video (step S15). More specifically, the control unit 20A uses the analysis unit 25A to sequentially analyze the images reflected in the mirrors 12A to 12C, which correspond to the regions of the mirrors 12A to 12C in the captured video information of each camera 11A to 11C. As described above, the mirrors 12A to 12C are formed by bending them with a predetermined curvature. Therefore, the image (virtual image) of an object reflected in the mirrors 12A to 12C is distorted relative to the real image of the object due to the curvature of the mirrors 12A to 12C. The control unit 20A uses the analysis unit 25A to identify the contours of the mirrors 12A to 12C based on the positions of the multiple reflective parts 13 provided on each mirror, and sequentially extracts partial video information 30 corresponding to the region of the mirrors 12A to 12C in the acquired captured video information. Furthermore, the control unit 20A sequentially generates corrected video information 31 by correcting the distortion of the partial video information 30 based on the curvature information included in the acquired mirror information, using the analysis unit 25A.

[0063] Next, the control unit 20A determines whether or not a passenger is reflected in the image corresponding to the region of at least one mirror 12A to 12C (step S16). More specifically, the control unit 20A sequentially determines whether or not a passenger is reflected in the image of each of the analyzed mirrors 12A to 12C using the second determination unit 27. In this case, it is sufficient to sequentially determine whether or not a passenger is reflected in the image in the order in which it was analyzed. The second determination unit 27 sequentially performs body shape recognition processing on the images (corrected image information) corresponding to the regions of the analyzed mirrors 12A to 12C. The second determination unit 27 determines whether or not a passenger is reflected in at least one corrected image, that is, whether or not characteristic points of the passenger's body shape (for example, the shape of the back of the head, back, and waist) are detected. In addition to body shape recognition processing, the second determination unit 27 may also perform face recognition processing. In this determination, if the passenger is not visible in the images corresponding to each region of all mirrors 12A to 12C (step S16; No), the passenger left behind in vehicle 1A will not be found, and the process will terminate. On the other hand, if the passenger is visible in the images corresponding to at least one region of mirrors 12A to 12C (step S16; Yes), the control unit 20A proceeds to step S17.

[0064] In steps S13 and S16, if a passenger is visible, the control unit 20A outputs a warning and notifies the driver's mobile phone (step S17) because the passenger has been left behind in vehicle 1A. Specifically, the control unit 20A, via the warning unit 28, outputs a warning to the driver of vehicle 1A that a passenger has been left behind in vehicle 1A. Then, the control unit 20A, via the communication control unit 29, controls the operation of the communication unit 14 to notify the driver of vehicle 1A's mobile phone that the passenger has been left behind.

[0065] The control unit 20A determines whether a predetermined time has elapsed after notifying the driver's mobile phone (step S18). This predetermined time can be set appropriately (for example, 5 minutes) as a grace period for the driver to return to the vehicle. If the predetermined time has not elapsed (step S18; No), the control unit 20A repeatedly executes the process in step S18 until the predetermined time has elapsed. On the other hand, if the predetermined time has elapsed (step S18; Yes), the control unit 20A determines whether the driver has returned to the vehicle (step S19). The determination that the driver has returned to vehicle 1A can be made, for example, by detecting the opening of the door of vehicle 1A.

[0066] If the driver returns to the vehicle (step S19; Yes), the control unit 20A terminates processing. On the other hand, if the driver does not return to the vehicle (step S19; No), the control unit 20A controls the operation of the communication unit 14 to make an emergency call to the police station or fire station (step S20). More specifically, the control unit 20A, with the control of the communication control unit 29, controls the operation of the communication unit 14 to make an emergency call to the police station or fire station and terminates processing.

[0067] The abandoned vehicle detection device 10A according to this embodiment includes multiple sets of camera units 50A to 50C, each including one camera 11A to 11C that photographs the interior of the vehicle 1A, and multiple mirrors 12A to 12C that are respectively positioned within the shooting range of these cameras 11A to 11C and each reflects the blind spots of the cameras 11A to 11C inside the vehicle, as well as a disembarkation detection unit 21 that detects the driver disembarking, and after the driver disembarking is detected, all cameras 11A to 50C of all camera units 50A to 50C The system includes a first determination unit 26 that determines whether or not passengers in vehicle 1A are visible in the video footage captured by camera 11C, an analysis unit 25A that sequentially analyzes the video footage corresponding to the area of ​​each mirror 12A to 12C in the video footage captured by all camera units 50A to 50C, and a second determination unit 27 that determines whether or not passengers are visible in the analyzed video. With this configuration, for example, by providing a camera unit 50A that includes one camera 11A and multiple mirrors 12A, it is possible to capture a part of the interior of vehicle 1A, including blind spots, thus realizing a simple device configuration. Furthermore, by providing multiple such camera units 50A to 50C, it is possible to detect unattended persons in the corresponding seats (areas) of each camera unit 50A to 50C even in large vehicles such as buses, with the number of cameras reduced to the minimum. Furthermore, this configuration allows for two-stage determination processing by the first determination unit 26 and the second determination unit 27, thereby reducing the processing load on the captured video while accurately detecting whether or not there are passengers left inside the vehicle.

[0068] Furthermore, in this embodiment, the camera unit 50A has the camera 11A positioned on one inner surface of the vehicle 1A and the mirror 12A positioned on the other inner surface. As a result, the blind spot range of the camera 11A is projected as a virtual image onto the mirror 12A in each row, making it possible to reliably determine whether or not there is a passenger in each row of seats 2 using the captured video.

[0069] Furthermore, in this embodiment, since the cameras 11A to 11C and the mirrors 12A to 12C are alternately arranged along the inner surface of the vehicle 1A for each camera unit 50A to 50C, the camera units 50A to 50C can be efficiently arranged so that the target area of ​​each camera unit 50A to 50C can be visually understood.

[0070] Although embodiments of the present invention have been described above, the embodiments are not limited to those described herein. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the embodiments described above.

[0071] In this embodiment, the passenger is described using a preschool-aged child as an example, but it is not limited to this age group and applies to all age groups. Furthermore, the passenger does not need to be a person; it can be anything detectable by the camera, such as a pet animal like a dog or cat.

[0072] Furthermore, the components of the control units (abandoned object detection control devices) 20 and 20A in this embodiment are functionally conceptual and do not necessarily have to be physically configured as shown. In other words, the specific form of each device is not limited to that shown, and all or part of them may be functionally or physically distributed or integrated in any unit depending on the processing load and usage of each device. The configuration of the control units 20 and 20A is realized, for example, as software, such as a program loaded into memory. In the above embodiment, these were described as functional blocks realized by the cooperation of hardware or software. In other words, these functional blocks can be realized in various forms by hardware alone, software alone, or a combination thereof. [Explanation of symbols]

[0073] 1, 1A Vehicle 2 seats 3. Windshield 4. Rear window 5. Ceiling 6. Blind Spots 7 Inside left side 8 Right inner surface 10, 10A Unattended object detection device 11, 11A, 11B, 11C Cameras 12, 12A, 12B, 12C Miller 13 Reflector 13A QR code (information code section) 14 Communications Department 20, 20A Control Unit 21 Disembarkation detection unit 22 Video Acquisition Unit 23 Mirror Information Acquisition Unit 24 Mirror position correction unit (position correction unit) 25, 25A analysis section 26 1st Judgment Section 27 Second Judgment Section 50A, 50B, 50C Camera Unit

Claims

1. A camera to film the interior of the vehicle, A mirror is placed within the shooting area of ​​the camera and reflects the blind spot of the camera within the room, A disembarkation detection unit that detects when the driver disembarks, After the driver has disembarked, a first determination unit determines whether or not the occupants of the vehicle are visible in the video footage captured by the camera, If it is determined that the occupant is not visible in the captured video, an analysis unit analyzes the area of ​​the mirror in the captured video and the image reflected in the mirror, A second determination unit that determines whether or not the occupant is visible in the analyzed video, An abandoned vehicle detection device equipped with the following features.

2. The abandoned object detection device according to claim 1, wherein the mirror has an outer edge and a plurality of reflective parts that reflect light emitted from the camera toward the camera.

3. The abandoned object detection device according to claim 2, wherein the mirror is provided with an information code section that indicates mirror information including curvature information of the mirror, in place of or superimposed on at least one of the reflective sections.

4. The abandoned object detection device according to claim 2 or 3, further comprising a position correction unit that corrects the relative position of the mirror with respect to the camera using the reflective portion as a marker.

5. An abandoned person detection method that uses a camera that photographs the interior of a vehicle and a mirror placed within the camera's field of view and reflecting the camera's blind spot in the interior to detect whether an occupant has been left behind in the interior, A step to detect the driver getting out of the vehicle, After the driver's exit is detected, the step of determining whether or not the occupants of the vehicle are visible in the video footage captured by the camera, If it is determined that the occupant is not visible in the recorded video, the step is to analyze the area of ​​the mirror in the recorded video and the image reflected in the mirror, A step of determining whether or not the occupant is visible in the analyzed video, A method for detecting abandoned items by having a computer perform the following operation.

Images