Warning device and warning method
A combined far-infrared and visible light camera system addresses the challenge of visually confirming pedestrians by overlaying detected object ranges on the visible light image, enhancing driver awareness and safety.
Patent Information
- Application Number
- JP2021045681
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-19
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-03-19
AI Technical Summary
Far-infrared cameras detect pedestrians and other objects as heat images, making it difficult for drivers to visually confirm their positions outside the vehicle, which reduces the frequency and duration of visual checks, compromising safety.
A system combining a far-infrared camera and a visible light camera to capture images of the vehicle's direction of travel, with a detection unit identifying objects of concern and overlaying their ranges on a visible light image displayed to the driver, enhancing visual confirmation.
Enables drivers to be appropriately informed of detected pedestrians and other objects, reducing the need for frequent visual checks and improving safety by clearly indicating their positions on the displayed image.
Smart Images

Figure 0007585909000001 
Figure 0007585909000002 
Figure 0007585909000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a warning device and a warning method that detects pedestrians and the like in the traveling direction of a vehicle, for example, by using a far-infrared camera, and warns the driver of the vehicle. [Background technology]
[0002] In recent years, far-infrared cameras have been used to detect objects in situations where it is difficult to detect them with a visible light camera in order to ensure safety in the direction in which a vehicle is traveling. An example of such an object detection technology using a far-infrared camera is disclosed in Patent Document 1.
[0003] The recognition processing device described in Patent Document 1 includes an image acquisition unit that acquires first photographed data of the area around the host vehicle captured by a far-infrared camera, an other vehicle detection unit that detects other vehicles parked around the host vehicle, a heat detection unit that detects heat dissipation based on the operation of the other vehicle from the heat distribution in the area corresponding to the other vehicle in the first photographed data, and a person detection unit that, when heat dissipation is detected, performs person recognition preferentially near the other vehicle to detect people. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2020-027380 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, since far-infrared cameras detect the distribution of heat emitted by pedestrians, etc. as an image, the driver of the vehicle may not be able to visually confirm the pedestrians, etc. detected by the far-infrared camera outside the vehicle. In such a situation, even though the pedestrians, etc. are detected by the infrared camera, the driver of the vehicle cannot visually confirm the position of the pedestrian outside the vehicle, so the time and number of times the driver views the displayed image increases, which may interfere with visual confirmation of safety in the direction of travel. [Means for solving the problem]
[0006] Therefore, one aspect of the present invention is a warning device having a first image acquisition unit that acquires a first image captured by a far-infrared camera that captures the direction in which the vehicle is traveling, a second image acquisition unit that acquires a second image captured by a visible light camera that captures the same direction as the far-infrared camera, a detection unit that detects an object that is to be warned to the driver of the vehicle from the first image acquired by the first image acquisition unit, a range identification unit that, when the detection unit detects an object that is to be warned, identifies a first object range that is a range indicating the detected object in the first image and identifies a second object range in the second image that corresponds to the first object range, and a display control unit that displays the second image depicting the second object range on a display unit that is visible to the driver of the vehicle.
[0007] Another aspect of the present invention is a warning method in which a warning device executes the following steps: a first image acquisition step of acquiring a first image captured by a far-infrared camera that captures the direction in which the vehicle is traveling; a second image acquisition step of acquiring a second image captured by a visible light camera that captures the same direction as the far-infrared camera; a detection step of detecting an object that is to be warned to the driver of the vehicle from the acquired first image; a range identification step of, when an object that is to be warned is detected, identifying a first object range that is a range indicating the detected object in the first image and identifying a second object range in the second image that corresponds to the first object range; and a display step of displaying the second image depicting the second object range on a display unit that is visible to the driver of the vehicle. Effect of the Invention
[0008] According to the present invention, it is possible to allow a driver of a vehicle to appropriately recognize a pedestrian or the like detected from an image captured by a far-infrared camera. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram of a warning device according to a first embodiment. [Diagram 2] 4 is a diagram for explaining the difference between the imaging range of a visible light camera and the imaging range of a far-infrared camera in the warning device according to the first embodiment. FIG. [Diagram 3] FIG. 2 is a diagram illustrating a warning display in the warning device according to the first embodiment. [Figure 4] FIG. 2 is a diagram illustrating the operation of the warning device according to the first embodiment. [Diagram 5] 6A to 6C are diagrams illustrating another mode of operation of the warning device according to the first embodiment. [Figure 6] FIG. 11 is a block diagram of a warning device according to a second embodiment. [Figure 7] FIG. 11 is a diagram for explaining the operation of the warning device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. In each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted as necessary.
[0011] Furthermore, each configuration of the functional blocks described below may be configured by hardware or software, or both, and may be configured by one piece of hardware or software, or may be configured by multiple pieces of hardware or software. Each function (each process) may be realized by a calculation unit (e.g., a computer) having a CPU, memory, etc. For example, a program for performing the creation method in the embodiment may be stored in a storage device, and each function may be realized by executing the program stored in the storage device with a CPU.
[0012] These programs can be stored and provided to a computer using various types of non-transitory computer readable media. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., hard disk drives), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (random access memories)). The programs may also be provided to a computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer readable media can provide the programs to a computer via wired communication paths such as electrical wires and optical fibers, or wireless communication paths.
[0013] First embodiment The warning device 1 according to the first embodiment warns the driver of the vehicle when a pedestrian or the like is detected from an image captured by a far-infrared camera by displaying on a display screen the image captured by the far-infrared camera and a frame line indicating the detected pedestrian or the like.
[0014] FIG. 1 shows a block diagram of the warning device 1 according to the first embodiment. As shown in FIG. 1, the warning device 1 according to the first embodiment includes an alarm control device 10, a far-infrared camera 11, a visible light camera 12, a recognition dictionary storage unit 13, and a display unit 14. The alarm control device 10 is realized by a calculation device capable of executing a program, such as a microcontroller unit (MCU) or a CPU. The microcontroller unit may include multiple calculation units, and various processes in a program may be performed in parallel by the multiple calculation units. Details of the processing blocks realized in the alarm control device 10 will be described later.
[0015] The far-infrared camera 11 is a camera that captures the front of the vehicle, that is, the main traveling direction of the vehicle, and receives far-infrared rays emitted from objects in the capture range, and outputs an image having a shade according to the temperature of the objects in the capture range, that is, a so-called thermal image. The visible light camera 12 is a camera that captures the same direction as the far-infrared camera 11, and outputs an image captured by using visible light in the capture range. The coincidence of the capture directions of the far-infrared camera 11 and the visible light camera 12 means that both cameras are capturing the front of the vehicle. In addition, since the capture angle of the far-infrared camera 11 is often narrower than the capture angle of the visible light camera 12, in such a case, the range of the capture angle of the far-infrared camera 11 is included within the range of the capture angle of the visible light camera 12.
[0016] The recognition dictionary storage unit 13 stores a recognition dictionary that has been machine-learned to learn features such as the shape of an object to be detected by the detection unit 23 described below. The recognition dictionary storage unit 13 is configured, for example, with a non-volatile storage device such as a flash memory. The display unit 14 displays an image captured by at least one of the far-infrared camera 11 and the visible light camera 12 to the driver of the vehicle. The display unit 14 can be, for example, a display installed on the dashboard of the vehicle.
[0017] The alarm control device 10 has a first image acquisition unit 21, a second image acquisition unit 22, a detection unit 23, a range identification unit 24, and a display control unit 25. The processing blocks in the alarm control device 10 are processing blocks realized by a program, or processing blocks implemented as dedicated hardware. Furthermore, the first image acquisition unit 21, the second image acquisition unit 22, the detection unit 23, the range identification unit 24, and the display control unit 25 are configured to be able to share information with each other through processing in the alarm control device 10 or a bus.
[0018] The first image acquisition unit 21 acquires a first image captured by the far-infrared camera 11 that captures the traveling direction of the vehicle. The second image acquisition unit 22 acquires a second image captured by the visible light camera 12. The detection unit 23 detects an object that is a target of warning the driver of the vehicle from the first image acquired by the first image acquisition unit. The object that is a target of warning the driver of the vehicle is, for example, a person such as a pedestrian, a bicycle ridden by a person, an animal, or other object that is detected based on a difference in temperature distribution from the surrounding temperature. In addition, the object that is a target of warning the driver of the vehicle may be, among the above-mentioned objects, an object that is present in front of the traveling direction of the vehicle and is located at a position that requires attention while the vehicle is traveling.
[0019] When the detection unit 23 detects an object to be warned, the range identification unit 24 identifies a first object range, which is a range showing the detected object in the first image, and identifies a second object range in the second image corresponding to the first object range. The first object range identified by the range identification unit 24 is a range surrounding the object detected in the first image, and is indicated by, for example, a rectangular frame line. The rectangular frame line is a frame line surrounding the entire outer shape of the detected object, and can indicate the position and size of the object on the display screen in addition to the presence of the object on the display screen. For example, when the detected object is a person, the first object range is indicated by a rectangular frame line with an aspect ratio of 2:1 surrounding the detected person. The second object range identified by the range identification unit 24 is a range showing the same position in the second image as the first object range identified in the first image, and is indicated by, for example, a frame line having the same shape as the frame line showing the first object range. The first object range and the second object range are not limited to the above-mentioned form, and various forms are possible.
[0020] The display control unit 25 causes the second image, in which a frame line or the like indicating the second object range is drawn, to be displayed on the display unit 14, where it is visible to the driver of the vehicle. Furthermore, when displaying the second image, the display control unit 25 preferably causes the second object range to be drawn in the second image with reduced saturation or contrast and displays it on the display unit 14. By reducing the saturation or contrast of the second image in this manner, the frame line or the like indicating the second object range stands out against the display of the second image, allowing the driver of the vehicle to properly confirm the position of the second object.
[0021] Here, the difference in the shooting range between the far-infrared camera 11 and the visible light camera 12 will be described. FIG. 2 shows a diagram for explaining the difference between the shooting range of the visible light camera and the shooting range of the far-infrared camera in the warning device 1 according to the first embodiment. As shown in FIG. 2, in the warning device 1 according to the first embodiment, the shooting ranges of the far-infrared camera 11 and the visible light camera 12 are the traveling direction of the vehicle, and the cameras shoot in the same direction. Also, in the warning device 1, the far-infrared camera shooting range IR shot by the far-infrared camera 11 is narrower than the visible light camera shooting range OP shot by the visible light camera 12. In the example shown in FIG. 2, the center of the visible light camera shooting range OP and the center of the far-infrared camera shooting range IR are set to coincide with each other. The visible light camera shooting range OP may be the shooting angle of the visible light camera 12, and the far-infrared camera shooting range IR may be the shooting angle of the far-infrared camera 11. The coordinates of the visible light camera shooting range OP are associated with the coordinates of the far-infrared camera shooting range IR in a range that coincides with the far-infrared camera shooting range IR. Therefore, the second object range in the second image can be identified based on the first object range identified in the first image.
[0022] Next, the operation of the warning device 1 according to the first embodiment will be described. FIG. 3 shows a diagram for explaining a warning display in the warning device 1 according to the first embodiment. As shown in FIG. 3, when a detection target object OBJ is detected in the far-infrared camera shooting range IR, the warning device 1 according to the first embodiment specifies a first object range OBJ_R1 and specifies a second object range OBJ_R2 at a position corresponding to the first object range OBJ_R1 in the visible light camera shooting range OP. Then, the second object range OBJ_R2 is displayed clearly in the second image, which is an image of the visible light camera shooting range OP, thereby warning the driver of the vehicle that the object OBJ is present.
[0023] The above operation will be described using a flowchart. FIG. 4 is a diagram for explaining the operation of the warning device 1 according to the first embodiment. As shown in FIG. 4, when the warning device 1 according to the first embodiment starts its operation, it starts shooting with the far-infrared camera 11 and the visible light camera 12 (step S1). The operation of step S1 starts when the power of the engine of the vehicle on which the warning device 1 is mounted is turned on. In addition, when the object detection by the far-infrared camera 11 is performed when the illuminance around the vehicle is low, for example, at night or inside a tunnel, it starts when the illuminance becomes low enough to indicate night or inside a tunnel. After or at the same time as the start of step S1, the warning device 1 starts detecting the object OBJ in the far-infrared camera shooting range IR from the first image acquired by the far-infrared camera 11 and the first image acquisition unit 21 by the detection unit 23 (step S2). In addition, the warning device 1 starts displaying the second image (for example, the image of the visible light camera shooting range OP) acquired by the visible light camera 12 and the second image acquisition unit 22 on the display unit 14 by the display control unit 25 (step S3). The second image displayed on the display unit 14 may display a portion from the visible light camera shooting range OP to the far-infrared camera shooting range IR.
[0024] Then, when the detection unit 23 detects the presence of the object OBJ in the far-infrared camera shooting range IR (YES in step S4), the range identification unit 24 identifies the position of the second object range OBJ_R2 corresponding to the first object range OBJ_R1 (step S5). Then, the warning device 1 causes the display control unit 25 to display the second image showing the second object range OBJ_R2 on the display unit 14 (step S6). This second image showing the second object range OBJ_R2 is continued until a warning termination condition is satisfied, in which the state requiring a warning, such as the object OBJ no longer being present in the far-infrared camera shooting range IR, is resolved (step S7). Then, after the warning termination condition is satisfied, the display control unit 25 displays the second image not including the second object range OBJ_R2 on the display unit 14 (step S8).
[0025] Then, the warning device 1 according to the first embodiment repeats the process of steps S4 to S8 until the operation, such as stopping the engine, is terminated. If the object OBJ is not detected in the far-infrared camera shooting range IR in step S4, the warning device 1 according to the first embodiment continues the detection process of the object OBJ without executing steps S5 to S8 until the object OBJ is detected in the far-infrared camera shooting range IR (NO in step S4, NO in step S9).
[0026] The operation of the warning device 1 according to the first embodiment may take another form. FIG. 5 shows a diagram for explaining another form of the operation of the warning device 1 according to the first embodiment. As shown in FIG. 5, the other form of the operation of the warning device 1 according to the first embodiment is obtained by changing steps S3 and S8 for displaying the second video to steps S10 and S11 for displaying the first video in the flowchart shown in FIG. 4. That is, in the other form of the operation of the warning device 1 according to the first embodiment, the first video is displayed on the display unit 14 in a state in which the object OBJ is not yet detected. In this way, in the warning device 1 according to the first embodiment, whether to display the first video or the second video on the display unit 14 in a period in which the object OBJ is not yet detected can be selected according to the specifications.
[0027] As described above, in the warning device 1 according to the first embodiment, when an object OBJ to be warned about is detected in the far-infrared camera shooting range IR, the second object range OBJ_R2 indicating the location of the object OBJ is superimposed on the second image, thereby presenting the warning to be notified to the driver in an easily understandable manner. Even if the driver of the vehicle cannot visually confirm the object to be warned about by looking outside the vehicle, the driver of the vehicle can appropriately grasp the position of the object to be warned about because the second object range OBJ_R2 is drawn in the second image similar to the scenery outside the vehicle. In addition, by allowing the driver to appropriately grasp the position of the object to be warned about, the time and number of times to look at the display image can be reduced, and the driver's attention can be directed more to looking at the traveling direction.
[0028] Embodiment 2 In the second embodiment, a warning device 2 will be described which is another embodiment of the warning device 1 according to the first embodiment. In the description of the second embodiment, the same components as those in the first embodiment will be denoted by the same reference numerals as those in the first embodiment, and the description thereof will be omitted.
[0029] Fig. 6 shows a block diagram of the warning device 2 according to the second embodiment. As shown in Fig. 6, the warning device 2 according to the second embodiment is obtained by adding an in-vehicle camera 15 to the warning device 1 according to the first embodiment and replacing the alarm control device 10 with an alarm control device 30. The alarm control device 30 is obtained by adding a third image acquisition unit 26 and a gaze detection unit 27 to the alarm control device 10.
[0030] The in-vehicle camera 15 photographs the driver in the vehicle. In particular, the in-vehicle camera 15 is installed to photograph the driver so that the driver's line of sight toward the display unit 14 can be seen. The third image acquisition unit 26 acquires the image captured by the in-vehicle camera 15 and transmits it to the line of sight detection unit 27. The line of sight detection unit 27 detects the direction of the driver's line of sight. Then, when the detection unit 23 detects an object that is the subject of a warning, the display control unit 25 displays the first image in which the first object range is drawn on the display unit, and when the driver of the vehicle looks at the display unit 14 for a predetermined time or more or looks back, the display control unit 25 displays the second image in which the second object range is drawn. The predetermined time is, for example, 3 seconds. Also, when looking back, it is when the driver of the vehicle looks at the display unit 14 on which the first image in which the first object range is drawn is displayed, looks ahead of the vehicle, and then looks at the display unit 14 on which the first image in which the first object range is drawn again within, for example, 3 seconds.
[0031] The case where the driver of the vehicle looks at the display unit 14 displaying the first image in which the first object range is drawn for a predetermined period of time or longer means that the object to be warned about cannot be visually confirmed outside the vehicle even though the first object range indicating the object to be warned about is displayed in the first image displayed on the display unit 14. The same applies when the driver of the vehicle looks back at the display unit 14 displaying the first image in which the first object range is drawn.
[0032] Here, Fig. 7 shows a diagram for explaining the operation of the warning device 2 according to the second embodiment. Fig. 7 explains the operation of the warning device 2 in the case where the second video in which the second object range is drawn is displayed only when the driver looks back at the display unit 14.
[0033] As shown in Fig. 7, the operation of the warning device 2 according to the second embodiment is the same as the operation of the warning device 1 according to the first embodiment shown in Fig. 5, except that an operation of step S21 is added. Step S21 is performed after the detection unit 23 detects the object OBJ in step S4. In the warning device 2, when the object OBJ is not detected, the first image captured by the far-infrared camera 11 is displayed on the display unit 14, and even if the second object range OBJ_R2 is not displayed, if the driver can recognize the object OBJ, the display of the first image is continued (step S21) without displaying the second image including the second object range OBJ_R2 (performing the processing of steps S5 to S7).
[0034] From the above explanation, in the warning device 2 of the second embodiment, when it is determined that the driver is in a state in which it is necessary to check the object OBJ using the second image, the warning device 2 can display the second image in which the second object range is drawn.
[0035] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit and scope of the present invention. [Explanation of symbols]
[0036] 1, 2 Warning device 10, 30 Alarm control device 11 Far-infrared camera 12 Visible light camera 13 Recognition dictionary storage unit 14 Display section 15 In-vehicle camera 21 First image acquisition unit 22 Second Image Acquisition Unit 23 Detection unit 24 Range specification section 25 Display control section 26 3rd Video Acquisition Unit 27 Gaze detection unit OP visible light camera shooting range IR Far-infrared camera shooting range OBJ Object OBJ_R1 First object range OBJ_R2 Second object range
Claims
1. a first image acquisition unit that acquires a first image captured by a far-infrared camera that captures an image in the traveling direction of the vehicle; A second image acquisition unit that acquires a second image captured by a visible light camera that captures an image in the same direction as the far-infrared camera; a detection unit that detects an object that is to be warned to a driver of the vehicle from the first image acquired by the first image acquisition unit; a range specifying unit that, when the detection unit detects an object that is a target of a warning from the first image when the illuminance around the vehicle is low, specifies a first object range that is a range indicating the detected object in the first image, and specifies a second object range in the second image that corresponds to the first object range; A gaze detection unit that detects a gaze direction of a driver of the vehicle; a display control unit that, when the detection unit detects an object that is a target of a warning, causes the first image in which the first object range is drawn to be displayed on a display unit visible to a driver of the vehicle, and, when the detection unit detects an object that is a target of a warning and the driver of the vehicle looks at the display unit for a predetermined period of time or more or looks back at the display unit, causes the second image in which a frame line indicating the second object range is drawn to be displayed on the display unit; A warning device having:
2. the display control unit causes the display unit to display a second image in which a frame line indicating the second object range is drawn during a period in which a warning based on the detected object is issued.
2. The warning device of claim 1.
3. the display control unit causes a frame line indicating the second object range to be drawn on the second image with reduced saturation or contrast, and causes the second image to be displayed on the display unit.
3. A warning device according to claim 1 or 2.
4. A first image acquisition step of acquiring a first image captured by a far-infrared camera that captures an image in the traveling direction of the vehicle; a second image acquisition step of acquiring a second image captured by a visible light camera that captures an image in the same direction as the far-infrared camera; a detection step of detecting an object to be warned to a driver of the vehicle from the acquired first image; a range specifying step of, when an object that is a target of a warning is detected from the first image when the illuminance around the vehicle is low, specifying a first object range that is a range indicating the detected object in the first image, and specifying a second object range in the second image that corresponds to the first object range; a gaze detection step of detecting a gaze direction of a driver of the vehicle; a display step of displaying the first image, in which the first object range is drawn, on a display unit visible to a driver of the vehicle when an object that is a target of a warning is detected in the detection step, and displaying the second image, in which a frame line indicating the second object range is drawn, on the display unit when an object that is a target of a warning is detected in the detection step and when the driver of the vehicle looks at the display unit for a predetermined period of time or more or looks back at the display unit; The warning device performs the warning method.
Citation Information
Patent Citations
Head-mounted display device for vehicle
JP1998206787A
Object detection system
JP2012131491A
Vehicle-use image-capturing device
JP2014197727A
Recognition processing device, recognition processing method, and recognition processing program
JP2020027380A
Periphery monitoring device
JP2020043400A
