Warning device and warning method

Abstract

In the past, there has been a problem that a driver of a vehicle cannot properly grasp a detected pedestrian or the like. A warning device (1) includes a first image acquisition unit (21) that acquires a first image captured by a far-infrared camera (11) that captures an advancing direction of a vehicle, a second image acquisition unit (22) that acquires a second image captured by a visible light camera (12) that captures a direction identical to the far-infrared camera (11), a detection unit (23) that detects a target that is a warning object from the first image acquired by the first image acquisition unit, a range determination unit (24) that determines a first target range in the first image when the detection unit (23) detects the target that is the warning object, and determines a second target range that is a range in the second image that represents the detected target corresponding to the first target range, and a display control unit (25) that causes a display unit (14) that a driver of the vehicle can visually recognize to display the second image that depicts the second target range.

Description

Technical Field

[0001] This invention relates to a warning device and method, for example, using an infrared camera to detect pedestrians or other objects present in the direction of a vehicle's travel and to issue a warning to the vehicle's driver. Background Technology

[0002] In recent years, in order to ensure the safety of a vehicle's direction of travel, far-infrared cameras have been used to detect objects that are difficult to detect with visible light cameras. Patent Document 1 discloses an example of an object detection technology using such a far-infrared camera.

[0003] The identification processing apparatus described in Patent Document 1 includes: an image acquisition unit that acquires first image data obtained by a far-infrared camera capturing images of the area around its own vehicle; an other vehicle detection unit that detects other vehicles parked around its own vehicle; a heat detection unit that detects heat dissipation based on the movement of other vehicles by analyzing the heat distribution of the areas corresponding to other vehicles in the first image data; and a person detection unit that, when heat dissipation is detected, prioritizes person recognition to detect people near other vehicles.

[0004] Existing technical documents

[0005] Patent documents

[0006] Patent Document 1: Japanese Patent Application Publication No. 2020-027380 Summary of the Invention

[0007] However, far-infrared cameras detect pedestrians by analyzing the distribution of heat emitted by them. Therefore, drivers sometimes cannot visually confirm pedestrians detected by the far-infrared camera by looking outside the vehicle. In such situations, although the infrared camera detects pedestrians, the driver cannot visually confirm their location, thus increasing the time and frequency of visual image viewing, which can sometimes hinder visual confirmation of the direction of travel.

[0008] Therefore, one embodiment of this invention is a warning device comprising: a first image acquisition unit that acquires a first image captured by a far-infrared camera, the far-infrared camera capturing images in the direction of the vehicle's travel; a second image acquisition unit that acquires a second image captured by a visible light camera, the visible light camera capturing images in the same direction as the far-infrared camera; a detection unit that detects a target that is a warning object for the driver of the vehicle from the first image acquired by the first image acquisition unit; a range determination unit that, when the detection unit detects a target that is a warning object, determines a first target range and determines a second target range in the second image corresponding to the first target range, the first target range being the range of the target detected in the first image; and a display control unit that enables the driver of the vehicle to visually view the second image depicting the second target range.

[0009] Another embodiment of this invention is a warning method executed by a warning device, comprising: a first image acquisition step, acquiring a first image captured by a far-infrared camera, the far-infrared camera capturing images in the direction of the vehicle's travel; a second image acquisition step, acquiring a second image captured by a visible light camera, the visible light camera capturing images in the same direction as the far-infrared camera; a detection step, detecting from the acquired first image a target that is a warning object for the driver of the vehicle; a range determination step, when a target that is a warning object is detected, determining a first target range and determining a second target range in the second image corresponding to the first target range, the first target range representing the range of the target detected in the first image; and a display step, displaying the second image depicting the second target range on a display unit that is visible to the driver of the vehicle.

[0010] According to this embodiment, the driver of the vehicle can properly grasp pedestrians and other objects detected from the images of the far-infrared camera. Attached Figure Description

[0011] Figure 1 This is a block diagram of the warning device according to the first embodiment.

[0012] Figure 2 This is a diagram illustrating the difference between the shooting range of the visible light camera and the shooting range of the far-infrared camera in the warning device according to the first embodiment.

[0013] Figure 3 This is a diagram illustrating the warning display in the warning device according to the first embodiment.

[0014] Figure 4 This is a diagram illustrating the operation of the warning device according to the first embodiment.

[0015] Figure 5 This diagram illustrates another way of operating the warning device according to the first embodiment.

[0016] Figure 6 This is a block diagram of a warning device according to the second embodiment.

[0017] Figure 7 This is a diagram illustrating the operation of the warning device according to the second embodiment. Detailed Implementation

[0018] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. For clarity, the following description and drawings have been appropriately omitted and simplified. In the drawings, the same elements are labeled with the same reference numerals, and repeated descriptions are omitted as necessary.

[0019] Furthermore, the structures of the functional blocks described below are composed of hardware or software, or both; they can be composed of one piece of hardware or software, or multiple pieces of hardware or software. Each function (each process) can also be implemented by an arithmetic unit (e.g., a computer) having a CPU, memory, etc. For example, a program for performing the manufacturing method in the embodiments can be stored in a storage device, and each function can be implemented by the CPU executing the program stored in the storage device.

[0020] These programs can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., hard disk drives), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash memory ROMs, and RAMs (random access memory). Additionally, programs can also be provided to a computer using various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. Transient computer-readable media can provide programs to a computer via wired communication paths such as wires and optical fibers, or via wireless communication paths.

[0021] (First Implementation)

[0022] The warning device 1 according to the first embodiment warns the driver of a vehicle by displaying the image captured by the infrared camera and a frame indicating the detected pedestrian in the image captured by the infrared camera on a display screen.

[0023] Figure 1 A block diagram of the warning device 1 according to the first embodiment is shown. (As shown) Figure 1 As shown, the warning device 1 according to the first embodiment includes an alarm control device 10, an infrared camera 11, a visible light camera 12, a recognition dictionary storage unit 13, and a display unit 14. The alarm control device 10 is implemented, for example, by a computing device capable of executing programs, such as a microcontroller unit (MCU) or a CPU. Furthermore, the microcontroller unit may include multiple computing units, and various processes in the program can be executed in parallel by the multiple computing units. Details regarding the processing modules implemented within the alarm control device 10 will be described later.

[0024] The far-infrared camera 11 is a camera that captures images of the front of the vehicle, i.e., the main direction of travel of the vehicle. It receives far-infrared radiation emitted from objects within its shooting range and outputs an image with intensity corresponding to the temperature of the objects within the shooting range, i.e., a thermal image. The visible light camera 12 is a camera that captures images in the same direction as the far-infrared camera 11 and outputs an image obtained by capturing the shooting range using visible light. The fact that the far-infrared camera 11 and the visible light camera 12 capture images of the front of the vehicle means that both cameras capture images of the front of the vehicle. In addition, in most cases, the shooting field of view of the far-infrared camera 11 is narrower than that of the visible light camera 12. Therefore, in such cases, the shooting field of view of the far-infrared camera 11 is included within the shooting field of view of the visible light camera 12.

[0025] The recognition dictionary storage unit 13 stores a recognition dictionary, which is obtained by machine learning using features such as the target shape of the detection object detected in the detection unit 23 (described later). The recognition dictionary storage unit 13 is, for example, a non-volatile storage device such as flash memory. The display unit 14 displays images captured by at least one of the far-infrared camera 11 and the visible light camera 12 to the driver of the vehicle. The display unit 14 may be, for example, a display installed on the vehicle's dashboard.

[0026] The alarm control device 10 includes a first image acquisition unit 21, a second image acquisition unit 22, a detection unit 23, a range determination unit 24, and a display control unit 25. The processing module within the alarm control device 10 is either a program-implemented processing module or a processing module installed as dedicated hardware. Furthermore, the first image acquisition unit 21, the second image acquisition unit 22, the detection unit 23, the range determination unit 24, and the display control unit 25 are configured to perform processing within the alarm control device 10 or to cooperate with each other via a bus.

[0027] The first image acquisition unit 21 acquires a first image captured by the far-infrared camera 11, which measures the direction of travel of the vehicle. The second image acquisition unit 22 acquires a second image captured by the visible light camera 12. The detection unit 23 detects targets that are warning objects for the driver of the vehicle from the first image acquired by the first image acquisition unit. Targets that are warning objects for the driver of the vehicle include, for example, pedestrians, bicycles ridden by people, animals, etc., which are detected based on the difference in temperature distribution between them and the surrounding temperature. In addition, targets that are warning objects for the driver of the vehicle can also be objects that are located in the direction of travel of the vehicle, i.e., in front of the vehicle, and in a position that requires attention while the vehicle is moving.

[0028] When the detection unit 23 detects a target that becomes a warning object, the range determination unit 24 determines a first target range and a second target range in a second image corresponding to the first target range. The first target range represents the range of the detected target in the first image. The first target range determined by the range determination unit 24 is the range surrounding the target detected in the first image, and is represented, for example, by a rectangular frame. The rectangular frame is a frame that surrounds the overall shape of the detected target, and in addition to showing the presence of the target on the screen, it can also indicate its position and size on the screen. For example, if the detected target is a person, the first target range is represented by a rectangular frame with an aspect ratio of 2:1 surrounding the detected person. The second target range determined by the range determination unit 24 is the range in the second image that represents the same position as the first target range determined in the first image, and is represented, for example, by a frame with the same shape as the frame representing the first target range. The first target range and the second target range are not limited to the above-described manner, and various methods can be implemented.

[0029] The display control unit 25 displays a second image, including outlines indicating the range of a second target, on the display unit 14, which is visible to the vehicle driver. Furthermore, when displaying the second image, the display control unit 25 preferably depicts the range of the second target within a second image with reduced saturation or contrast and displays it on the display unit 14. By reducing the saturation or contrast of the second image in this way, the outlines indicating the range of the second target become more prominent relative to the display of the second image, thus allowing the vehicle driver to properly confirm the position of the second target.

[0030] Here, the difference in the shooting range between the far-infrared camera 11 and the visible light camera 12 will be explained. Therefore, Figure 2 A diagram illustrating the difference between the shooting range of the visible light camera and the shooting range of the far-infrared camera in the warning device 1 according to the first embodiment is shown. Figure 2 As shown, in the warning device 1 according to the first embodiment, the shooting range of the far-infrared camera 11 and the visible light camera 12 is the direction of vehicle travel, and they shoot in the same direction. Furthermore, in the warning device 1, the far-infrared camera shooting range IR of the far-infrared camera 11 is narrower than the visible light camera shooting range OP of the visible light camera 12. Figure 2 In the example shown, the center of the visible light camera's shooting range OP is set to coincide with the center of the far-infrared camera's shooting range IR. The visible light camera's shooting range OP can be the shooting field of view of the visible light camera 12, and the far-infrared camera's shooting range IR can be the shooting field of view of the far-infrared camera 11. The coordinates of the visible light camera's shooting range OP correspond to the coordinates of the far-infrared camera's shooting range IR within a range that coincides with the far-infrared camera's shooting range IR. Therefore, a second target range in the second image can be determined based on the first target range determined in the first image.

[0031] Next, the operation of the warning device 1 according to the first embodiment will be explained. Therefore, in Figure 3 The diagram illustrates the warning display in the warning device 1 according to the first embodiment. (See diagram below.) Figure 3 As shown, in the first embodiment, when the warning device 1 detects a target OBJ within the infrared camera's shooting range IR, it determines a first target range OBJ_R1 and a second target range OBJ_R2 corresponding to the first target range OBJ_R1 within the visible light camera's shooting range OP. Then, by displaying the second target range OBJ_R2 explicitly in a second image that is part of the visible light camera's shooting range OP, the driver of the vehicle is warned of the presence of a target OBJ.

[0032] Use a flowchart to illustrate the above actions. Figure 4A diagram illustrating the operation of the warning device 1 according to the first embodiment is shown. Figure 4 As shown, when the warning device 1 according to the first embodiment is activated, it begins to capture images using an infrared camera 11 and a visible light camera 12 (step S1). Step S1 begins when the engine or other power source of the vehicle equipped with the warning device 1 is turned on. Additionally, when the brightness around the vehicle is low, such as at night or in a tunnel, the infrared camera 11 performs target detection. At this time, detection begins when the brightness is reduced to a level indicating nighttime or a tunnel. After or simultaneously with the start of step S1, the warning device 1 begins to detect the target OBJ within the infrared camera's field of view (IR) from the first image acquired by the infrared camera 11 and the first image acquisition unit 21 via the detection unit 23 (step S2). Furthermore, the warning device 1 begins to display the second image (e.g., an image within the visible light camera's field of view (OP)) acquired by the visible light camera 12 and the second image acquisition unit 22 on the display unit 14 via the display control unit 25 (step S3). The second image displayed on the display unit 14 can also display a portion of the visible light camera's field of view (OP) that corresponds to the infrared camera's field of view (IR).

[0033] Then, if the detection unit 23 detects the presence of a target OBJ within the infrared camera's IR field of view ("Yes" in step S4), the range determination unit 24 determines the position of the second target range OBJ_R2 corresponding to the first target range OBJ_R1 (step S5). Then, the warning device 1 displays a second image representing the second target range OBJ_R2 on the display unit 14 via the display control unit 25 (step S6). The second image representing the second target range OBJ_R2 continues until a warning termination condition is met, which is a condition where the warning is eliminated, such as the target OBJ no longer existing within the infrared camera's IR field of view (step S7). Then, after the warning termination condition is met, the display control unit 25 displays a second image that does not include the second target range OBJ_R2 on the display unit 14 (step S8).

[0034] Then, the warning device 1 according to the first embodiment repeatedly performs the processing of steps S4-S8 until the action such as engine stoppage becomes the end state. In addition, if the warning device 1 according to the first embodiment does not detect the target OBJ within the shooting range IR of the far-infrared camera in step S4, it does not execute steps S5-S8 but continues the target OBJ detection processing until the target OBJ is detected within the shooting range IR of the far-infrared camera ("No" in step S4, "No" in step S9).

[0035] Furthermore, the operation of the warning device 1 according to the first embodiment can also be considered in other ways. Therefore, Figure 5A diagram illustrating another mode of operation of the warning device 1 according to the first embodiment is shown. (See diagram for example.) Figure 5 As shown, another way in which the warning device 1 according to the first embodiment operates is, in Figure 4 In the flowchart shown, steps S3 and S8 for displaying the second image are changed to steps S10 and S11 for displaying the first image. That is, in other ways of operating the warning device 1 according to the first embodiment, the first image is displayed on the display unit 14 when no target OBJ is detected. In this way, in the warning device 1 according to the first embodiment, during the period when no target OBJ is detected, it is possible to select whether to display the first image or the second image on the display unit 14 according to the specifications.

[0036] According to the above description, in the warning device 1 according to the first embodiment, when a target OBJ of a warning object is detected in the infrared camera's shooting range IR, a second target range OBJ_R2 indicating the location of the target OBJ is overlaid on a second image, thus providing the driver with an easily understandable warning. Even when the target of the warning object cannot be confirmed by visually looking outside the vehicle, since the second target range OBJ_R2 is depicted in a second image identical to the scenery outside the vehicle, the driver can properly grasp the target location of the warning object. Furthermore, by enabling the driver to properly grasp the target location of the warning object, the time and number of times the image is visually displayed can be reduced, allowing the driver's attention to be more focused on visual observation in the direction of travel.

[0037] (Second Implementation)

[0038] In the second embodiment, a warning device 2, which is a different type of warning device 1 as described in the first embodiment, will be explained. Furthermore, in the description of the second embodiment, structural elements with the same structural uses as those described in the first embodiment are labeled with the same reference numerals as those in the first embodiment, and their descriptions are omitted.

[0039] Figure 6 This is a block diagram illustrating the warning device 2 according to the second embodiment. Figure 6 As shown, the warning device 2 according to the second embodiment adds an in-car camera 15 to the warning device 1 according to the first embodiment, and replaces the alarm control device 10 with an alarm control device 30. The alarm control device 30 is a device that adds a third image acquisition unit 26 and a line-of-sight detection unit 27 to the alarm control device 10.

[0040] An in-car camera 15 captures images of the driver inside the vehicle. Specifically, the in-car camera 15 is configured to capture the driver's gaze so that the driver is aware of their line of sight towards the display unit 14. A third image acquisition unit 26 acquires the images captured by the in-car camera 15 and transmits them to a gaze detection unit 27. The gaze detection unit 27 detects the direction of the driver's gaze. Furthermore, when the detection unit 23 detects a target that is a warning object, the display control unit 25 causes the display unit to display a first image depicting a first target area. If the driver views the display unit 14 for a predetermined time or reviews the display unit, a second image depicting a second target area is displayed. The predetermined time is, for example, 3 seconds. Reviewing the display unit refers to situations where, after viewing the display unit 14 displaying the first image depicting the first target area, the driver looks ahead of the vehicle and then, for example, within 3 seconds, views the display unit 14 displaying the first image depicting the first target area again.

[0041] The situation where the driver of the vehicle views the display unit 14, which shows the first image depicting the first target range, for a predetermined period of time or more refers to a situation where, although the first image displayed on the display unit 14 shows the first target range indicating the target to be warned, the target to be warned cannot be confirmed by visually looking outside the vehicle. The same applies when the driver of the vehicle re-views the display unit 14 showing the first image depicting the first target range.

[0042] Here, Figure 7 A diagram illustrating the operation of the warning device 2 according to the second embodiment is shown. Figure 7 This diagram illustrates the operation of the warning device 2 when a second image depicting the second target area is displayed only when the driver looks back at the display unit 14.

[0043] like Figure 7 As shown, the operation of the warning device 2 in the second embodiment is... Figure 5 The warning device 1 according to the first embodiment shown is obtained by adding step S21 to its operation. Step S21 is performed after the detection unit 23 detects the target OBJ in step S4. In the warning device 2, when the target OBJ is not detected, the first image captured by the far-infrared camera 11 is displayed on the display unit 14. Even if the second target range OBJ_R2 is not displayed, if the driver can identify the target OBJ, the second image containing the second target range OBJ_R2 is not played (processing steps S5-S7 is performed), and the display of the first image continues (step S21).

[0044] According to the above description, in the warning device 2 according to the second embodiment, when it is determined that the driver needs to confirm the target OBJ based on the second image, a second image depicting the range of the second target can be displayed.

[0045] Furthermore, the present invention is not limited to the above-described embodiments and can be appropriately modified without departing from the spirit of the invention.

[0046] This application claims priority to Japanese Patent Application No. 2021-45681, filed on March 19, 2021, and incorporates the entire contents thereof.

[0047] Symbol explanation:

[0048] 1, 2 Warning devices

[0049] 10, 30 Alarm Control Device

[0050] 11. Far-infrared camera

[0051] 12 Visible light cameras

[0052] 13. Recognition dictionary storage unit

[0053] 14 Display Section

[0054] 15 Cameras inside the carriage

[0055] 21 First Image Acquisition Department

[0056] 22 Second Image Acquisition Unit

[0057] 23. Testing Department

[0058] 24. Scope Determination Department

[0059] 25 Display Control Unit

[0060] 26 Third Image Acquisition Department

[0061] 27. Line of Sight Inspection Department

[0062] OP visible light camera shooting range

[0063] IR far-infrared camera shooting range

[0064] OBJ Target

[0065] OBJ_R1 First target range

[0066] OBJ_R2 Second Target Range

Claims

1. A warning device, comprising: The first image acquisition unit acquires the first image captured by the far-infrared camera, which captures the vehicle's direction of travel. The second image acquisition unit acquires a second image captured by a visible light camera, wherein the visible light camera captures an image in the same direction as the shooting direction of the far-infrared camera. The detection unit detects targets that are warning objects for the driver of the vehicle from the first image acquired by the first image acquisition unit. The range determination unit determines a first target range when the detection unit detects a target that is a warning object, and determines a second target range in a second image that corresponds to the first target range, wherein the first target range represents the range of the target detected in the first image; as well as The display control unit enables the driver of the vehicle to select either the first image or the second image for display when the detection unit does not detect a target as a warning object, and enables the display unit to display a second image depicting the range of the second target when the detection unit detects a target as a warning object.

2. The warning device as claimed in claim 1, wherein, During the issuance of a warning based on a detected target, the display control unit causes the display unit to display a second image depicting the range of the second target.

3. The warning device as claimed in claim 1 further comprises a gaze detection unit that detects the direction of the driver's gaze in the vehicle. When the detection unit detects a target that is a warning target, the display control unit causes the display unit to display the first image depicting the first target range. When the detection unit detects a target that is a warning target and the driver of the vehicle has viewed the display unit for a predetermined time or has looked back at the display unit, the display control unit causes the display unit to display the second image depicting the second target range.

4. The warning device according to any one of claims 1 to 3, wherein, The display control unit depicts the second target range in a second image with reduced chroma or contrast and displays it on the display unit.

5. A warning method, comprising: The first image acquisition step involves acquiring a first image captured by a far-infrared camera, wherein the far-infrared camera captures the vehicle's direction of travel. The second image acquisition step involves acquiring a second image captured by a visible light camera, wherein the visible light camera captures an image in the same direction as the shooting direction of the far-infrared camera. The detection step involves detecting targets from the acquired first image that would be a warning target for the driver of the vehicle. The range determination step involves determining a first target range when a target that becomes a warning object is detected, and determining a second target range in the second image that corresponds to the first target range. The first target range represents the range of the target detected in the first image. as well as The display steps include enabling the driver of the vehicle to select either the first image or the second image for display when no target as a warning object is detected, and enabling the display unit to display a second image depicting the range of the second target when a target as a warning object is detected.

Images