Image processing apparatus and imaging method

The image processing device enhances road visibility in poor conditions by extracting and superimposing infrared-derived auxiliary images on visible light images, addressing the limitations of existing technologies in combining infrared and visible light images for improved road recognition.

JP2024160450AInactive Publication Date: 2024-11-14JVC KENWOOD CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023075441
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-01
Publication Date
2024-11-14
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Existing image processing devices fail to enhance the visibility of roads during poor visibility conditions, such as at night or in fog or snow, by combining visible light and infrared images effectively.

Method used

An image processing device that extracts heat source regions from infrared images, determines specific heat source regions based on speed and shape, and generates auxiliary images to superimpose on visible light images, improving road visibility by enhancing key road features.

Benefits of technology

The device improves road visibility by clearly delineating road edges and obstacles, even in adverse weather conditions, enabling safer driving.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024160450000001_ABST
    Figure 2024160450000001_ABST
Patent Text Reader

Abstract

To provide an image processing apparatus capable of improving visibility of a road on which a vehicle travels.SOLUTION: A heat source region extraction unit 306 extracts a heat source region which is a region having a predetermined temperature or more included in each frame of an infrared image generated by a thermo-camera 2 by capturing an image in front of a vehicle. A specific heat source region determination unit 308 determines the heat source region as a specific heat source region when a moving speed of the heat source region extracted by the heat source region extraction unit 306 is less than a predetermined speed. An auxiliary image generation unit 310 generates an auxiliary image that connects positions to each other a predetermined distance below positions of bottom ends of a plurality of the heat source regions determined as the specific heat source regions. An auxiliary image superimposition unit 311 superimposes the auxiliary image onto each frame of a visible light image generated by a visible light camera 1 by capturing an image in front of the vehicle.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to an image processing device and an imaging method. [Background technology]

[0002] As described in Patent Document 1, an image processing device is known that combines a visible light image captured by a visible light camera and an infrared image captured by an infrared camera and displays the combined image on a display unit to assist driving of a vehicle during poor visibility. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 11-308609 Summary of the Invention [Problem to be solved by the invention]

[0004] At night, or even during the day when fog is present or snow is falling, it may be impossible to recognize the road on which the vehicle is traveling. According to the image processing device described in Patent Document 1, although it is possible to cut out the area of ​​a pedestrian in an infrared image and combine it with a visible light image, it is not possible to improve the visibility of the road on which the vehicle is traveling.

[0005] An object of the present invention is to provide an image processing device and an imaging method that can improve the visibility of a road on which a vehicle is traveling. [Means for solving the problem]

[0006] The present invention provides an image processing device including a heat source area extraction unit that extracts a heat source area, which is an area having a temperature equal to or higher than a predetermined temperature, included in each frame of an infrared image generated by a thermal camera photographing the area in front of a vehicle; a specific heat source area determination unit that determines the heat source area extracted by the heat source area extraction unit to be a specific heat source area if the moving speed of the heat source area is less than a predetermined speed; an auxiliary image generation unit that generates a first auxiliary image that connects positions a predetermined distance below each lower end position of a plurality of heat source areas determined to be specific heat source areas by the specific heat source area determination unit, among the heat source areas included in each frame of the infrared image; and an auxiliary image superimposition unit that superimposes the first auxiliary image on each frame of a visible light image generated by a visible light camera photographing the area in front of the vehicle.

[0007] The present invention provides an imaging method in which a visible light camera photographs the area in front of a vehicle to generate a visible light image, a thermal camera photographs the area in front of the vehicle to generate an infrared image, an image processing unit extracts a heat source area which is an area having a temperature equal to or higher than a predetermined temperature contained in each frame of the infrared image, the image processing unit determines the heat source area to be a specific heat source area if the moving speed of the extracted heat source area is less than a predetermined speed, the image processing unit generates a connected auxiliary image that connects positions a predetermined distance below the positions of each lower end of a plurality of heat source areas which are determined to be the specific heat source areas among the heat source areas contained in each frame of the infrared image, the image processing unit superimposes the connected auxiliary image on each frame of the visible light image to generate a superimposed visible light image, and a display unit displays the superimposed visible light image. Effect of the Invention

[0008] According to the image processing device and the imaging method of the present invention, it is possible to improve the visibility of the road on which a vehicle is traveling. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an image processing apparatus according to an embodiment. [Diagram 2]FIG. 2 is a diagram showing an example of a mounting position of a vehicle imaging device including an image processing device according to an embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of the arrangement of heat source markers used when the image processing device according to an embodiment performs calibration. [Figure 4] FIG. 4 is a diagram showing a visible light image generated by a visible light camera when an image processing device according to an embodiment performs calibration. [Diagram 5] FIG. 5 is a diagram showing an infrared image generated by a thermal camera when an image processing device according to an embodiment performs calibration. [Figure 6] FIG. 6 is a diagram showing a composite image obtained by combining the visible light image shown in FIG. 4 and the infrared light image shown in FIG. [Figure 7] FIG. 7 is a diagram showing a state in which the infrared image is corrected so that the positions of the visible light image and the infrared image are aligned. [Figure 8] FIG. 8 is a diagram showing an example of an actual scene of a road on which a vehicle is traveling. [Figure 9] FIG. 9 is a diagram showing a state in which a snowstorm has occurred in the actual scenery shown in FIG. [Figure 10] FIG. 10 is a diagram showing a visible light image generated by a visible light camera when a vehicle is traveling on the road in the state shown in FIG. [Figure 11] FIG. 11 is a diagram showing a state in which heat source regions having a temperature equal to or higher than a predetermined temperature are extracted from an infrared image generated by a thermo camera when a vehicle is traveling on the road in the state shown in FIG. [Figure 12] FIG. 12 is a diagram showing a vertical auxiliary image and a connected auxiliary image generated based on the position of the bottom end of a specific heat source region among the heat source regions shown in FIG. [Figure 13] FIG. 13 is a diagram showing a superimposed visible light image obtained by superimposing the vertical auxiliary image and the connected auxiliary image shown in FIG. 12 on the visible light image shown in FIG. [Figure 14]FIG. 14 is a diagram showing a vertical auxiliary image and a connected auxiliary image generated based on the position of the bottom end of each specific heat source region when six specific heat source regions exist within a frame. [Figure 15] FIG. 15 is a diagram showing a state in which the area surrounded by the connected auxiliary image in FIG. 14 is filled with the fill-in auxiliary image. [Figure 16] FIG. 16 is a diagram showing a vertical auxiliary image and a connected auxiliary image generated based on the position of the bottom end of each specific heat source region when four specific heat source regions exist within a frame. [Figure 17] FIG. 17 is a diagram showing a state in which the area surrounded by the connected auxiliary image in FIG. 64 is filled with the fill-in auxiliary image. [Figure 18A] FIG. 18A is a partial flowchart showing the operation of an image processing device according to an embodiment and the processing of an imaging method according to an embodiment. [Figure 18B] FIG. 18B is a partial flowchart following FIG. 18A, showing the operation of the image processing device according to an embodiment and the processing of the imaging method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An image processing device and an imaging method according to an embodiment will be described below with reference to the accompanying drawings. FIG. 1 shows an example of the configuration of a vehicle imaging device 100 including an image processing device according to an embodiment. In FIG. 1, the vehicle imaging device 100 includes a visible light camera 1, a thermal camera 2, an image processing unit 3, an operation unit 4, and a display unit 5. The image processing unit 3 constitutes an image processing device according to an embodiment. The display unit 5 may be an external configuration of the vehicle imaging device 100. The operation unit 4 may be an operation button provided on the housing of the vehicle imaging device 100, or may be a remote controller.

[0011] As shown in FIG. 2, the vehicle imaging device 100 is mounted, as an example, on the inner surface of the windshield 20w of the vehicle 200 (see FIG. 3) near the rearview mirror 20m. The vehicle imaging device 100 shown in FIG. 2 includes a visible light camera 1, a thermal camera 2, and an image processing unit 3 (not shown in FIG. 2) in a housing, and a display unit 5 on the inner surface of the housing. The visible light camera 1 and the thermal camera 2 may be attached externally to the housing. The display unit 5 may be attached externally to the housing, or may be configured externally. The vehicle imaging device 100 may be mounted in the front grille of the vehicle 200.

[0012] 1, the image processing unit 3 includes a control unit 301, a switch 302, a heat source image correction unit 303, a correction amount storage unit 304, and an image synthesis unit 305. The image processing unit 3 also includes a heat source region extraction unit 306, a heat source region correction unit 307, a specific heat source region determination unit 308, a specific shape storage unit 309, an auxiliary image generation unit 310, an auxiliary image superimposition unit 311, and a switch 312. The image processing unit 3 may be configured as any of an integrated circuit, a programmable logic controller (PLC), a field-programmable gate array (FPGA), an application specific integrated circuit (ASIC), and a microcomputer.

[0013] The control unit 301 can selectively set the image processing unit 3 to a calibration mode or a shooting mode in response to the driver's operation of the operation unit 4. The driver first operates the operation unit 4 to set the image processing unit 3 to the calibration mode.

[0014] The operation of the image processing unit 3 in the calibration mode will be described. In Fig. 3, the vehicle 200 is parked in a place other than a road, such as a parking lot or an open space. The driver places heat source markers 6a to 6g in front of the vehicle 200 on the ground. The heat source marker 6 generates heat by storing a predetermined heat source in, for example, a cylindrical case.

[0015] Specifically, the driver places heat source markers 6a and 6e at a distance D1 from the left and right ends of the front end of the vehicle 200 in the left-right direction and a distance D2 forward. The driver places heat source markers 6b and 6f at a distance D3 forward from the heat source markers 6a and 6e, respectively. The driver places heat source markers 6c and 6g at a distance D3 forward from the heat source markers 6b and 6f, respectively. The driver places heat source marker 6d in the center between the heat source markers 6b and 6f.

[0016] A first line connecting the heat source markers 6a, 6b, and 6c is a straight line, and a second line connecting the heat source markers 6e, 6f, and 6g is a straight line parallel to the first line. The first and second lines are parallel to the front-rear direction of the vehicle 200. The distances D2 and D3 are about 10 m to 20 m. The distances D2 and D3 do not need to be the same.

[0017] It is preferable that the heat source marker 6a and the heat source marker 6b are connected with connecting lines 7a and 7b, and that the heat source marker 6b and the heat source marker 6c are connected with connecting lines 7c and 7d, respectively. It is preferable that the heat source marker 6e and the heat source marker 6f are connected with connecting lines 7c and 7d, and that the heat source marker 6f and the heat source marker 6g are connected with connecting lines 7c and 7d, respectively. The connecting lines 7a to 7d are heated by the heat generated by the heat source markers 6a to 6c and 6e to 6g, or generate heat by having a heat source. The number and arrangement of the heat source markers shown in FIG. 3 are merely an example. The shape of the heat source marker is not limited to a cylindrical shape.

[0018] 3, the visible light camera 1 and the thermal camera 2 capture an image of the area in front of the vehicle 200. When the image processing unit 3 is in the calibration mode, the switches 302 and 312 are connected to the terminal Ta under the control of the control unit 301. A visible light image as shown in FIG. 4, which is generated by the visible light camera 1 capturing an image of the area in front of the vehicle 200, is supplied to the image synthesis unit 305.

[0019] 4 shows frame F1 of a visible light image. Frame F1 includes the leading edge of the vehicle 200, heat source markers 6a-6g, and connecting lines 7a-7d. Note that the leading edge of the vehicle 200 may not be captured by the visible light camera 1 and may not be present in frame F1. The image of frame F1 does not change over time, so even if the frames of the visible light image progress, it will be referred to as frame F1.

[0020] An infrared image (far-infrared image) as shown in FIG. 5, which is generated by the thermo camera 2 by capturing an image of the area in front of the vehicle 200, is supplied to the heat source image correction unit 303 via the switch 302. FIG. 5 shows frame F2 of the infrared image. In frame F2, the portion shown by the dashed line, or the area surrounded by the dashed line and hatched, is a location (heat source area) where the temperature is higher than the surroundings and where a heat source exists. The heat source image correction unit 303 supplies the input infrared image to the image synthesis unit 305. Similarly, since the image of frame F2 does not change over time, the image will be referred to as frame F2 even if the infrared image frame advances.

[0021] Frame F2 includes a heat source area 200ir corresponding to the front end of the vehicle 200, heat source areas 6air to 6gir corresponding to the heat source markers 6a to 6g, and heat source areas 7air to 7dir corresponding to the connecting lines 7a to 7d. Similarly, the front end of the vehicle 200 may not be photographed by the thermo camera 2 and may not be present in frame F2.

[0022] The image synthesis unit 305 supplies a synthesized image obtained by synthesizing the input visible light image and infrared image to the display unit 5 via the switch 312. The image synthesis unit 305 delays the frames of the visible light image as necessary so that the visible light image and the infrared image are synthesized with frames at the same timing.

[0023] The display unit 5 displays the input composite image. Fig. 6 shows frame F12, which is a composite of frame F1 of a visible light image and frame F2 of an infrared image. Similarly, even if a composite image frame progresses, it will be called frame F12. The scale ratio when the visible light camera 1 captures the subject and the scale ratio when the thermal camera 2 captures the subject may not be the same, and in frame F12, the visible light image and the infrared image are misaligned.

[0024] While viewing the composite image of frame F12 displayed on the display unit 5, the driver operates the operation unit 4 to instruct the control unit 301 to enlarge or reduce the infrared image captured by the thermo camera 2. The control unit 301 controls the heat source image correction unit 303 to enlarge or reduce the infrared image. Figure 6 shows a state in which the scale ratio of the thermo camera 2 is larger than that of the visible light camera 1, and the infrared image is displayed smaller.

[0025] By the control unit 301 controlling the heat source image correction unit 303 to enlarge the infrared image, as shown in FIG. 7, the positions of the front end of the vehicle 200, the heat source markers 6a to 6g, the connecting lines 7a to 7d, and the heat source regions 200ir, 6air to 6gir, and 7air to 7dir are approximately aligned with each other.

[0026] Even if the scale ratio when the visible light camera 1 captures the subject is the same as the scale ratio when the thermo camera 2 captures the subject, the visible light image and the infrared image may be misaligned in the horizontal or vertical direction. In this case, the driver can operate the operation unit 4 while looking at the composite image of the frame F12 displayed on the display unit 5 to shift the infrared image captured by the thermo camera 2 in the horizontal or vertical direction.

[0027] The correction amount storage unit 304 stores the correction amount obtained by correcting the infrared image by the heat source image correction unit 303 so that the positions of the visible light image and the infrared image coincide with each other. The correction amount includes a scaling ratio for enlarging or reducing the infrared image when the scale ratios of the visible light camera 1 and the thermo camera 2 are not the same. The correction amount includes a displacement amount for shifting the infrared image in the horizontal or vertical direction when the positions of the visible light image and the infrared image are shifted in the horizontal or vertical direction.

[0028] When the correction amounts for matching the positions of the heat source markers 6a-6g and the connecting lines 7a-7d with the positions of the heat source regions 6air-6gir and 7air-7dir are stored in the correction amount storage unit 304 in the above manner, the calibration of the image processing unit 3 is completed. The above-described calibration for matching the positions of the visible light image and the infrared image is merely an example, and other known techniques may be used. Furthermore, if the scale ratios of the visible light camera 1 and the thermo camera 2 are the same and there is no misalignment between the visible light image and the infrared image in the horizontal or vertical direction, the calibration may be omitted.

[0029] When the calibration, which is performed as necessary, is completed, the driver operates the operation unit 4 to set the image processing unit 3 to the shooting mode. In response to the driver's operation of the operation unit 4, the control unit 301 sets the image processing unit 3 to the shooting mode. When the image processing unit 3 is set to the shooting mode, the switches 302 and 312 are connected to the terminal Tb under the control of the control unit 301.

[0030] Fig. 8 shows an example of an actual scene of a road on which a vehicle 200 is traveling. In the example shown in Fig. 8, the vehicle 200 is traveling in the left lane of a road 10 in a snowy area. A leading vehicle 11 is traveling ahead of the vehicle 200. Feathered poles 12Pa to 12Pc are installed on both sides of the road 10. An arrow feathered pole is a sign whose official name is a fixed visual guidance post. Feathers 12a to 12c in the shape of a downward arrow are attached to the tips of the feathered poles 12Pa to 12Pc, respectively.

[0031] The positions of the tips of the arrow feathers 12a to 12c in a direction perpendicular to the road 10 indicate the positions of the outer road line (the outer white line) drawn on the edge of the road 10. In Fig. 8, the outer road line is covered with snow and cannot be seen. The tips of the arrow feathers 12a to 12c are assumed to be 5 meters above the ground.

[0032] A feathered pole that does not specify which of the feathered poles 12Pa to 12Pc is used, or any other feathered pole other than the feathered poles 12Pa to 12Pc, will be referred to as a feathered pole 12P. A feather that does not specify which of the feathers 12a to 12c is used, or any other feather other than the feathers 12a to 12c will be referred to as a feather 12. The feather 12 incorporates a light source such as a light-emitting diode and serves as a heat source.

[0033] If, for example, a snowstorm occurs in the actual scene shown in FIG. 8, visibility becomes poor as shown in FIG. 9, and visibility of the road 10, the leading vehicle 11, and the arrow feathers 12a to 12c deteriorates significantly.

[0034] The vehicle imaging device 100 includes an image processor 3, thereby enabling the driver to travel safely along the road 10 even in poor visibility conditions as shown in Fig. 9. Fig. 10 shows a frame F10 of a visible light image generated by the visible light camera 1 when the vehicle 200 is traveling along the road 10 in the state shown in Fig. 9. The leading edge of the vehicle 200 may not be captured by the visible light camera 1 and may not be present in the frame F10. The visible light image generated by the visible light camera 1 is a moving image, and therefore the frames of the visible light image change sequentially. Frame F10 shown in Fig. 10 shows one of the sequentially changing frames.

[0035] The infrared image generated by the thermo camera 2 is supplied to a heat source region extraction unit 306 via a switch 302. The heat source region extraction unit 306 extracts a heat source region, which is a region that is contained in each frame of the infrared image and has a temperature equal to or higher than a predetermined temperature. FIG. 11 shows a heat source region extracted in frame F20 of the infrared image. Similarly, frame F20 shown in FIG. 11 shows one frame among frames that change sequentially. Frames F10 and F20 are frames of the same timing.

[0036] The frame F20 includes a heat source area 200ir corresponding to the tip of the vehicle 200, a heat source area 11ir corresponding to the preceding vehicle 11, and heat source areas 12air to 12cir corresponding to the arrow feathers 12a to 12c. The tip of the vehicle 200 is not necessarily captured by the thermal camera 2 and does not necessarily have to be present in the frame F20.

[0037] The heat source region extraction unit 306 supplies each frame of the infrared image including the extracted heat source region to a heat source region correction unit 307. The heat source region correction unit 307 corrects each frame of the infrared image by the correction amount stored in the correction amount storage unit 304. The heat source region correction unit 307 supplies each corrected frame to a specific heat source region determination unit 308 and an auxiliary image generation unit 310.

[0038] The heat source region correction unit 307 may not correct each frame of the infrared image, and each frame of the infrared image output from the heat source region extraction unit 306 may be directly supplied to the specific heat source region determination unit 308 and the auxiliary image generation unit 310.

[0039] The vehicle speed is input to the specific heat source region determination unit 308. The vehicle speed may be detected based on a change in position information of the vehicle 200, or may be acquired from an ECU (Electronic Control Unit). In the former case, a GNSS receiving unit (not shown) receives GNSS signals transmitted from three or more satellites for a Global Navigation Satellite System (GNSS). A position calculation unit (not shown) calculates the current position (position information) of the vehicle 200 based on the received GNSS signals. A vehicle speed detection unit (not shown) detects the vehicle speed based on a change in the position information supplied from the position calculation unit.

[0040] The specific shape storage unit 309 stores data indicating a specific shape, which is the shape of an object installed on the road 10 used to display the road 10 on which the vehicle 200 travels on the visible light image. The shape of the feathers 12 of a downward arrow is an example of the specific shape. The specific heat source area determination unit 308 excludes the heat source area of ​​the moving body from the extracted heat source area based on the input vehicle speed, and determines the presence or absence of a specific heat source area based on the data indicating the specific shape stored in the specific shape storage unit 309. The specific heat source area is a heat source area of ​​an object installed on the road 10 used to display the road 10 on which the vehicle 200 travels on the visible light image.

[0041] The arrow feathers 12 attached to the tip of the feathered pole 12P are a suitable example of an object installed on the road 10 used to show the road 10 on which the vehicle 200 travels on a visible light image. The object installed on the road 10 used to show the road 10 on which the vehicle 200 travels on a visible light image is not limited to the arrow feathers 12. In addition to the arrow feathers 12, street lamps having a specific shape can be used as the object installed on the road 10 used to show the road 10 on a visible light image. The specific heat source region determination unit 308 determines that the heat source regions 12air to 12cir are specific heat source regions in the frame F20 of the infrared image shown in FIG. 11.

[0042] The specific heat source region determination unit 308 notifies the supplementary image generation unit 310 that the heat source regions 12air to 12cir are specific heat source regions. As an example, the specific heat source region determination unit 308 may supply the supplementary image generation unit 310 with position information indicating pixel positions of the heat source regions 12air to 12cir that are specific heat source regions.

[0043] The auxiliary image generating unit 310 generates an auxiliary image to be superimposed on the visible light image based on the infrared image frame supplied from the heat source region correcting unit 307. Of the heat source regions 200ir, 11ir, and 12air to 12cir included in the infrared image frame F20 shown in Fig. 11, the heat source regions 12air to 12cir are specific heat source regions.

[0044] Therefore, the auxiliary image generating unit 310 generates a frame F11 including vertical auxiliary images 13a to 13c extending vertically downward from the positions of the respective bottom ends of the heat source regions 12air to 12cir, and connected auxiliary images 14ab, 14ac, and 14bc connecting the bottom ends of the vertical auxiliary images 13a to 13c to each other, as shown in Fig. 12. The vertical auxiliary images 13a to 13c and the connected auxiliary images 14ab to 14bc are made of visible light images. In the frame F11, the heat source regions 12air to 12cir indicated by two-dot chain lines are drawn to indicate the positions of the heat source regions 12air to 12cir in the frame F20 shown in Fig. 11, and the frame F11 does not include an image showing the heat source regions 12air to 12cir.

[0045] A vertical auxiliary image that does not specify any one of the vertical auxiliary images 13a to 13c or any other vertical auxiliary image other than the vertical auxiliary images 13a to 13c will be referred to as a vertical auxiliary image 13. A concatenated auxiliary image that does not specify any one of the concatenated auxiliary images 14ab to 14bc or any other concatenated auxiliary image other than the concatenated auxiliary images 14ab to 14bc will be referred to as a concatenated auxiliary image 14.

[0046] As described above, the tip of the feather 12 is located 5 meters above the ground, so the auxiliary image generating unit 310 generates vertical auxiliary images 13a to 13c having a number of pixels in the vertical direction equivalent to 5 meters from each position of the lower end of the heat source regions 12air to 12cir in the frame F11. The auxiliary image generating unit 310 may generate only the connected auxiliary images 14ab to 14bc without generating the vertical auxiliary images 13a to 13c. In this case, the auxiliary image generating unit 310 may generate the connected auxiliary images 14ab to 14bc that connect together positions that are vertically downward by a number of pixels equivalent to 5 meters from each position of the lower end of the heat source regions 12air to 12cir in the frame F11. If the tip of the feather 12 is located at a position other than 5 meters above the ground, the auxiliary image generating unit 310 may generate a vertical auxiliary image 13 that corresponds to the distance from the tip of the feather 12 to the ground. The height direction position of the feather 12 is arbitrary.

[0047] If the connected auxiliary images 14ab to 14bc are the first auxiliary images and the vertical auxiliary images 13a to 13c are the second auxiliary images, it is preferable that the auxiliary image generating unit 310 generates at least the first auxiliary image and generates the second auxiliary image in addition to the first auxiliary image.

[0048] The auxiliary image generating unit 310 may generate an auxiliary image based on a heat source area of ​​a moving object such as the heat source area 11ir of the leading vehicle 11. The auxiliary image generating unit 310 may take as an auxiliary image an image obtained by filling in the outline of the heat source area of ​​the moving object or the heat source area of ​​the moving object. If a pedestrian is photographed by the visible light camera 1 and the thermal camera 2, the auxiliary image generating unit 310 can generate an auxiliary image based on the pedestrian heat source area.

[0049] The auxiliary image superimposing unit 311 superimposes the auxiliary image generated by the auxiliary image generating unit 310 on the visible light image supplied from the visible light camera 1. The auxiliary image superimposing unit 311 superimposes the vertical auxiliary images 13a to 13c and the connected auxiliary images 14ab to 14bc in frame F11 shown in Fig. 12 on frame F10 of the visible light image shown in Fig. 10. The auxiliary image superimposing unit 311 delays the frame of the visible light image as necessary so as to superimpose, on the frame of the visible light image, an auxiliary image based on a specific heat source region included in the frame of the infrared image at the same timing as the frame of the visible light image.

[0050] The visible light image on which the auxiliary image is superimposed is supplied to the display unit 5 via the switch 312. The display unit 5 displays the visible light image on which the auxiliary image is superimposed. When the visible light image is frame F10 shown in Fig. 10 and the auxiliary image generated by the auxiliary image generation unit 310 is frame F11 shown in Fig. 12, a frame F111 as shown in Fig. 13 is displayed on the display unit 5.

[0051] As shown in Fig. 13, due to poor visibility caused by a snowstorm, the leading vehicle 11 and the arrow feathers 12a-12c are barely visible as a visible light image. The driver can recognize that the area surrounded by the connected auxiliary images 14ab-14bc is the road surface of the road 10. The driver can recognize the edges of the road 10 by the vertical auxiliary images 13a-13c. If the vertical auxiliary image 13 is superimposed on the visible light image, the edges of the road 10 can be easily recognized, so it is preferable to superimpose the vertical auxiliary image 13 in addition to the connected auxiliary image 14.

[0052] As described above, if the auxiliary image generation unit 310 generates an auxiliary image corresponding to a heat source region other than the specific heat source region, the auxiliary image corresponding to the heat source region other than the specific heat source region is also superimposed on the frame F111 of the visible light image shown in FIG. 13.

[0053] 14 shows a frame F11 in which a total of six feathered poles 12P are installed on both sides of a road 10, vertical auxiliary images 13 are added to the lower end of heat source regions 12ir corresponding to each feather 12, and the lower ends of each vertical auxiliary image 13 are connected by connected auxiliary images 14. As shown in FIG. 14, if many connected auxiliary images 14 intersect, it becomes difficult to see.

[0054] Therefore, it is preferable that the auxiliary image generating unit 310 generates a frame F11' as the third auxiliary image, in which the area surrounded by the connected auxiliary image 14 is filled in with, for example, a single color as a filled-in auxiliary image 15, as shown in Fig. 15. The driver can recognize the filled-in auxiliary image 15 as the road surface of the road 10. The color of the filled-in auxiliary image 15 may be any conspicuous color.

[0055] FIG. 16 shows a frame F11 in which a vertical auxiliary image 13 is added to the lower end of the heat source region 12ir corresponding to each of the arrow feathers 12 when a total of four arrow feather poles 12P are installed on both sides of the road 10, and the lower ends of each vertical auxiliary image 13 are connected with a connected auxiliary image 14. FIG. 17 shows a frame F11' in which the area surrounded by the connected auxiliary image 14 is filled with, for example, a single color to form a filled-in auxiliary image 15. When there are four or more vertical auxiliary images 13, the auxiliary image generating unit 310 may form the area surrounded by the connected auxiliary image 14 as a filled-in auxiliary image 15. When there are three vertical auxiliary images 13 shown in FIG. 12, the area surrounded by the connected auxiliary image 14 may be formed as a filled-in auxiliary image 15.

[0056] When the vertical auxiliary image 13 is equal to or greater than a predetermined number, that is, when there are equal to or greater than a predetermined number of heat source regions determined to be specific heat source regions in one frame, the auxiliary image generating unit 310 generates a filled-in auxiliary image 15 by filling in an area surrounded by the connected auxiliary image 14 with a predetermined color. The predetermined number is any multiple number. The auxiliary image superimposing unit 311 superimposes the filled-in auxiliary image 15 on a frame in which there is equal to or greater than a predetermined number of heat source regions determined to be specific heat source regions.

[0057] 18A and 18B, the operation of the image processing device according to an embodiment, which is the operation in the shooting mode by the image processing unit 3, and the imaging method according to an embodiment will be described. The above-mentioned calibration in the calibration mode is completed before the process shown in FIG. 18A is started.

[0058] 18A, when the image processor 3 starts the process of the shooting mode, the heat source region extractor 306 determines in step S1 whether or not there is a heat source region, which is a region with a temperature equal to or higher than a predetermined temperature, within the frame of the infrared image. If there is no heat source region (NO), the heat source region extractor 306 repeats the process of step S1.

[0059] If a heat source region is present in the frame of the infrared image in step S1 (YES), the heat source region extraction unit 306 determines the edge of the heat source region and extracts the heat source region in step S2. In step S3, the heat source region extraction unit 306 calculates the position and number of pixels of the heat source region in the frame. In step S4, the heat source region correction unit 307 corrects the position and number of pixels of the heat source region. Step S4 may be omitted.

[0060] In step S5, the specific heat source area determination unit 308 calculates the actual size of the heat source area. In step S6, the specific heat source area determination unit 308 starts a timer, and in step S7, determines whether a predetermined time has elapsed. If the predetermined time has not elapsed (NO), the control unit 301 repeats the process of step S7. If the predetermined time has elapsed (YES), in step S8, the specific heat source area determination unit 308 subtracts the vehicle speed from the apparent movement speed of the heat source area to calculate the actual movement speed of the heat source area.

[0061] In step S9, the specific heat source area determination unit 308 determines whether the moving speed of the heat source area is less than 2 km / h. If the moving speed of the heat source area is less than 2 km / h (YES), the specific heat source area determination unit 308 determines whether the heat source area has a specific shape in step S10. If the moving speed of the heat source area is less than 2 km / h, the heat source area is assumed to be a fixed object such as a sign, rather than a moving object. If the heat source area has a specific shape (YES), the specific heat source area determination unit 308 determines in step S11 that the heat source area is a specific heat source area, and transitions the process to step S12 in FIG. 18B.

[0062] In Fig. 18A, the specific heat source region is determined based on the moving speed and shape of the heat source region, but the specific heat source region may be determined based only on the moving speed of the heat source region, or the specific heat source region may be determined based only on the shape of the heat source region. It is preferable to determine the specific heat source region based on the moving speed and shape of the heat source region. Note that whether or not the moving speed for determining that the heat source region is a fixed object rather than a moving object is less than 2 km per hour is one example, and is not limited to less than 2 km per hour.

[0063] 18A, the shape of the heat source region is determined after the moving speed of the heat source region is determined, but the moving speed of the heat source region may be determined after the shape of the heat source region is determined. More specifically, if the heat source region has a specific shape (YES), the specific heat source region determination unit 308 determines whether the moving speed of the heat source region is less than 2 km per hour, and if the heat source region does not have a specific shape (NO), the process proceeds to step S19 in FIG. 18B. If the moving speed of the heat source region is less than 2 km per hour (YES), the specific heat source region determination unit 308 determines that the heat source region is a specific heat source region, and if the moving speed of the heat source region is not less than 2 km per hour (NO), the heat source region is determined not to be a specific heat source region.

[0064] If the moving speed of the heat source region is not less than 2 km per hour in step S9 (NO), the specific heat source region determination unit 308 determines in step S20 that the heat source region is a moving body, and shifts the process to step S19 in Fig. 18B. If the heat source region does not have a specific shape in step S10 (NO), the specific heat source region determination unit 308 determines in step S21 that the heat source region is not a specific heat source region, and shifts the process to step S19 in Fig. 18B.

[0065] 18B , in step S12, the auxiliary image generation unit 310 generates vertical auxiliary images 13 extending downward from the positions of the bottom ends of the multiple specific heat source regions, and in step S13, generates a connected auxiliary image 14 by connecting the positions of the bottom ends of the vertical auxiliary images 13. In step S14, the auxiliary image generation unit 310 determines whether there are four or more vertical auxiliary images 13. If there are not four or more vertical auxiliary images 13 (NO), in step S15, the auxiliary image superimposition unit 311 superimposes the vertical auxiliary images 13 and the connected auxiliary images 14 on the visible light image, and supplies the superimposed visible light image to the display unit 5.

[0066] If the number of vertical auxiliary images 13 is four or more in step S14 (YES), in step S16, the auxiliary image generation unit 310 generates a fill-in auxiliary image 15 in the area surrounded by the connected auxiliary image 14. In step S17, the auxiliary image superimposition unit 311 superimposes the vertical auxiliary image 13 and the fill-in auxiliary image 15 on the visible light image, and supplies the superimposed visible light image to the display unit 5.

[0067] Following step S15 or S17, the display unit 5 displays the superimposed visible light image in step S18, and the process proceeds to step S19. In step S19, the control unit 301 determines whether or not to end the operation by, for example, turning off the power of the vehicle imaging device 100. If the operation is not to be ended (NO), the image processing unit 3 returns the process to step S1 in Fig. 18A, and repeats the processes from step S1 onwards. If the operation is to be ended (YES), the image processing unit 3 ends the process.

[0068] As described above, according to the image processing device and imaging method of one embodiment, it is possible to improve the visibility of the road 10 when it is difficult to recognize the road 10 on which the vehicle 200 is traveling at night, or during the daytime when fog is present or snow is falling.

[0069] The present invention is not limited to the above-described embodiment, and various modifications are possible without departing from the gist of the present invention. [Explanation of symbols]

[0070] 1. Visible light camera 2. Thermal camera 3. Image Processing Section 4 Control section 5 Display section 100 Vehicle imaging device 200 vehicles 301 Control Unit 302,312 Switches 303 Heat source image correction unit 304 Correction amount storage unit 305 Image Synthesis Unit 306 Heat source area extraction part 307 Heat source area correction section 308 Specific heat source area determination section 309 Specific shape memory section 310 Auxiliary Image Generation Unit 311 Auxiliary image overlay unit

Claims

1. a heat source region extraction unit that extracts a heat source region that is a region having a temperature equal to or higher than a predetermined temperature included in each frame of an infrared image generated by the thermal camera by photographing the area in front of the vehicle; a specific heat source region determination unit that determines the heat source region as a specific heat source region when the movement speed of the heat source region extracted by the heat source region extraction unit is less than a predetermined speed; an auxiliary image generating unit that generates a first auxiliary image that connects positions a predetermined distance below the positions of the lower ends of a plurality of heat source regions that are determined to be specific heat source regions by the specific heat source region determining unit among the heat source regions included in each frame of the infrared image; an auxiliary image superimposing unit that superimposes the first auxiliary image on each frame of a visible light image generated by a visible light camera capturing an image of a scene ahead of the vehicle; An image processing device comprising:

2. the auxiliary image generating unit generates a second auxiliary image extending downward by the predetermined distance from each lower end position of the plurality of heat source regions determined to be the specific heat source region; The auxiliary image superimposing unit superimposes the second auxiliary image on each frame of the visible light image. The image processing device according to claim 1 .

3. the auxiliary image generating unit generates a third auxiliary image by filling an area surrounded by the first auxiliary image with a predetermined color when the number of heat source areas determined to be the specific heat source areas in one frame is equal to or greater than a predetermined number; The auxiliary image superimposing unit superimposes the third auxiliary image onto a frame of the visible light image in which the heat source regions determined to be the specific heat source regions are equal to or greater than the predetermined number.

3. The image processing device according to claim 1 or 2.

4. The image processing device according to claim 1 , wherein the specific heat source region determining unit determines the heat source region extracted by the heat source region extracting unit to be the specific heat source region when the specific heat source region determining unit determines that the heat source region extracted by the heat source region extracting unit has a specific shape.

5. A visible light camera captures the image of the area in front of the vehicle to generate a visible light image. A thermal camera captures an image of the area in front of the vehicle and generates an infrared image; an image processing unit extracts a heat source region, which is a region having a temperature equal to or higher than a predetermined temperature, included in each frame of the infrared image; The image processing unit determines the heat source area as a specific heat source area when the movement speed of the extracted heat source area is less than a predetermined speed, The image processing unit generates a connected auxiliary image that connects positions of a plurality of heat source regions determined to be the specific heat source regions among the heat source regions included in each frame of the infrared image, the positions being a predetermined distance below the positions of the respective lower ends of the plurality of heat source regions, the image processing unit superimposes the connected auxiliary image on each frame of the visible light image to generate a superimposed visible light image; A display unit displays the superimposed visible light image. Imaging method.

Citation Information

Patent Citations

  • Display processing unit

    JP1999308609A