Detection device

Abstract

To create a detection device that can detect traffic lights even in low-light environments using inexpensive sensors. [Solution] The detection device includes: a first detection unit that detects a light region representing the lights of a traffic light from a visible light image representing the environment in front of a vehicle acquired using a visible light camera; a setting unit that sets a traffic light region in the visible light image that is estimated to include the housing of the traffic light based on the light region; a determination unit that determines a corresponding region in the infrared image that corresponds to the traffic light region based on the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image is acquired, and the yaw angle, pitch angle, and roll angle of the infrared camera when an infrared image representing the environment in front of a vehicle acquired using an infrared camera is acquired; and a second detection unit that detects the housing of the traffic light within the corresponding region.

Description

Technical Field

[0001] The present disclosure relates to a detection device.

Background Art

[0002] Conventionally, a detection device for detecting a traffic signal has been mounted on a vehicle. The detection device detects a traffic signal in front of the vehicle and determines the color of the traffic signal light.

[0003] The detection device uses a visible light camera to acquire an image representing the environment in front of the vehicle, and detects the traffic signal represented in this image. The detection device detects the area where the light and the housing on which the light is arranged are represented as the area representing the traffic signal in the image.

[0004] On the other hand, in a low-light environment such as at night, a reflection of a guardrail or a street lamp may be represented in the image. However, in an image acquired by a visible light camera in a low-light environment, it is difficult to identify the housing, so it may be difficult to distinguish between a reflection of a guardrail or a street lamp and a traffic signal light.

[0005] Therefore, in Patent Document 1, it has been proposed to detect a traffic signal based on an image acquired using a visible light camera and three-dimensional point cloud information acquired using a LiDAR sensor. The LiDAR sensor can acquire three-dimensional point cloud information of an object even in a low-light environment.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] LiDAR sensors have a trade-off relationship between distance to the target object and resolution. Therefore, to detect a traffic light located high up in front of a vehicle, a separate LiDAR sensor capable of detecting objects at angles higher than horizontal is required, in addition to the LiDAR sensor used to detect other vehicles around the vehicle.

[0008] Therefore, there was a problem in that multiple expensive LiDAR sensors had to be installed on the vehicle in order to detect traffic lights.

[0009] Therefore, the objective of this disclosure is to provide a detection device that can detect traffic signals even in low-light environments using inexpensive sensors. [Means for solving the problem]

[0010] (1) According to one embodiment, a detection device is provided. This detection device is characterized by comprising: a first detection unit that detects a light region representing the lights of a traffic signal from a visible light image representing the environment in front of a vehicle acquired using a visible light camera; a setting unit that sets a traffic signal region in the visible light image that is estimated to include the housing of the traffic signal based on the light region; a determination unit that determines a corresponding region in the infrared image that corresponds to the traffic signal region based on the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image is acquired, and the yaw angle, pitch angle, and roll angle of the infrared camera when an infrared image representing the environment in front of a vehicle acquired using an infrared camera is acquired; and a second detection unit that detects the housing of the traffic signal within the corresponding region.

[0011] (2) In the detection device of (1), it is preferable that the setting unit sets the signal area as a position represented in the visible light image coordinate system of the visible light image, the determination unit converts the signal area represented in the visible light image coordinate system to a position represented in a three-dimensional visible light camera coordinate system with the visible light camera as the origin, converts the signal area represented in the visible light camera coordinate system to a position represented in a three-dimensional vehicle coordinate system with the vehicle as the origin, based on the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image was acquired, converts the signal area represented in the vehicle coordinate system to a position represented in a three-dimensional infrared camera coordinate system with the infrared camera as the origin, based on the yaw angle, pitch angle, and roll angle of the infrared camera when the infrared image was acquired, and projects the signal area represented in the infrared camera coordinate system onto the infrared image and determines the projected signal area as the corresponding area in the infrared image.

[0012] (3) In the detection device of (2), it is preferable that the determination unit converts the signal area represented in the visible light camera coordinate system to a position represented in the vehicle coordinate system based on the representative values ​​of the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image is acquired, and converts the signal area represented in the vehicle coordinate system to a position represented in the infrared camera coordinate system based on the representative values ​​of the yaw angle, pitch angle, and roll angle of the infrared camera when the infrared image is acquired.

[0013] (4) In the detection device of (1), it is preferable that the setting unit sets the signal area as a position represented in the visible light image coordinate system of the visible light image, and the determination unit determines the corresponding area represented in the infrared image coordinate system of the infrared image that corresponds to the signal area, based on the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image is acquired, and the yaw angle, pitch angle, and roll angle of the infrared camera when the infrared image is acquired.

[0014] (5) In the detection devices of (1) to (4), it is preferable that the second detection unit detects the housing of the traffic signal within the corresponding area using a signal representing an unstandardized infrared image acquired using an infrared camera. [Effects of the Invention]

[0015] The detection device according to the present disclosure can detect a traffic signal even in a low-light environment by using inexpensive sensors such as a visible light camera and an infrared camera.

Brief Description of the Drawings

[0016] [Figure 1] It is a diagram for explaining an outline of the operation of the detection device of the present embodiment. [Figure 2] It is a hardware configuration diagram of a vehicle in which the detection device of the present embodiment is mounted. [Figure 3] It is an example of an operation flowchart regarding the detection process of the detection device of the present embodiment. [Figure 4] It is a diagram for explaining a visible light image coordinate system, a visible light camera coordinate system, and a vehicle coordinate system. [Figure 5] It is a diagram for explaining an infrared image coordinate system, an infrared camera coordinate system, and a vehicle coordinate system. [Figure 6] It is an example of an operation flowchart regarding a modification of the detection process of the detection device of the present embodiment.

Modes for Carrying Out the Invention

[0017] FIG. 1 is a diagram for explaining an outline of the operation of the detection device of the present embodiment. As shown in FIG. 1, the vehicle 10 includes a control device 11 and a detection device 12. The detection device 12 detects a traffic signal 30 in front of the vehicle 10 based on an image acquired using a visible light camera 2 and an infrared camera 3. The control device 11 performs automatic driving control or driving support control of the vehicle 10 based on the color of the light of the traffic signal 30 detected by the detection device 12.

[0018] The vehicle 10 is traveling on a road 50 in a low-light environment such as at night. The road 50 has lanes 51 and 52. The vehicle 10 is traveling in lane 51. A traffic signal 30 is located in front of the vehicle 10. In the traffic signal 30, a light 31 is lit.

[0019] The visible light camera 2 acquires a visible light image P1 representing the environment in front of the vehicle 10. Even in a low light environment, the lit lamp 31 of the traffic signal 30 can be represented in the visible light image P1.

[0020] Also, the infrared camera 3 acquires an infrared image P2 representing the environment in front of the vehicle 10. Even in a low light environment, the lit lamp 31 and the housing 32 of the traffic signal 30 can be represented in the infrared image P2.

[0021] The detection device 12 detects a lamp area including the lit lamp 31 of the traffic signal 30 from the visible light image P1. Based on the lamp area, the detection device 12 sets a traffic signal area Q that is estimated to include the housing 32 of the traffic signal 30 within the visible light image P1. The traffic signal area Q is set as an area that is estimated to include the lit lamp 31 and a portion representing the housing 32 in which this lamp 31 is arranged in the visible light image P1.

[0022] Based on the yaw angle, pitch angle, and roll angle of the visible light camera 2 when the visible light image P1 is acquired, and the yaw angle, pitch angle, and roll angle of the infrared camera 3 when the infrared image P2 is acquired, the detection device 12 determines a corresponding area R corresponding to the traffic signal area Q within the infrared image P2 acquired using the infrared camera 3.

[0023] When the vehicle 10 is located on a horizontal ground, the traffic signal area Q of the visible light image P1 and the corresponding area R within the infrared image P2 corresponding to the traffic signal area Q are determined based on the internal parameters including the respective focal lengths of the visible light camera 2 and the infrared camera 3, and the external parameters including the installation position and installation orientation, etc.

[0024] However, since the posture of the vehicle 10 during travel may be different from when the vehicle 10 is located on a horizontal ground, the corresponding area R within the infrared image P2 corresponding to the traffic signal area Q of the visible light image P1 is determined including the influence of the posture of the vehicle 10.

[0025] The detection device 12 detects the housing 32 of the traffic light 30 that is represented within the corresponding area R. The detection of the housing 32 of the traffic light 30 that is represented within the corresponding area R means that the traffic light 30 has been detected in front of the vehicle 10.

[0026] When the housing 32 of the traffic light 30 is detected, the detection device 12 notifies the control device 11 of the color of the traffic light 30's light. If the control device 11 is, for example, red, it provides driving assistance to slow down the vehicle 10.

[0027] As described above, the detection device 12 of this embodiment can detect the traffic light 30 even in a low-light environment using inexpensive sensors such as the visible light camera 2 and the infrared camera 3. In the above description, the detection device 12 detected the traffic light 30 when the vehicle 10 was in a low-light environment, but the detection device 12 can also detect the traffic light 30 when the vehicle 10 is not in a low-light environment.

[0028] Figure 2 is a hardware configuration diagram of a vehicle 10 on which the detection device 12 of this embodiment is installed. The vehicle 10 includes a visible light camera 2, an infrared camera 3, an angular velocity sensor 4, a user interface (UI) 5, a control device 11, and the detection device 12, etc.

[0029] The visible light camera 2, the infrared camera 3, the angular velocity sensor 4, the user interface (UI) 5, the control device 11, and the detection device 12 are connected to each other via an in-vehicle network 13 that conforms to standards such as a controller area network.

[0030] The visible light camera 2 is mounted on the vehicle 10 so as to face forward. The visible light camera 2 acquires a visible light image representing the environment of a predetermined field of view in front of the vehicle 10. The visible light image may represent the road included in the predetermined area in front of the vehicle 10, as well as road features such as lane markings and traffic lights on the road surface. The visible light camera 2 outputs the acquired visible light image to the control device 11 and detection device 12, etc., via the in-vehicle network 13.

[0031] The visible light camera 2 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to visible light, such as a CCD or C-MOS. The visible light camera 2 also has an imaging optical system that forms an image of the area to be imaged on the two-dimensional detector.

[0032] The infrared camera 3 is mounted on the vehicle 10 so as to face forward. The infrared camera 3 acquires an infrared image representing the environment of a predetermined field of view in front of the vehicle 10. The infrared image may show roads included in the predetermined area in front of the vehicle 10, as well as road features such as lane markings and traffic lights on the road surface. The infrared camera 3 outputs the acquired infrared image to the control device 11 and detection device 12, etc., via the in-vehicle network 13. It is preferable that the field of view of the infrared camera 3 and the field of view of the visible light camera 2 overlap. For example, it is preferable that the field of view of the infrared camera 3 and the field of view of the visible light camera 2 overlap by 50 to 80% or more.

[0033] The infrared camera 3 has a two-dimensional detector composed of an array of photoelectric conversion elements sensitive to infrared light, such as a CCD or C-MOS. Preferably, the two-dimensional detector of the infrared camera 3 is also sensitive to far-infrared light. The infrared camera 3 also has an imaging optical system that forms an image of the area to be imaged on the two-dimensional detector.

[0034] Infrared images can display images of objects such as traffic lights more clearly than visible light images, even in low-light environments such as at night. Although the visible light is weak, traffic lights emit infrared radiation, albeit at low temperatures, so the lights and their casings can be displayed in infrared images.

[0035] The angular velocity sensor 4 detects the yaw angular velocity, pitch angular velocity, and roll angular velocity of the vehicle 10 and outputs them to the control device 11, etc., via the in-vehicle network 13. The yaw angular velocity is measured as the angular velocity around the Yv axis of the vehicle coordinate system Sv (see Figure 4 or Figure 5), which has its origin at the vehicle 10. The pitch angular velocity is measured as the angular velocity around the Xv axis of the vehicle coordinate system Sv. The roll angular velocity is measured as the angular velocity around the Zv axis of the vehicle coordinate system Sv. For example, gyroscopes placed on each axis can be used as the angular velocity sensor 4.

[0036] The vehicle coordinate system Sv is defined, for example, with the origin Ov at the center of the rear axle connecting the two rear wheels of vehicle 10. The Zv axis is set in the direction of travel of vehicle 10, the Xv axis is set perpendicular to the Zv axis and parallel to the ground, and the Yv axis is set in the vertical direction.

[0037] The control device 11 and the detection device 12 determine the yaw angle θv of the vehicle 10 based on the yaw angular velocity, the pitch angle ψv based on the pitch angular velocity, and the roll angle Φv based on the roll angular velocity.

[0038] The reference values ​​(e.g., 0 degrees) for the yaw angle, pitch angle, and roll angle of vehicle 10 are those values ​​when vehicle 10 is facing straight ahead and stopped on level ground.

[0039] As the attitude of the vehicle 10 changes, the orientation of the optical axes of the visible light camera 2 and the infrared camera 3 also changes in yaw angle, pitch angle, and roll angle by the same amount as the vehicle 10.

[0040] UI5 is an example of a notification unit. UI5 is controlled by the control device 11 and the detection device 12 to notify the driver 40 of information regarding the vehicle 10. UI5 has a display device 5a, such as a liquid crystal display or a touch panel, to display the information. UI5 may also have an acoustic output device (not shown) for notifying the driver 40 of the information. UI5 also has an input device, such as a touch panel or operation buttons, to input operation information from the driver 40 to the vehicle 10. UI5 outputs the input information to the control device 11 and the detection device 12, etc., via the in-vehicle network 13.

[0041] The control device 11 performs automatic driving control or driving assistance control of the vehicle 10 based on the color of the traffic light notified by the detection device 12. Based on the yaw angular velocity, pitch angular velocity, and roll angular velocity of the vehicle 10, the control device 11 determines the yaw angle, pitch angle, and roll angle of the vehicle 10 and outputs them to the detection device 12 via the in-vehicle network 13.

[0042] The detection device 12 performs detection processing, setting processing, and determination processing. To this end, the detection device 12 has a communication interface (IF) 21, a memory 22, and a processor 23. The communication interface 21, the memory 22, and the processor 23 are connected via a signal line 24. The communication interface 21 has an interface circuit for connecting the detection device 12 to the in-vehicle network 13.

[0043] Memory 22 is an example of a storage unit and includes, for example, volatile semiconductor memory and non-volatile semiconductor memory. Memory 22 stores computer programs and various data of applications used in information processing performed by the processor 23.

[0044] All or part of the functions of the detection device 12 are functional modules implemented, for example, by a computer program running on the processor 23. The processor 23 includes a detection unit 231, a setting unit 232, and a determination unit 233. Alternatively, the functional modules of the processor 23 may be dedicated arithmetic circuits provided on the processor 23. The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. The processor 23 may further include other arithmetic circuits such as a logic unit, a numerical unit, or a graphics processing unit.

[0045] The control device 11 and the detection device 12 are, for example, an Electronic Control Unit (ECU). In Figure 2, the control device 11 and the detection device 12 are described as separate devices (for example, an Electronic Control Unit (ECU)), but these devices may be configured as a single device.

[0046] Figure 3 is an example of an operation flowchart relating to the detection process of the detection device 12 in this embodiment. The detection device 12 executes the detection process according to the operation flowchart shown in Figure 3 at detection times having a predetermined period. The period of the change time can be, for example, from 0.01 seconds to 0.1 seconds.

[0047] First, the detection unit 231 acquires the yaw angle, pitch angle, and roll angle of the vehicle 10 along with the visible light image P1 and infrared image P2 via the in-vehicle network 13. The visible light image P1 and infrared image P2 represent the environment in front of the vehicle 10, which are acquired simultaneously. The detection unit 231 notifies the determination unit 233 of the infrared image P2. The acquired yaw angle, pitch angle, and roll angle of the vehicle 10 represent the respective angles of the vehicle 10 from which the visible light image P1 and infrared image P2 were acquired.

[0048] The detection unit 231 determines the yaw angle, pitch angle, and roll angle of the visible light camera 2 when the visible light image P1 is acquired, based on the yaw angle, pitch angle, and roll angle of the vehicle 10 and external parameters including the installation position and orientation of the visible light camera 2 on the vehicle 10. The detection unit 231 can also determine the yaw angle, pitch angle, and roll angle of the infrared camera 3 when the infrared image P2 is acquired, based on the yaw angle, pitch angle, and roll angle of the vehicle 10 and external parameters including the installation position and orientation of the infrared camera 3 on the vehicle 10.

[0049] The detection unit 231 notifies the determination unit 233 of the yaw angle, pitch angle, and roll angle of the visible light camera 2 when the visible light image P1 is acquired, and the yaw angle, pitch angle, and roll angle of the infrared camera 3 when the infrared image P2 is acquired.

[0050] Next, the detection unit 231 detects a light area representing the lights 31 of the traffic light 30 from the visible light image P1 representing the environment in front of the vehicle 10, which is acquired using the visible light camera 2 (step S102).

[0051] For example, if the visible light image P1 is a color image, the detection unit 231 calculates the RGB value (color information) of each pixel in the visible light image P1 and determines whether the calculated RGB value of each pixel falls within the "average RGB value range of a light source". Here, the "average RGB value range of a light source" is a range determined by experiment or simulation and is pre-stored in the memory 22. If the RGB value of a certain pixel falls within the above range, the detection unit 231 determines that the pixel is a light source candidate. If the distance between pixels determined to be light source candidates is less than or equal to a predetermined reference distance, the detection unit 231 extracts these pixel groups as a light source region. Furthermore, the detection unit 231 detects whether the color of the light source is red, yellow, or blue based on the RGB values ​​of the pixel group extracted as a light source region. Even if the visible light image P1 is acquired in a low-light environment, it is possible to extract a light source region representing an illuminated light source in this way.

[0052] At this point, multiple lighting areas may be detected. If multiple lighting areas are detected, steps S103 to S108 are executed for each of the multiple lighting areas.

[0053] Next, the setting unit 232 sets a signal area Q in the visible light image P1 that is estimated to include the housing of the signal light, based on the lighting area (step S103). The signal area Q may be set as a rectangular area including the periphery of the pixel group representing the lighting area. Preferably, the signal area Q is set as an area including the housing on which the lights are arranged. Since the size of the lighting area in the visible light image P1 changes according to the distance between the vehicle 10 and the signal light, it is preferable that the signal area Q is also set to change according to the size of the lighting area.

[0054] Figure 4 illustrates the visible light image coordinate system Sp1. The signal light region Q is represented by the visible light image coordinate system Sp1, which has its origin in the visible light image P1. The visible light image coordinate system Sp1 has its origin Op1 in the upper left of the visible light image P1, and has an Xp1 axis extending to the right from the origin Op1, and a Yp1 axis extending downward from the origin Op1, perpendicular to the Xp1 axis.

[0055] Next, the determination unit 233 converts the signal area Q, represented in the visible light image coordinate system Sp1 of the visible light image P1, into a position represented in a three-dimensional visible light camera coordinate system with the visible light camera 2 as the origin (step S104).

[0056] The determination unit 233 determines the distance from the visible light camera 2 to the traffic light 30 based on the size of the pixel group representing the light area. For example, the determination unit 233 determines the number of pixels representing the diameter if the pixel group representing the light area is assumed to be a circle, and then determines the distance from the visible light camera 2 to the traffic light 30 by referring to the relationship between the number of pixels and the distance from the visible light camera 2 to the traffic light 30 based on this number of pixels representing the diameter. Since the lights of the traffic light 30 are of a predetermined size, the relationship between the number of pixels and the distance from the visible light camera 2 to the traffic light 30 is determined in advance. In step S103 described above, the size of the traffic light area Q in the visible light image P1 may be set based on the pixel group representing the light area and the distance from the visible light camera 2 to the traffic light 30.

[0057] Figure 4 illustrates the visible light camera coordinate system Sc1. In the visible light camera coordinate system Sc1, the Zc1 axis is set in the direction of the optical axis of the visible light camera 2, the Xc1 axis is set perpendicular to the Zc1 axis and parallel to the ground, and the Yc1 axis is set perpendicular to the Zc1 axis and the Xc1 axis. The origin Oc1 is at the height from the ground where the visible light camera 2 is installed.

[0058] The determination unit 233 determines the area represented by the visible light camera coordinate system Sc1 as the signal area Q in the visible light image P1, for each of the multiple pixels representing the boundary of the signal area Q in the visible light image P1, where the center of the visible light image P1 is perpendicular to the Zc1 axis and positioned at the focal length of the visible light camera 2 from the origin Oc1. This area is determined by connecting the positions at the distance from the visible light camera 2 to the signal light 30 on the extension of the straight line connecting the origin Oc1 and the boundary pixel. Note that in Figure 4, for the sake of clarity, the center of the visible light image P1 is shown to be perpendicular to the Zc1 axis but offset from it.

[0059] Next, the determination unit 233 converts the signal area Q, represented in the visible light camera coordinate system Sc1, into a position represented in the three-dimensional vehicle coordinate system Sv, which has the origin Ov on the vehicle 10 (step S105).

[0060] The transformation formula for converting the position of the traffic light area Q from the visible light camera coordinate system Sc1 to the vehicle coordinate system Sv is expressed as a combination of a rotation matrix representing rotation between coordinate systems and a parallel matrix representing translation between coordinate systems.

[0061] The rotation matrix is ​​represented based on the orientation of the visible light camera 2 on the vehicle 10 when the vehicle 10 is located on a level ground, and the yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 when the visible light image P1 is acquired. The parallel matrix is ​​represented based on the mounting position of the visible light camera 2 on the vehicle 10 when the vehicle 10 is located on a level ground. The rotation matrix is ​​determined for each process based on the yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 when the visible light image P1 is acquired.

[0062] The change in the orientation of the optical axis of the visible light camera 2 is represented by changes in the yaw angle θc1, pitch angle ψc1, and roll angle φc1. The yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 change in accordance with changes in the yaw angle θv, pitch angle ψv, and roll angle φv of the vehicle 10.

[0063] Furthermore, from the viewpoint of reducing the computational load in the determination unit 233, the determination unit 233 may convert the signal area Q to a position represented in the vehicle coordinate system Sv based on representative values ​​of the yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 when the visible light image P1 is acquired. This allows the use of a pre-prepared rotation matrix. The representative values ​​of the yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 can be set based on the attitudes that the vehicle 10 can take during normal driving.

[0064] Next, the determination unit 233 converts the signal area Q, represented in the vehicle coordinate system Sv, into a position represented in the three-dimensional infrared camera coordinate system Sc2, with the infrared camera 3 as the origin Oc2 (step S106).

[0065] Figure 5 illustrates the infrared camera coordinate system Sc2. In the infrared camera coordinate system Sc2, the Zc2 axis is set in the direction of the optical axis of the infrared camera 3, the Xc2 axis is set perpendicular to the Zc2 axis and parallel to the ground, and the Yc2 axis is set perpendicular to the Zc2 axis and Xc2 axis. The origin Oc2 is at the height from the ground where the infrared camera 3 is installed.

[0066] The transformation formula for converting the position of the traffic signal area Q from the vehicle coordinate system Sv to the infrared camera coordinate system Sc2 is expressed as a combination of a rotation matrix representing rotation between coordinate systems and a parallel matrix representing translation between coordinate systems.

[0067] The rotation matrix is ​​represented based on the orientation of the infrared camera 3 on the vehicle 10 when the vehicle 10 is located on a level ground, and the yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 when the infrared image P2 is acquired. The parallel matrix is ​​represented based on the mounting position of the infrared camera 3 on the vehicle 10 when the vehicle 10 is located on a level ground. The rotation matrix is ​​determined for each process based on the yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 when the infrared image P2 is acquired.

[0068] The change in the orientation of the optical axis of the infrared camera 3 is represented by changes in the yaw angle θc2, pitch angle ψc2, and roll angle φc2. The yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 change in accordance with changes in the yaw angle θv, pitch angle ψv, and roll angle φv of the vehicle 10.

[0069] Furthermore, from the viewpoint of reducing the computational load in the determination unit 233, the determination unit 233 may convert the signal area Q to a position represented in the infrared camera coordinate system Sc2 based on representative values ​​of the yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 when the infrared image P2 is acquired. This allows the use of a pre-prepared rotation matrix. The representative values ​​of the yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 can be set based on the attitudes that the vehicle 10 can take during normal driving.

[0070] Next, the determination unit 233 projects the signal area Q, represented in the infrared camera coordinate system Sc2, onto the infrared image P2, and determines the signal area Q projected onto the infrared image P2 as the corresponding area R within the infrared image P2 (step S107).

[0071] As shown in Figure 5, the determination unit 233 determines the signal area Q projected onto the infrared image P2 as a set of intersection points between the infrared image P2, whose center is perpendicular to the Zc2 axis and positioned at the focal distance of the infrared camera 3 from the origin Oc2, and a straight line connecting the origin Oc2 of the infrared camera coordinate system Sc2 and a point representing the boundary of the signal area Q. This signal area Q is then obtained as the corresponding area R. Note that in Figure 5, for the sake of clarity, the center of the infrared image P2 is shown to be perpendicular to the Zc2 axis but offset from it.

[0072] Next, the detection unit 231 detects the housing of the traffic light represented within the corresponding region R of the infrared image P2 and terminates the series of processes (step S108). The detection of the housing 32 of the traffic light 30 represented within the corresponding region R means that the traffic light 30 has been detected in front of the vehicle 10. It is preferable that the detection unit 231 detects the housing of the traffic light represented within the corresponding region R using the signal representing the unstandardized infrared image P2 acquired using the infrared camera 3. This is to prevent the problem that when the brightness of pixels in the infrared image P2 is standardized to a range of 0 to 255, pixels with low brightness become difficult to detect as pixels representing the housing of the traffic light.

[0073] The detection unit 231 performs edge processing such as the Canny method on the corresponding region R, and then detects straight lines using the Hough transform. If two orthogonal straight lines are detected within the corresponding region R, the detection unit 231 determines that the housing of a traffic light for the lane in which the vehicle 10 is traveling has been detected within the corresponding region R. The two orthogonal straight lines correspond to a vertical straight section and a horizontal straight section that define the housing of the traffic light.

[0074] Furthermore, if the detection unit 231 detects two straight lines that intersect but are not orthogonal, it determines that a traffic light housing for a lane different from the lane in which the vehicle 10 is traveling has been detected within the corresponding region R. In addition, if the detection unit 231 does not detect two straight lines that are orthogonal or intersecting, it determines that no traffic light housing has been detected within the corresponding region R.

[0075] In step S102, a guardrail reflector or streetlamp may be detected as a light area. If a guardrail reflector or streetlamp is represented in the light area, there is no risk of detecting two orthogonal lines in the corresponding area R. Therefore, the detection unit 231 prevents the guardrail reflector or streetlamp from being detected as a traffic light.

[0076] Furthermore, even if the resolution of the infrared image acquired by the infrared camera 3 is low, it is easy to detect two orthogonal lines in the corresponding region R, so the detection device 12 can accurately detect the traffic light housing.

[0077] When the detection unit 231 detects a traffic light housing in the lane the vehicle 10 is traveling in, it notifies the control device 11 of the color of the light.

[0078] The control device 11 may, for example, provide driving assistance by slowing down the vehicle 10 when the traffic light is red. Alternatively, the control device 11 may notify the driver via the UI 5 to start the vehicle 10 when the traffic light changes from red to green.

[0079] As detailed above, the detection device of this embodiment can detect traffic signals even in low-light environments using inexpensive sensors such as visible light cameras and infrared cameras.

[0080] Furthermore, the detection device of this embodiment can accurately determine the position of the corresponding area in the infrared image by converting the signal area of ​​the visible light image from the visible light image coordinate system to the vehicle coordinate system, and then converting it to the infrared image coordinate system to determine the corresponding area.

[0081] Next, modified examples of the detection device disclosed herein will be described below with reference to Figure 6.

[0082] Figure 6 is an example of an operation flowchart relating to a modified version of the detection process of the detection device of the embodiment described above. In this modified version, steps S104 to S106 are omitted from the operation flowchart of the detection process shown in Figure 3 described above. The processes of steps S201 to S203 and S205 are the same as steps S101 to S103 and S108 described above. The process of step S204 is different from the process of S107 described above.

[0083] In this modified example, the determination unit 233 determines the corresponding region R, represented in the infrared camera coordinate system Sc2 of the infrared image P2, which corresponds to the signal region Q, represented in the visible light camera coordinate system Sc1 of the visible light image P1 (step S204).

[0084] The determination unit 233 represents the signal area Q in the visible light camera coordinate system Sc1 of the visible light image P1 at the position in the infrared camera coordinate system Sc2 of the infrared image P2, and determines the signal area Q represented in the infrared camera coordinate system Sc2 as the corresponding area R in the infrared image P2.

[0085] The transformation formula for converting the signal area Q in the visible light camera coordinate system Sc1 of the visible light image P1 to the infrared camera coordinate system Sc2 of the infrared image P2 is expressed based on internal parameters including the focal length of the visible light camera 2 and the infrared camera 3, external parameters including their installation position and orientation, and representative values ​​of the yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 when the visible light image P1 was acquired, and representative values ​​of the yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 when the infrared image P2 was acquired.

[0086] Specifically, this transformation formula is expressed as a combination of a rotation matrix representing rotation between coordinate systems and a parallel matrix representing translation between coordinate systems.

[0087] The rotation matrix is ​​expressed based on the orientation of the visible light camera 2 on the vehicle 10 when the vehicle 10 is located on a horizontal ground, the representative values ​​of the yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 when the visible light image P1 is acquired, and the orientation of the infrared camera 3 on the vehicle 10 when the vehicle 10 is located on a horizontal ground, and the representative values ​​of the yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 when the infrared image P2 is acquired.

[0088] The parallel matrix is ​​represented based on the mounting position of the visible light camera 2 on the vehicle 10 when the vehicle 10 is located on a level ground, and the mounting position of the infrared camera 3 on the vehicle 10 when the vehicle 10 is located on a level ground.

[0089] In this specification, converting the signal area Q within the infrared image P2 to a corresponding area R based on the yaw angle θc1, pitch angle ψc1, and roll angle φc1 of the visible light camera 2 when the visible light image P1 was acquired, and the yaw angle θc2, pitch angle ψc2, and roll angle φc2 of the infrared camera 3 when the infrared image P2 was acquired, means determining the corresponding area R based on these angles and information other than these angles.

[0090] As described in the embodiments described above, a conversion formula for converting the signal area Q in the visible light camera coordinate system Sc1 of the visible light image P1 to the infrared camera coordinate system Sc2 of the infrared image P2 can be determined in advance using internal parameters including the focal length of the visible light camera 2 and the infrared camera 3, external parameters including the installation position and installation orientation, and representative values ​​of the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image is acquired. This conversion formula represents the process of converting the signal area of ​​the visible light image from the visible light image coordinate system to the vehicle coordinate system, and then from the vehicle coordinate system to the infrared image coordinate system.

[0091] Next, the detection unit 231 detects the housing of the traffic light represented within the corresponding area R of the infrared image P2 and terminates the series of processes (step S205).

[0092] As detailed above, in this modified detection device, the transformation formula for converting the signal area Q in the visible light camera coordinate system Sc1 of the visible light image P1 to the infrared camera coordinate system Sc2 of the infrared image P2 is predetermined, thus reducing the amount of processing required for detection. Although the accuracy of determining the corresponding area of ​​the infrared image is inferior to that of the embodiment described above, this modified detection device can sufficiently detect the signal housing even in low-light environments.

[0093] In this disclosure, the detection devices of the embodiments described above may be modified as appropriate without departing from the spirit of this disclosure. Furthermore, the technical scope of this disclosure is not limited to those embodiments, but extends to the invention described in the claims and its equivalents. [Explanation of Symbols]

[0094] 2. Visible light camera 3. Infrared camera 4 Angular velocity sensor 5. User Interface (UI) 5a Display device 10 vehicles 11 Control device 12 Detection device 21 Communication Interface 22 memory 23 processors 231 Detection unit 232 Settings Section 233 Decision Section 13. In-vehicle network

Claims

1. A first detection unit detects a light region representing a traffic light from a visible light image representing the environment in front of the vehicle, acquired using a visible light camera, A setting unit sets a signal area in the visible light image that is presumed to include the housing of a signal, based on the aforementioned lighting area. A determination unit determines a corresponding area within the infrared image that corresponds to the signal area, based on the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image is acquired, and the yaw angle, pitch angle, and roll angle of the infrared camera when an infrared image representing the environment in front of the vehicle is acquired using an infrared camera. A second detection unit detects the housing of a traffic signal within the aforementioned corresponding area, A detection device characterized by having the following.

2. The setting unit sets the signal area as a position represented in the visible light image coordinate system of the visible light image, The aforementioned determination unit, The signal area represented in the visible light image coordinate system is transformed into a position represented in a three-dimensional visible light camera coordinate system with the visible light camera as the origin. The signal area represented in the visible light camera coordinate system is transformed into a position represented in a three-dimensional vehicle coordinate system with the origin at the vehicle, based on the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image was acquired. The signal area represented in the aforementioned vehicle coordinate system is transformed into a position represented in a three-dimensional infrared camera coordinate system with the infrared s-line camera as the origin, based on the yaw angle, pitch angle, and roll angle of the infrared camera when the infrared image was acquired. The detection device according to claim 1, wherein the signal area represented in the infrared camera coordinate system is projected onto the infrared image, and the projected signal area is determined to be the corresponding area in the infrared image.

3. The aforementioned determination unit, The signal area represented in the visible light camera coordinate system is transformed to a position represented in the vehicle coordinate system based on representative values ​​of the yaw angle, pitch angle, and roll angle of the visible light camera at the time the visible light image was acquired, and, The detection device according to claim 2, which converts the signal area represented in the vehicle coordinate system to a position represented in the infrared camera coordinate system based on representative values ​​of the yaw angle, pitch angle, and roll angle of the infrared camera at the time the infrared image was acquired.

4. The setting unit sets the signal area as a position represented in the visible light image coordinate system of the visible light image, The detection device according to claim 1, wherein the determination unit determines the corresponding region of the infrared image, represented in the infrared image coordinate system of the infrared image, which corresponds to the signal area, based on the yaw angle, pitch angle, and roll angle of the visible light camera when the visible light image is acquired, and the yaw angle, pitch angle, and roll angle of the infrared camera when the infrared image is acquired.

5. The detection device according to any one of claims 1 to 3, wherein the second detection unit detects the housing of a traffic light within the corresponding area using a signal representing the unstandardized infrared image acquired using the infrared camera.

Images