Traffic light recognition method and traffic light recognition device

Abstract

To improve recognition performance of a traffic signal even when a plurality of cameras are switched to recognize the traffic signal.SOLUTION: A traffic signal recognition device 1 acquires a first image obtained by imaging the front area of a vehicle by first imaging means mounted on the vehicle and a second image obtained by imaging the front of the vehicle by second imaging means having a narrower angle of view and a higher magnification than those of the first imaging means, selects an image in which the distance from a position of a traffic signal appearing on the image to an image end is a predetermined value or more and the size of the traffic signal appearing on the image is the largest, among a plurality of images composed of the acquired first and second images, as a processing image, and based on the selected processing image, recognizes the light of the traffic signal that is lit in front of the vehicle.SELECTED DRAWING: Figure 1

Description

【Technical Field】 【0001】 The present invention relates to a traffic signal recognition method and apparatus for recognizing the lights of a traffic signal that is lit in front of a vehicle. 【Background Art】 【0002】 Conventionally, a vehicle front monitoring device that monitors the front of a vehicle using a narrow-angle camera and a wide-angle camera is disclosed in Patent Document 1. In the vehicle front monitoring device disclosed in Patent Document 1, depending on the relative distance between the vehicle and an object, the narrow-angle camera and the wide-angle camera are switched to recognize the object in front of the vehicle. 【Prior Art Documents】 【Patent Documents】 【0003】 【Patent Document 1】 Japanese Patent Application Laid-Open No. 2011-121398 【Summary of the Invention】 【Problems to be Solved by the Invention】 【0004】 However, in the conventional vehicle front monitoring device described above, since the narrow-angle camera and the wide-angle camera are switched according to the relative distance between the vehicle and the object, there is a problem that the recognition performance of the object deteriorates immediately after switching the narrow-angle camera and the wide-angle camera. For example, immediately after switching from the narrow-angle camera to the wide-angle camera, even though the object is imaged large by the narrow-angle camera, the object is recognized by the wide-angle camera, so the object is imaged small and the recognition performance of the object deteriorates. 【0005】 Therefore, the present invention has been proposed in view of the above circumstances, and an object thereof is to provide a traffic signal recognition method and apparatus capable of improving the recognition performance of a traffic signal even when recognizing the traffic signal by switching a plurality of cameras. 【Means for Solving the Problems】 【0006】 To solve the above-mentioned problems, a traffic light recognition method and apparatus according to one aspect of the present invention acquires a first image captured of the area in front of the vehicle by a first imaging means mounted on the vehicle, and a second image captured of the area in front of the vehicle by a second imaging means having a narrower field of view and a higher magnification than the first imaging means. Then, from among the multiple images consisting of the acquired first and second images, the image in which the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, and in which the size of the traffic light captured on the image is the largest, is selected as the processing image. As a result, the traffic light recognition method and apparatus according to one aspect of the present invention recognizes the lights of the traffic lights illuminated in front of the vehicle based on the selected processing image. [Effects of the Invention] 【0007】 According to the present invention, even when multiple cameras are switched to recognize traffic signals, the recognition performance of traffic signals can be improved. [Brief explanation of the drawing] 【0008】 [Figure 1] Figure 1 is a block diagram showing the configuration of a vehicle system equipped with a traffic signal recognition device according to the first embodiment. [Figure 2] Figure 2 is a flowchart showing the processing procedure for traffic signal recognition by the traffic signal recognition device according to the first embodiment. [Figure 3] Figure 3 is a flowchart showing the processing procedure for image selection by the signal recognition device according to the first embodiment. [Figure 4] Figure 4 is a diagram illustrating the method for selecting a processed image by the signal recognition device according to the first embodiment. [Figure 5] Figure 5 is a diagram illustrating the method for determining an area outside the field of view by the traffic signal recognition device according to the first embodiment. [Figure 6] Figure 6 is a diagram illustrating the method for determining whether an object is outside the field of view by the traffic signal recognition device according to the first embodiment. [Figure 7]Figure 7 is a diagram illustrating the method for determining whether a traffic signal is obstructed by the traffic signal recognition device according to the first embodiment. [Figure 8] Figure 8 is a diagram illustrating the method for selecting a processed image by the signal recognition device according to the first embodiment. [Figure 9A] Figure 9A is a diagram illustrating the method for selecting a processed image by the signal recognition device according to the first embodiment. [Figure 9B] Figure 9B is a diagram illustrating the method for selecting a processed image by the signal recognition device according to the first embodiment. [Figure 10A] Figure 10A is a diagram illustrating the method for determining the obstruction of a traffic signal by a traffic signal recognition device according to the first embodiment. [Figure 10B] Figure 10B is a diagram illustrating the method for determining whether a traffic signal is obstructed by a traffic signal recognition device according to the first embodiment. [Figure 11] Figure 11 is a flowchart showing the processing procedure for image selection by the signal recognition device according to the second embodiment. [Modes for carrying out the invention] 【0009】 [First Embodiment] A first embodiment to which the present invention is applied will be described below with reference to the drawings. In the drawings, the same parts are denoted by the same reference numerals, and detailed descriptions are omitted. 【0010】 [Vehicle System Configuration] Figure 1 is a block diagram showing the configuration of a vehicle system equipped with a traffic signal recognition device according to this embodiment. As shown in Figure 1, the vehicle system 100 includes a traffic signal recognition device 1, a radar 20, a Lidar 30, an object recognition camera 40, a wide-angle camera 50, a narrow-angle camera 60, a map database 70, a GPS receiver 80, and a vehicle control device 90. The vehicle system 100 is installed in vehicles capable of autonomous driving or vehicles that provide driving assistance to the driver. 【0011】 The wide-angle camera (first imaging means) 50 and the narrow-angle camera (second imaging means) 60 are both cameras that image the front of the vehicle. The narrow-angle camera 60 has a narrower angle of view and a higher magnification than the wide-angle camera 50, and the imaging range of the narrow-angle camera 60 is included in the imaging range of the wide-angle camera 50. 【0012】 The traffic signal recognition device 1 is a device that recognizes the lights of a traffic signal that is lit in front of the vehicle from the images captured by the wide-angle camera 50 and the narrow-angle camera 60. Therefore, the traffic signal recognition device 1 recognizes, by performing image processing on the images captured by the wide-angle camera 50 and the narrow-angle camera 60, whether the lights of the traffic signal that is currently lit in front of the vehicle are blue, red, or yellow, and outputs the recognition result to the vehicle control device 90. 【0013】 The radar 20 and the lidar (Light Detection and Ranging) 30 are sensors mounted on the vehicle that detect objects existing around the vehicle, and measure the distance to each point on the detected object. The object recognition camera 40 is an imaging means mounted on the vehicle that images the surroundings of the vehicle. The detected distance and the captured image are output to the traffic signal recognition device 1 and recorded in a storage device such as a memory (not shown). 【0014】 The wide-angle camera 50 is an imaging means mounted on the vehicle that images the front of the vehicle. The narrow-angle camera 60 is an imaging means mounted on the vehicle that images the front of the vehicle, and captures an image with a narrower angle of view and a higher magnification than the wide-angle camera 50. The captured image is output to the traffic signal recognition device 1 and recorded in a storage device such as a memory (not shown). 【0015】 The map database 70 is mounted on the vehicle and stores map information. The map information is high-precision three-dimensional map data such as an HD map, and includes, in addition to the position information on the map of three-dimensional objects such as traffic signals and utility poles provided on roads and sidewalks or lane boundary lines, road structure information such as the number of lanes of the road, road boundary lines, and stop lines. 【0016】 The GPS receiver 80 detects the vehicle's current location on the ground by receiving radio waves from artificial satellites. The detected data is output to the traffic signal recognition device 1 and recorded in a storage device such as a memory (not shown). 【0017】 The vehicle control device 90 is a device that acquires the recognition result of a traffic light from the traffic light recognition device 1 and controls the vehicle's movement. Specifically, when the vehicle is in autonomous driving mode, the vehicle control device 90 acquires the recognition result of the traffic light ahead of the vehicle from the traffic light recognition device 1, controls the vehicle's brakes to stop the vehicle if the traffic light is red, and controls the accelerator to pass the traffic light if the traffic light is green. Furthermore, when the vehicle is a vehicle that assists the driver, the vehicle control device 90 informs the driver of the recognition result of the traffic light ahead of the vehicle acquired from the traffic light recognition device 1. 【0018】 As shown in Figure 1, the traffic light recognition device 1 comprises an object position detection unit 3, a traffic light position acquisition unit 5, a self-position acquisition unit 7, an image selection unit 9, a traffic light recognition unit 11, and an output unit 13. 【0019】 The object position detection unit 3 detects the position of objects around the vehicle from the distance to the object measured by the radar 20 and Lidar 30 and the image captured by the object recognition camera 40. In particular, the object position detection unit 3 detects the position, width, height, etc., of a preceding vehicle traveling in front of the vehicle. 【0020】 The traffic light position acquisition unit 5 acquires the position of traffic lights located in front of the vehicle from the map information stored in the map database 70. Specifically, the traffic light position acquisition unit 5 places the vehicle's own position acquired by the self-position acquisition unit 7 onto the high-precision map recorded in the map information, and, considering the direction of travel of the vehicle, acquires the position of traffic lights located in front of the vehicle as recorded in the map information. At this time, if the height of the traffic lights is recorded in the map information, the traffic light position acquisition unit 5 acquires the height of the traffic lights from the map information. If the height of the traffic lights is not recorded in the map information, it uses the height of the traffic lights stipulated by law. 【0021】 The self-position acquisition unit 7 obtains the vehicle's current position received by the GPS receiver 80 and odometry calculated using the vehicle's sensor values, and compares this with map information stored in the map database 70 to obtain the vehicle's own position. At this time, the self-position acquisition unit 7 obtains a highly accurate self-position that identifies the lane the vehicle is traveling in and its position within an intersection. 【0022】 The image selection unit 9 acquires multiple images of the area in front of the vehicle from multiple imaging means mounted on the vehicle, and selects a processing image from the acquired multiple images. The acquired multiple images each have different field of view and different magnification ratios. In this embodiment, the image from the wide-angle camera 50 (first image) and the image from the narrow-angle camera 60 (second image) are acquired, and a processing image is selected from the acquired images from the wide-angle camera 50 and the narrow-angle camera 60. 【0023】 Specifically, the image selection unit 9 selects the image from among the acquired multiple images in which the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, and in which the size of the traffic light captured on the image is the largest, as the processing image. In other words, from the images captured by the wide-angle camera 50 and the narrow-angle camera 60, the image in which the captured traffic light is not outside the field of view, and in which the size of the captured traffic light is the largest, is selected as the processing image. 【0024】 Furthermore, the image selection unit 9 sets a processing area on multiple images, according to the size of the traffic lights captured on the images, at the location of the traffic lights captured on the images. This processing area is a region of interest (ROI) set on the images captured by the wide-angle camera 50 and the narrow-angle camera 60. The specific method for setting the processing area will be described later. In this case, the image selection unit 9 selects the image among the multiple images in which the distance from the processing area to the edge of the image is greater than or equal to a predetermined value, and in which the size of the processing area is the largest, as the processing image. That is, among the images captured by the wide-angle camera 50 and the narrow-angle camera 60, the image in which the set processing area is not outside the field of view, and in which the processing area is the largest, is selected as the processing image. 【0025】 Furthermore, the image selection unit 9 determines whether the traffic lights captured in multiple images are obscured by objects surrounding the vehicle. Specifically, the image selection unit 9 detects objects surrounding the vehicle and determines whether the traffic lights captured in multiple images are obscured by the detected objects. Then, from among the multiple images, it selects the image in which the traffic lights captured are not obscured as the processing image. 【0026】 The traffic light recognition unit 11 recognizes the lights of traffic lights illuminated in front of the vehicle based on the processed image selected by the image selection unit 9. Specifically, the traffic light recognition unit 11 uses the image from the wide-angle camera 50 or the narrow-angle camera 60 selected by the image selection unit 9 to recognize whether the lights of the traffic lights illuminated in front of the vehicle are red, blue, or yellow. In particular, if a processing area is set on the image, the traffic light recognition unit 11 detects traffic lights within the set processing area and recognizes the lights of the traffic lights illuminated in front of the vehicle. 【0027】 The output unit 13 outputs the lights of the traffic signals recognized by the traffic signal recognition unit 11 to the vehicle control device 90. 【0028】 The traffic signal recognition device 1 is a controller composed of a general-purpose electronic circuit including a microcomputer, microprocessor, and CPU, and peripheral devices such as memory, and has a computer program installed for performing traffic signal recognition processing. Each function of the traffic signal recognition device 1 can be implemented by one or more processing circuits. The processing circuits include, for example, programmed processing devices including electrical circuits, and may also include devices such as application-specific integrated circuits (ASICs) or conventional circuit components arranged to perform the functions described in the embodiments. 【0029】 [Traffic light recognition processing] Next, the traffic signal recognition process performed by the traffic signal recognition device 1 according to this embodiment will be described. Figure 2 is a flowchart showing the processing procedure of the traffic signal recognition process by the traffic signal recognition device 1 according to this embodiment. 【0030】 As shown in Figure 2, in step S101, the image selection unit 9 acquires images captured by the narrow-angle camera 60 at predetermined intervals. Also, in step S103, the image selection unit 9 acquires images captured by the wide-angle camera 50 at predetermined intervals. 【0031】 In step S105, the self-position acquisition unit 7 acquires the vehicle's current position received by the GPS receiver 80 and odometry calculated using the vehicle's sensor values, and compares it with map information stored in the map database 70 to acquire the vehicle's own position. The acquired self-position information is output to the image selection unit 9. 【0032】 In step S107, the signal position acquisition unit 5 places the vehicle's own position, acquired in step S105, onto the high-precision map recorded in the map information, and, considering the vehicle's direction of travel, acquires the position of the signal located in front of the vehicle as recorded in the map information. At this time, if the height of the signal is recorded in the map information, the height of the signal is also acquired from the map information. The acquired signal position information is output to the image selection unit 9. 【0033】 In step S109, the object position detection unit 3 detects the position, width, height, etc., of the preceding vehicle traveling in front of the vehicle from the distance to the object measured by the radar 20 and Lidar 30 and the image captured by the object recognition camera 40. The detected object position information is output to the image selection unit 9. 【0034】 In step S111, the image selection unit 9 selects a processing image from among multiple images captured by multiple cameras mounted on the vehicle. In this embodiment, the processing image is selected from the image of the wide-angle camera 50 and the image of the narrow-angle camera 60. Specifically, the image selection unit 9 selects the processing image by executing the processing image selection process shown in Figure 3. 【0035】 Figure 3 is a flowchart showing the processing procedure for image selection by the traffic light recognition device 1 according to this embodiment. As shown in Figure 3, in step S201, the image selection unit 9 sets a processing area in the images of the narrow-angle camera 60 and the wide-angle camera 50. The image selection unit 9 determines the relative position from the vehicle's own position acquired in step S105 and the position of the traffic light acquired in step S107, and identifies the position of the traffic light captured on the images of the narrow-angle camera 60 and the wide-angle camera 50 based on the determined relative position. Then, the image selection unit 9 calculates the size of the traffic light captured on the image based on the distance to the traffic light, and sets a processing area in the images of the narrow-angle camera 60 and the wide-angle camera 50 at the position where the traffic light is captured, according to the size of the captured traffic light. 【0036】 For example, as shown in Figure 4, the image selection unit 9 sets two processing areas RA1 and RA2 on the image from the narrow-angle camera 60. Similarly, the image selection unit 9 sets two processing areas RB1 and RB2 on the image from the wide-angle camera 50. These processing areas are not set by detecting traffic lights in the image, but rather by estimating the position of the traffic lights in the image based on the vehicle's own position and the position of the traffic lights. The size of the processing area is also not set based on the size of the traffic lights in the image, but rather by estimating the size of the traffic lights in the image based on the distance from the vehicle to the traffic lights. Furthermore, the size of the processing area is set larger than the actual size of the traffic lights. For example, if the actual size of the traffic lights is 30cm x 120cm, it is set to 300cm x 600cm by multiplying the vertical dimension by 10 and the horizontal dimension by 5. By setting the processing area larger than the actual size, even if there are errors in the vehicle's own position or the position of the traffic lights, the traffic lights can be reliably included within the processing area. 【0037】 However, the size of the processing area does not always have to be larger than the actual size of the traffic light. For example, if the vehicle's own position and the position of the traffic light can be accurately estimated, the processing area may be set to the same size as the actual traffic light. In this case, the processing area will be set to the position of the traffic light on the image. 【0038】 In step S203, the image selection unit 9 ranks the images from the narrow-angle camera 60 and the wide-angle camera 50 in descending order of processing area size. For example, in the image from the narrow-angle camera 60 in Figure 4, processing area RA1 is larger than processing area RA2, so the ranking is done in the order of processing area RA1, then processing area RA2. Similarly, in the image from the wide-angle camera 50, processing area RB1 is larger than processing area RB2, so the ranking is done in the order of processing area RB1, then processing area RB2. 【0039】 In step S205, the image selection unit 9 determines whether each processing area is outside the field of view. Specifically, the image selection unit 9 determines that a traffic light captured on the image is outside the field of view if the distance from the position of the traffic light captured on the image to the edge of the image is less than a predetermined value. However, in this embodiment, since a processing area is set, the processing area may be determined to be outside the field of view if the distance from the processing area to the edge of the image is less than a predetermined value. On the other hand, if the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, the image selection unit 9 determines that the traffic light captured on the image is not outside the field of view. However, in this embodiment, since a processing area is set, the processing area may be determined to be not outside the field of view if the distance from the processing area to the edge of the image is greater than or equal to a predetermined value. 【0040】 For example, as shown in Figure 5, if the distance L from the position X of the traffic light captured on the image to the nearest edge of the image is less than a predetermined value, and the processing area R extends beyond the image, it is determined that the processing area R is outside the field of view. On the other hand, if the distance L from the position X of the traffic light to the nearest edge of the image is greater than or equal to a predetermined value, and the processing area R does not extend beyond the image, it is determined that the processing area R is not outside the field of view. 【0041】 The predetermined value may be set to the length from the position X of the traffic light to the processing area R, or to the length from the position X of the traffic light to the edge of the traffic light. In other words, it may be determined that the area is out of the field of view if the processing area R extends beyond the image, or it may be determined that the area is out of the field of view if the captured traffic light extends beyond the image. 【0042】 Furthermore, the method for determining whether or not an area is outside the field of view may be based on whether or not the proportion of the processing area included in the image is less than a predetermined value. For example, as shown in Figure 6, if area Rin of the processing area R is included in the image, it is determined to be outside the field of view if the proportion of area Rin to processing area R is less than a predetermined value. For example, it is determined to be outside the field of view if the proportion of area Rin is less than 70%. On the other hand, if the proportion of area Rin to processing area R is equal to or greater than the predetermined value, it is determined not to be outside the field of view. 【0043】 In step S207, the image selection unit 9 determines whether or not the traffic lights captured in the image are obscured. For example, as shown in Figure 7, the image selection unit 9 sets an obscuring area S based on the position, width, and height of the preceding vehicle detected in step S109. Then, if the percentage of the processing area R overlapping with the obscuring area S is greater than or equal to a predetermined value, it is determined that the traffic lights captured in the image are obscured. On the other hand, if the percentage of the processing area R overlapping with the obscuring area S is less than a predetermined value, it is determined that the traffic lights captured in the image are not obscured. 【0044】 In step S209, the image selection unit 9 selects a processing image from the images of the wide-angle camera 50 and the narrow-angle camera 60. Specifically, the image selection unit 9 selects the image as the processing image in which the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, and in which the size of the traffic light captured on the image is the largest. However, in this embodiment, since a processing area is set, the image is selected as the processing image in which the distance from the processing area to the edge of the image is greater than or equal to a predetermined value, and in which the size of the processing area is the largest. 【0045】 For example, as shown in Figure 8, in the T-1 frame, the distance from the processing area RA of the narrow-angle camera 60 to the edge of the image is a predetermined value, for example, 0 or greater, so both processing areas RA and RB are included within the field of view. Also, since the processing area RA of the narrow-angle camera 60 is larger than the processing area RB of the wide-angle camera 50, the image selection unit 9 selects the image from the narrow-angle camera 60 as the processing image. 【0046】 Next, in the T-frame, the processing area RA of the narrow-angle camera 60 is determined to be outside the field of view because the distance from the processing area to the edge of the image is less than a predetermined value, for example, 0. Therefore, the image selection unit 9 selects the image from the wide-angle camera 50 as the processing image. 【0047】 Next, we will explain the case where multiple traffic lights are captured in the images from the wide-angle camera 50 and the narrow-angle camera 60, referring to Figures 9A and 9B. As shown in Figure 9A, in the T-3 frame, multiple processing areas RA1 and RA2 are set in the image from the narrow-angle camera 60, and multiple processing areas RB1 and RB2 are set in the image from the wide-angle camera 50. Since the distance from the processing area to the edge of the image is greater than or equal to a predetermined value for all of the processing areas RA1, RA2, RB1, and RB2, all of these processing areas are included within the field of view. 【0048】 Furthermore, since the processing area sizes are ranked in step S203, for the image from the narrow-angle camera 60, processing area RA1 is larger than processing area RA2, and for the image from the wide-angle camera 50, processing area RB1 is larger than processing area RB2. Therefore, comparing the processing area RA1 of the narrow-angle camera 60 with the processing area RB1 of the wide-angle camera 50, the image selection unit 9 selects the image from the narrow-angle camera 60 as the processing image because the processing area RA1 of the narrow-angle camera 60 is larger. 【0049】 Next, in the T-2 frame, the processing area RA1 of the narrow-angle camera 60 is determined to be outside the field of view because the distance from the processing area to the edge of the image is less than a predetermined value. On the other hand, the processing areas RA2, RB1, and RB2 are all within the field of view because the distance from the processing area to the edge of the image is greater than or equal to a predetermined value. Also, in the image from the wide-angle camera 50, the processing area RB1 is larger than the processing area RB2, so the processing area RA2 of the narrow-angle camera 60 and the processing area RB1 of the wide-angle camera 50 are compared. As a result, the processing area RB1 of the wide-angle camera 50 is larger, so the image selection unit 9 selects the image from the wide-angle camera 50 as the image to be processed. 【0050】 Next, in the T-1 frame, as shown in Figure 9B, the processing area RA1 of the narrow-angle camera 60 is not set, and the processing area RB1 of the wide-angle camera 50 is determined to be outside the field of view because the distance from the processing area to the edge of the image is less than a predetermined value. On the other hand, the processing areas RA2 and RB2 are both within the field of view because the distance from the processing area to the edge of the image is greater than or equal to a predetermined value. Therefore, the processing area RA2 of the narrow-angle camera 60 and the processing area RB2 of the wide-angle camera 50 are compared, and since the processing area RA2 of the narrow-angle camera 60 is larger, the image selection unit 9 selects the image from the narrow-angle camera 60 as the processing image. 【0051】 Next, in the T-frame, the processing area RA1 of the narrow-angle camera 60 and the processing area RB1 of the wide-angle camera 50 are not set. The processing area RA2 of the narrow-angle camera 60 is determined to be outside the field of view because the distance from the processing area to the edge of the image is less than a predetermined value. On the other hand, the processing area RB2 is within the field of view because the distance from the processing area to the edge of the image is greater than or equal to a predetermined value. Therefore, the image selection unit 9 selects the image from the wide-angle camera 50 as the processing image. 【0052】 Next, the case where the traffic light captured in the image is obscured will be explained with reference to Figures 10A and 10B. As shown in Figure 10A, in the T-2 frame, a processing area RA is set in the image from the narrow-angle camera 60, and a processing area RB is set in the image from the wide-angle camera 50. In addition, an obscuring area S is set in the images from both the narrow-angle camera 60 and the wide-angle camera 50 at the position of the preceding vehicle. In the image from the narrow-angle camera 60, the proportion of the overlap between the processing area RA and the obscuring area S is greater than or equal to a predetermined value, so it is determined that the traffic light captured in the image is obscured. On the other hand, in the image from the wide-angle camera 50, the proportion of the overlap between the processing area RB and the obscuring area S is less than a predetermined value, so it is determined that the traffic light captured in the image is not obscured. As a result, the image selection unit 9 selects the image from the wide-angle camera 50 as the processing image. In this way, even if the traffic light in front of the vehicle is obscured by an object such as a preceding vehicle, the camera can be switched and the image of the traffic light on the opposing vehicle's side can be used to recognize the traffic light's illumination. 【0053】 Next, in the T-1 frame, the image from the narrow-angle camera 60 shows that the processing area RA overlaps with the occluding area S to less than a predetermined value, so it is determined that the traffic lights captured in the image are not occluded. On the other hand, the image from the wide-angle camera 50 shows that the processing area RB overlaps with the occluding area S to more than a predetermined value, so it is determined that the traffic lights captured in the image are occluded. As a result, the image selection unit 9 selects the image from the narrow-angle camera 60 as the processing image. 【0054】 Next, in the T-frame, as shown in Figure 10B, in the image from the narrow-angle camera 60, the proportion of the processing area RA overlapping with the occluding area S is less than a predetermined value, so it is determined that the traffic lights captured in the image are not occluded. However, the distance from the processing area to the edge of the image is less than a predetermined value, so it is determined that the processing area RA is outside the field of view. On the other hand, in the image from the wide-angle camera 50, the proportion of the processing area RB overlapping with the occluding area S is less than a predetermined value, so it is determined that the traffic lights captured in the image are not occluded. As a result, the image selection unit 9 selects the image from the wide-angle camera 50 as the processing image. 【0055】 In this way, once a processing image is selected in step S209, the processing image selection process ends, and the system returns to the flowchart in Figure 2. In step S113 of the flowchart in Figure 2, the traffic light recognition unit 11 recognizes the lights of the traffic lights illuminated in front of the vehicle based on the processing image selected in step S111. Specifically, the traffic light recognition unit 11 uses the image from the wide-angle camera 50 or the image from the narrow-angle camera 60 selected as the processing image to detect the traffic lights within the processing area set on the image. It then recognizes whether the lights of the traffic lights illuminated in front of the vehicle are red, blue, or yellow. The method for detecting the traffic lights can be template matching or detection using machine learning. 【0056】 In step S115, the output unit 13 outputs the recognition result of the traffic light lights recognized in step S113 to the vehicle control device 90. When the vehicle control device 90 obtains the recognition result of the traffic light lights, if it is an autonomous vehicle, it controls the brakes of the vehicle to stop the vehicle when the traffic light is red, and controls the accelerator to pass the traffic light when the light is green. On the other hand, if it is a vehicle that provides driving assistance to a driver, the vehicle control device 90 notifies the driver of the recognition result of the traffic light lights. Once the recognition result of the traffic light lights is output to the vehicle control device 90, the traffic light recognition process according to this embodiment is completed. 【0057】 [Effects of the First Embodiment] As described in detail above, the traffic light recognition device 1 according to this embodiment acquires a first image captured by a first imaging means mounted on the vehicle, capturing the area in front of the vehicle, and a second image captured by a second imaging means having a narrower field of view and higher magnification than the first imaging means, capturing the area in front of the vehicle. Then, from among the multiple images consisting of the first and second images, the image in which the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, and the image in which the traffic light captured is the largest, is selected as the processing image, and the lights of the traffic light are recognized based on the selected processing image. As a result, the captured traffic light is not outside the field of view, and the image in which the captured traffic light is the largest is used for recognition, so the recognition performance of traffic lights can be improved even when switching between multiple cameras to recognize traffic lights. In addition, since the process of recognizing the lights of the traffic light is performed after narrowing it down to a single image, the processing time can be shortened. 【0058】 Furthermore, in the traffic light recognition device 1 according to this embodiment, a processing area is set on multiple images at the location of the traffic light captured in the image, according to the size of the traffic light captured in the image, and the lights of the traffic light are recognized within the set processing area. As a result, the lights of the traffic light are recognized within the processing area set at the location of the traffic light, so false detection of traffic lights can be prevented. 【0059】 Furthermore, in the traffic light recognition device 1 according to this embodiment, among multiple images, the image in which the distance from the processing area to the edge of the image is greater than or equal to a predetermined value, and which has the largest processing area, is selected as the processing image. By using the processing area to determine whether the captured traffic light is outside the field of view and using the processing area to determine the largest traffic light, the selection of the processing image can be easily performed. As a result, the recognition performance of traffic lights can be stabilized. 【0060】 Furthermore, in the traffic light recognition device 1 according to this embodiment, among multiple images, the image in which the processing area is included within the image is greater than or equal to a predetermined value, and which has the largest processing area, is selected as the processing image. By using the processing area to determine whether the captured traffic light is outside the field of view and using the processing area to determine the largest traffic light, the selection of the processing image can be easily performed. As a result, the recognition performance of traffic lights can be stabilized. 【0061】 Furthermore, in the traffic light recognition device 1 according to this embodiment, it is determined whether or not the traffic lights captured in multiple images are obscured by objects surrounding the vehicle, and from among the multiple images, the image in which the traffic lights are not obscured is selected as the processing image. As a result, even if the traffic lights in front of the vehicle are obscured by an object such as a preceding vehicle, the camera can be switched and the image of the traffic lights on the opposing vehicle's side can be used to recognize the lights of the traffic lights. Therefore, the recognition performance of traffic lights can be improved. 【0062】 [Second Embodiment] A second embodiment to which the present invention is applied will be described below with reference to the drawings. In the drawings, the same parts are denoted by the same reference numerals, and detailed descriptions are omitted. In this embodiment of the traffic signal recognition device 1, only the processing image selection process differs from that of the first embodiment; the configuration of the traffic signal recognition device 1 and the traffic signal recognition process are the same as those of the first embodiment. Therefore, the traffic signal recognition device 1 in this embodiment has the same configuration as shown in Figure 1 and performs the same processing as the traffic signal recognition process shown in Figure 2. 【0063】 In step S111 of Figure 2, the image selection unit 9 selects a processing image from multiple images acquired from multiple cameras mounted on the vehicle. In this embodiment, the processing image is selected from the image of the wide-angle camera 50 and the image of the narrow-angle camera 60. Specifically, the image selection unit 9 selects the processing image by executing the processing image selection process shown in Figure 11. 【0064】 Figure 11 is a flowchart showing the processing procedure for image selection by the traffic light recognition device 1 according to this embodiment. As shown in Figure 11, in step S301, the image selection unit 9 detects traffic lights in the images from the narrow-angle camera 60 and the wide-angle camera 50 by, for example, pattern matching which detects a location in the image that matches the shape of a traffic light that has been stored in advance, and calculates the position and size of the detected traffic lights in the image. Note that the size of the traffic lights is not determined by detecting the size of the traffic lights in the image, but is calculated by estimating the size of the captured traffic lights in the image based on the distance from the vehicle to the traffic lights. 【0065】 In step S303, the image selection unit 9 ranks the images from the narrow-angle camera 60 and the wide-angle camera 50 in descending order of the size of the traffic lights. 【0066】 In step S305, the image selection unit 9 determines whether the captured traffic light is outside the field of view. Specifically, the image selection unit 9 determines that the traffic light is outside the field of view if the distance from the position of the captured traffic light on the image to the edge of the image is less than a predetermined value. On the other hand, if the distance from the position of the captured traffic light on the image to the edge of the image is greater than or equal to a predetermined value, the image selection unit 9 determines that the traffic light is not outside the field of view. The predetermined value is set to the length from the position of the traffic light to the edge of the traffic light. In other words, if the captured traffic light extends beyond the image, it is determined to be outside the field of view. 【0067】 In step S307, the image selection unit 9 determines whether or not the traffic light captured on the image is obscured. Based on the position, width, and height of the preceding vehicle detected in step S109, the image selection unit 9 sets an obscuring area S on the image, and determines that the traffic light captured on the image is obscured if the traffic light on the image overlaps with the obscuring area S. On the other hand, if the traffic light on the image does not overlap with the obscuring area S, it determines that the traffic light captured on the image is not obscured. 【0068】 In step S309, the image selection unit 9 selects a processing image from the images of the wide-angle camera 50 and the narrow-angle camera 60. Specifically, the image selection unit 9 selects the image in which the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, and in which the size of the traffic light captured on the image is the largest, as the processing image. In other words, of the images captured by the wide-angle camera 50 and the narrow-angle camera 60, the image in which the captured traffic light is not outside the field of view, and in which the captured traffic light is the largest, is selected as the processing image. 【0069】 In this embodiment, instead of setting a processing area on the image, the processing image is selected based on the size of the traffic lights captured on the image. Once the processing image is selected in step S309, the processing image selection process ends, and the process returns to the flowchart in Figure 2 to execute the traffic light recognition process. 【0070】 [Effects of the second embodiment] As described in detail above, in the traffic signal recognition device 1 according to the present embodiment, a first image obtained by imaging the front of the vehicle with the first imaging means mounted on the vehicle, and a second imaging means having a narrower angle of view and a higher magnification than the first imaging means are used to obtain a second image obtained by imaging the front of the vehicle. Then, among the plurality of images including the first image and the second image, an image in which the distance from the position of the traffic signal imaged on the image to the image edge is greater than or equal to a predetermined value and the size of the traffic signal imaged on the image is the largest is selected as a processed image, and the lights of the traffic signal are recognized based on the selected processed image. As a result, the traffic signal can be recognized using an image in which the imaged traffic signal is not outside the angle of view and the imaged traffic signal is the largest. Therefore, even when recognizing the traffic signal by switching a plurality of cameras, the recognition performance of the traffic signal can be improved. In addition, since the process of recognizing the lights of the traffic signal is executed after narrowing down to one image, the processing time can be shortened. Furthermore, since an image can be selected without setting a processing area on the image, the processing load can be reduced. 【0071】 Note that the above-described embodiment is an example of the present invention. Therefore, the present invention is not limited to the above-described embodiment, and various modifications can be made according to the design and the like as long as they do not depart from the technical idea of the present invention even in forms other than this embodiment. 【Explanation of Reference Numerals】 【0072】 1 Traffic signal recognition device 3 Object position detection unit 5 Traffic signal position acquisition unit 7 Self-position acquisition unit 9 Image selection unit 11 Traffic signal recognition unit 13 Output unit 20 Radar 30 Lidar 40 Object recognition camera 50 Wide-angle camera 60 Narrow-angle camera 70 Map database 80 GPS receiver 90 Vehicle control system 100 Vehicle Systems

Claims

[Claim 1] A traffic light recognition method for recognizing the lights of a traffic light illuminated in front of a vehicle, A first image is obtained by capturing the area in front of the vehicle using a first imaging means mounted on the vehicle, and a second image is obtained by capturing the area in front of the vehicle using a second imaging means that has a narrower field of view and a higher magnification than the first imaging means. Among the multiple images consisting of the acquired first image and the second image, the image in which the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, and in which the size of the traffic light captured on the image is the largest, is selected as the processing image. A traffic light recognition method that recognizes the lights of a traffic light illuminated in front of the vehicle based on the selected processed image. [Claim 2] In the aforementioned plurality of images, a processing area is set at the position of the traffic light captured in the image, according to the size of the traffic light captured in the image. The traffic light recognition method according to claim 1, which recognizes the lights of a traffic light that are illuminated in front of the vehicle within the set processing area. [Claim 3] In place of the condition that the image in which the traffic light is captured on the image is captured is the largest in size, and the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, The traffic signal recognition method according to claim 2, wherein the distance from the processing area to the edge of the image is greater than or equal to a predetermined value, and the image with the largest processing area is selected as the processing image. [Claim 4] Instead of the condition that the image in which the traffic light is captured on the image is captured in the image and the distance from the position of the traffic light to the edge of the image is greater than or equal to a predetermined value, the image in which the traffic light is captured on the image is captured in the image is selected as the processing image, The traffic signal recognition method according to claim 2, wherein the proportion of the image in which the processing region is included is greater than or equal to a predetermined value, and the image with the largest processing region is selected as the processing image. [Claim 5] A traffic light recognition method according to any one of claims 1 to 4, comprising detecting an object present around the vehicle, determining whether the traffic light captured on the plurality of images is obscured by the detected object, and selecting from the plurality of images an image on which the traffic light captured is not obscured as the processing image. [Claim 6] A traffic light recognition device equipped with a controller that recognizes the lights of a traffic light illuminated in front of a vehicle, The aforementioned controller, A first image is obtained by capturing the area in front of the vehicle using a first imaging means mounted on the vehicle, and a second image is obtained by capturing the area in front of the vehicle using a second imaging means that has a narrower field of view and a higher magnification than the first imaging means. Among the multiple images consisting of the acquired first image and the second image, the image in which the distance from the position of the traffic light captured on the image to the edge of the image is greater than or equal to a predetermined value, and in which the size of the traffic light captured on the image is the largest, is selected as the processing image. A traffic light recognition device that recognizes the lights of a traffic light illuminated in front of the vehicle based on the selected processed image.

Images