Road information sign recognition device

Abstract

To provide a road information indication object recognition device capable of accurately recognizing road information of a road information indication object.SOLUTION: A road information indication object recognition device 1 comprises a camera 2 which images a front side of a vehicle 10, a LiDAR 3 including an imaging range of the camera 2 in a detection range, and an information processing ECU 4. In a case where an indication object candidate detected by the LiDAR 3 is determined to be a road information indication object, the information processing ECU 4 performs image recognition on a range corresponding to the road information indication object in an image picked up by the camera 2 and recognizes road information of the road information indication object.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a road information sign recognition device. [Background technology]

[0002] A road sign recognition device is also called TSR [Traffic Sign Recognition], and as a technology related to the road sign recognition device, for example, a road sign judgment device described in Patent Document 1 is known. The road sign judgment device described in Patent Document 1 includes an image recognition unit that detects a first road regulation sign by executing image recognition processing on a captured image of the front of the vehicle captured by an imaging unit, a map information acquisition unit that acquires a road link including a second road regulation sign at the vehicle position from a map data storage unit, and a road sign judgment unit that determines a final road regulation sign on the current road link by using at least one of the current first road regulation sign and the second road regulation sign detected on the current road link when the road link changes. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-102898 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, road information markers (such as road signs, billboards, and traffic lights) are easily installed, relocated, and removed, and therefore the information on them tends to be stale in map information. Therefore, the above-mentioned technology for recognizing road information on road information markers based on map information may have poor recognition accuracy.

[0005] The present invention has been made in consideration of the above-mentioned circumstances, and an object of the present invention is to provide a road information sign recognition device capable of accurately recognizing road information of road information signs. [Means for solving the problem]

[0006] A road information sign recognition device according to one embodiment of the present invention includes a camera that captures an image of at least a portion of the area around a vehicle, an active sensor that includes within its detection range at least a portion of the imaging range of the camera, and a processing device that, when it is determined that a sign candidate detected by the active sensor is a road information sign, performs image recognition on the area corresponding to the road information sign in the image captured by the camera, and recognizes the road information of the road information sign.

[0007] In this road information sign recognition device, the camera and the active sensor can effectively cooperate to recognize road information of road information signs without using map information including information about the road information signs. Even when a road information sign is newly installed, relocated, or removed, it is possible to recognize road information corresponding to the installation, relocation, or removal. Therefore, the road information sign recognition device according to one aspect of the present invention makes it possible to accurately recognize road information of road information signs.

[0008] In the road information sign recognition device according to one aspect of the present invention, the processing device may extract a sign candidate from the detection result of the active sensor, and determine whether or not the sign candidate is a road information sign based on at least one of the shape, position, and reflectance of the sign candidate. In this case, it is possible to determine whether or not the sign candidate detected by the active sensor is a road information sign by using at least one of the shape, position, and reflectance of the sign candidate.

[0009] In the road information sign recognition device according to one aspect of the present invention, the active sensor may be a LiDAR. In this case, the LiDAR is used as the active sensor to accurately recognize road information of the road information sign. Effect of the Invention

[0010] According to the present invention, it is possible to provide a road information sign recognition device capable of accurately recognizing road information of road information signs. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram showing the configuration of a road information sign recognition device according to an embodiment. [Diagram 2] FIG. 2 is a flowchart showing an example of recognition of road information of a road information sign performed by the road information sign recognition device according to the embodiment. [Diagram 3] FIG. 3 is a diagram showing an example of an image captured by a camera. [Figure 4] FIG. 4 is a diagram illustrating an example of point cloud information acquired by the LiDAR. [Diagram 5] FIG. 5 is a diagram illustrating an example of point cloud information acquired by the LiDAR. [Figure 6] FIG. 6 is a diagram illustrating an example of point cloud information acquired by the LiDAR. [Figure 7] FIG. 7 is a diagram showing an example of an image captured by a camera. [Figure 8] FIG. 8 is a block diagram showing a configuration of a road information sign recognition device according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the following description, the same or corresponding elements are designated by the same reference numerals, and duplicated description will be omitted.

[0013] Fig. 1 is a block diagram showing the configuration of a road information sign recognition device 1. As shown in Fig. 1, the road information sign recognition device 1 is mounted on a vehicle 10. The road information sign recognition device 1 recognizes road information of road information signs. The vehicle 10 is not particularly limited, but here it is assumed that the vehicle has a limited autonomous driving function excluding fully autonomous driving.

[0014] The road information sign is, for example, a sign, a billboard, a traffic light, etc. The road information sign is not particularly limited and includes any sign that indicates road information. The road information sign targeted by this embodiment is, for example, a circular road sign and an inverted triangular road sign from the viewpoint of recognition effectiveness. The road information sign recognition device 1 includes a camera 2, a LiDAR [Light Detection and Ranging: Lidar] 3, an information processing ECU [Electronic Control Unit] 4, and an HMI [Human Machine Interface] 5.

[0015] The camera 2 is an imaging device that captures an image of the external environment of the vehicle 10. The camera 2 is provided, for example, on the back side of the windshield of the vehicle 10, and captures an image in front of the vehicle 10. The camera 2 may be provided on the back and side of the vehicle 10. The camera 2 acquires an image of the area in front of the vehicle, and transmits the acquired image to the information processing ECU 4. The camera 2 may be a monocular camera or a stereo camera. There is no particular limitation on the camera 2, and various cameras may be used. The camera 2 may capture an image of at least a part of the area around the vehicle 10.

[0016] The LiDAR 3 is an active sensor that detects objects around the vehicle 10 using light. The LiDAR 3 detects objects by outputting light for detection to the surroundings of the vehicle 10 and receiving light reflected by objects around the vehicle 10. The LiDAR 3 here is mounted toward the front of the vehicle and detects objects in front of the vehicle. The mounting direction of the LiDAR 3 is not particularly limited, and the LiDAR 3 may be mounted toward the rear of the vehicle, or toward the left or right side of the vehicle. The objects are, for example, road information signs, structures, and curbs on the road. The LiDAR 3 includes at least a part of the imaging range of the camera 2 in its detection range. In other words, the detection range of the LiDAR 3 includes the imaging range of the camera 2, or the detection range of the LiDAR 3 is included in the imaging range of the camera 2.

[0017] The LiDAR 3 acquires point cloud information related to a three-dimensional point cloud in front of the vehicle, and transmits the acquired point cloud information to the information processing ECU 4. The point cloud information is information including at least a plurality of points (position data) constituting an object. The point cloud information is information including three-dimensional distance and reflectance for each point. The camera 2 and the LiDAR 3 are aligned (calibrated), and acquire images and point cloud information in front of the vehicle at the same fixed period and in synchronization, respectively. The LiDAR 3 may be a flash type that acquires a point cloud in front of the vehicle at once, or a rotation type that performs a 360-degree scan while rotating.

[0018] The information processing ECU 4 is an electronic control unit having a processor such as a CPU (Central Processing Unit) and memories such as a ROM (Read Only Memory) and a RAM (Random Access Memory). For example, the information processing ECU 4 loads a program recorded in the ROM into the RAM and executes the program loaded into the RAM by the CPU, thereby realizing various functions. The information processing ECU 4 may be composed of multiple electronic control units. The information processing ECU 4 constitutes a processing device.

[0019] The information processing ECU 4 extracts a sign candidate from the detection result (point cloud information) of the LiDAR 3. As a method of extraction, various known methods can be used. The information processing ECU 4 recognizes at least one of the shape, position, and reflectance of the extracted sign candidate, and determines whether or not the sign candidate is a road information sign based on at least one of the shape, position, and reflectance of the sign candidate. When the information processing ECU 4 determines that the sign candidate is a road information sign, it performs image recognition of a range corresponding to the road information sign in the image captured by the camera 2, and recognizes the road information of the road information sign. The information processing ECU 4 notifies the recognized road information to the occupants (including the driver) via the HMI 5. Details of the processing by the information processing ECU 4 will be described later.

[0020] The HMI 5 is an interface for inputting and outputting information to and from the occupant. The HMI 5 has, for example, a display unit and a speaker. The HMI 5 outputs images from the display unit and sounds from the speaker in response to a control signal from the information processing ECU 4. The HMI 5 may also be equipped with a HUD [Head Up Display]. The HMI 5 notifies the occupant of road information of road information signs recognized by the information processing ECU 4, for example, by outputting images from the display unit and outputting sounds from the speaker.

[0021] Next, an example of recognition of road information performed by the road information sign recognition device 1 will be described with reference to the flowchart of FIG.

[0022] In the road information sign recognition device 1, for example, when the vehicle 10 starts traveling, the following road information recognition process is executed. That is, an image G (see FIG. 3) of the area in front of the vehicle is acquired by the camera 2, and point cloud information T (see FIG. 4) of the area in front of the vehicle is acquired by the LiDAR 3 (step S1). In the point cloud information T in FIG. 4, the point clouds on the left and right are reflected by the curbstones, and the center line is not reflected because it is a white line. In the point cloud information T in FIG. 4, the point cloud reflected by the road surface has been removed in advance.

[0023] Next, the information processing ECU 4 detects left and right lane markers in the image G in front of the vehicle, and specifies an existence range (usually a range on the sidewalk near the roadway) in which a road information sign is likely to exist based on the detection result of the lane marker (step S2). The information processing ECU 4 sets an existence range (a dotted range W1 in FIG. 5) of the specified road information sign in the point cloud information, and extracts a point cloud of a sign candidate that is located in the existence range of the road information sign and is a candidate that is likely to be a road information sign (step S3). In the example of FIG. 6, the point cloud of the sign candidate is a point cloud within a dotted rectangular range W2.

[0024] Next, the information processing ECU 4 judges whether the extracted sign candidate is a road information sign (step S4). In the above step S4, the extracted sign candidate is evaluated using its shape, position (height position), and reflectance. For example, the information processing ECU 4 stores in advance a data table in which the shape, position, and reflectance are associated with each type of a plurality of road information signs. In the above step S4, the extracted sign candidate can be scored and evaluated using the data table to be judged. The shape of the sign candidate can be recognized using known image processing such as pattern matching. The position of the sign candidate can be based on, for example, the center position, the upper end position, the lower end position, the left end position, or the right end position of the sign candidate. The reflectance of the sign candidate can be based on, for example, the maximum value, the minimum value, or the average value of the reflectance of each point group constituting the sign candidate.

[0025] In this embodiment, the road information sign is a speed sign 9, and therefore the road information sign is designed to have a circular shape with a diameter within a certain range (several tens of centimeters to just under 1.5 meters). The installation height of the speed sign 9 is within a range of about 1 to 2 meters. The speed sign 9 has tiny glass beads or the like that are not visible to the naked eye provided on its surface to improve the visibility of the sign by the headlights, and therefore the reflectance of the speed sign is stable and above a certain level. Therefore, in the above step S4, the extracted sign candidate is determined to be a road information sign because the shape of the sign candidate is a circle within a certain range, the position is within a range of 1 to 2 meters, and the reflectance is above a certain level.

[0026] If it is determined that a road information sign is present, the information processing ECU 4 cuts out an area (a dotted rectangular area W3 in FIG. 7) corresponding to the road information sign in the image G ahead of the vehicle (YES in step S5, step S6). The area cut out from the image G ahead of the vehicle is subjected to pre-processing (tilt correction, enlargement / reduction, etc.) and image recognition, and the road information of the road information sign is recognized (step S7). In this example, it is recognized that the road information sign is a speed sign and the speed limit value is "60km / h". The method of pre-processing and image processing performed in the above step S7 is not particularly limited, and various known processes may be used.

[0027] The HMI 5 notifies the occupant of the road information of the recognized road information sign (step S8). After step S8 or in the case of NO in step S5, if the vehicle 10 is not stopped, the process returns to step S1, whereas if the vehicle 10 is stopped, the image recognition process is terminated (step S9).

[0028] As described above, in the road information sign recognition device 1, the camera 2 and the LiDAR 3 can effectively cooperate to recognize road information of road information signs without using map information including information about the road information signs. Even when a road information sign is newly installed, relocated, or removed, it is possible to recognize road information corresponding to the installation, relocation, or removal. Therefore, the road information sign recognition device 1 can accurately recognize road information of road information signs. It is possible to reduce the need to prepare map information on the vehicle 10 side and record the positions of road information signs therein in advance. Robust recognition is possible even when a road information sign is newly installed, relocated, or removed (disappeared).

[0029] In the road information sign recognition device 1, the information processing ECU 4 extracts a sign candidate from the detection result of the LiDAR 3, and determines whether or not the sign candidate is a road information sign based on at least one of the shape, position, and reflectance of the sign candidate. In this case, whether or not the sign candidate detected by the LiDAR 3 is a road information sign can be determined by using at least one of the shape, position, and reflectance of the sign candidate.

[0030] The road information sign recognition device 1 includes the LiDAR 3 as an active sensor. With this, the LiDAR 3 can be used as an active sensor to accurately recognize road information of road information signs.

[0031] Although various exemplary embodiments have been described above, various omissions, substitutions, and modifications may be made without being limited to the above-described exemplary embodiments.

[0032] 8 is a block diagram showing the configuration of a road information sign recognition device 101 according to a modified example. The road information sign recognition device 101 according to the modified example further includes a GNSS [Global Navigation Satellite System] receiver 6, an internal sensor 7, and a map database 8 in addition to the above-mentioned road information sign recognition device 1 (see FIG. 1).

[0033] The GNSS receiver 6 constitutes a satellite positioning device that measures the position of the vehicle 10 (e.g., the latitude and longitude of the vehicle 10) by receiving signals from four or more GNSS satellites. The GNSS receiver 6 transmits the measured position information of the vehicle 10 to the information processing ECU 4. The internal sensor 7 includes a vehicle speed sensor, an acceleration sensor, and a yaw rate sensor. The vehicle speed sensor is a detector that detects the speed of the vehicle 10. The acceleration sensor is a detector that detects the acceleration of the vehicle 10. The yaw rate sensor is a detector that detects the yaw rate (rotational angular velocity) around the vertical axis of the center of gravity of the vehicle 10. The internal sensor 7 transmits the detection results to the information processing ECU 4.

[0034] The map database 8 is a database that records map information. The map database 8 is formed, for example, in a recording device such as a HDD (Hard Disk Drive) mounted on the vehicle 10. The map information includes, for example, road position information, road shape information (for example, number of lanes, lane width, curvature information), intersection and branch point position information, etc. The map database 8 may be configured in a server that can communicate with the vehicle 10.

[0035] The information processing ECU 4 of this road information sign recognition device 101 can estimate the positions of the left and right lane markers in the image G ahead of the vehicle while complementing them using the GNSS receiver 6, the internal sensor 7, and the map database 8. Therefore, even if the lane markers are difficult to identify from the camera 2 due to blurring or the like, the location where the road information sign exists (the range where the road information sign exists (the range W1 indicated by the dotted line in FIG. 5)) can be estimated well, and more stable operation can be expected.

[0036] In the above embodiment, the LiDAR 3 is used as the active sensor, but the active sensor is not limited, and other active sensors may be used. In the above embodiment, image recognition by machine learning may be used. In the above embodiment, for example, in the recognition of a road information sign such as a speed limit sign, the format is known and the displayed numerical value is limited, so simple pattern matching may be used.

[0037] In the above embodiment, an example has been described in which one road information sign (marker candidate) exists in the image G and the point cloud information T, but the number of road information signs existing in the image G and the point cloud information T is not limited and may be more than one. When multiple road information signs of different types exist in the image G and the point cloud information T, road information for each of them may be recognized and notified to the occupant. On the other hand, when multiple road information signs of the same type exist in the image G and the point cloud information T, road information for any one of them (for example, any one closest to the vehicle 10) may be recognized and notified to the occupant. [Explanation of symbols]

[0038] 1...road information sign recognition device, 2...camera, 3...LiDAR (active sensor), 4...information processing ECU (processing device), 9...speed sign (road information sign), 10...vehicle, G...image, T...point cloud information.

Claims

1. a camera that captures an image of at least a portion of an area around the vehicle; an active sensor having a detection range that includes at least a part of the imaging range of the camera; a processing device that, when it is determined that the sign candidate detected by the active sensor is a road information sign, performs image recognition on a range corresponding to the road information sign in the image captured by the camera, and recognizes road information of the road information sign; The processing device includes: extracting sign candidates from the detection results of the active sensor; determining whether the sign candidate is a road information sign based on at least one of a shape, a position, and a reflectance of the sign candidate; The processing device includes: Detecting left and right lane markers in the image, and specifying an area on the sidewalk as an area in which the road information sign is present based on the detection result of the lane markers; setting an existence range of the identified road information sign in point cloud information relating to a three-dimensional point cloud in front of the vehicle detected by the active sensor, and extracting the sign candidate located within the existence range of the road information sign; and a road information sign recognition device that determines whether the extracted sign candidate is the road information sign based on at least one of the shape, position, and reflectance of the sign candidate.

2. The processing device comprises:

2. The road information sign recognition device according to claim 1, wherein, when it is determined that a plurality of road information signs of the same type are present in the point cloud information, a corresponding range in the image for any one of the road information signs is image-recognized, and the road information of the one road information sign is recognized.

3. The road information sign recognition device according to claim 1 or 2, wherein the active sensor is a LiDAR.

Images