Road marking recognition device

Abstract

A road marking recognition device (50) includes an information acquisition unit (111) that acquires environmental information including a position of the sun; a defect rate recognition unit (113) that recognizes a road marking from a captured image of a camera (4) and recognizes a defect degree of the road marking; an illuminance estimation unit (114) that estimates an illuminance of the road marking from an illuminance detected by an illuminance sensor (5); a brightness ratio estimation unit (115) that estimates a brightness ratio between a first region inside the road marking and a second region outside the road marking, based on the defect degree of the road marking; a reflectance estimation unit (116) that estimates a reflectance of the first region, based on the estimated brightness ratio; a brightness estimation unit (117) that estimates a brightness of the first region, based on the estimated illuminance of the road marking and the reflectance of the first region; and an end portion recognition unit (118) that recognizes a position of an end portion of the road marking, based on the estimated brightness of the first region.

Description

TECHNICAL FIELD

[0001] The present application relates to a road marking recognition device that recognizes a road marking of a road. BACKGROUND

[0002] As such a device, a device that recognizes a lane marking on a road surface based on the brightness of a captured image of the road surface has been known (see, for example, Patent Literature 1). The device described in Patent Literature 1 calculates the reflectance of the road surface based on the captured image of the road surface, and extracts, as a candidate for a lane marking, a pixel whose brightness difference from an adjacent pixel is equal to or greater than a threshold value set in accordance with the reflectance of the road surface.

[0003] However, there are cases in which, in the captured image of the road surface, there are pixels whose brightness difference from an adjacent pixel is equal to or greater than the threshold value even in a region in which there is no lane marking. Therefore, as with the device described in Patent Literature 1, it is possible that the pixels extracted as candidates for a lane marking based on the brightness difference cannot be recognized with high accuracy.

[0004] PRIOR ART DOCUMENTS

[0005] PATENT LITERATURE

[0006] Patent Literature 1: Japanese Patent No. 4023333 SUMMARY

[0007] A road marking recognition device according to one aspect of the present application includes: a photographing unit that photographs a situation around a host vehicle; an illuminance detector that detects an illuminance of the surroundings of the host vehicle; a first recognition unit that recognizes a road marking of a lane of a road and recognizes a degree of defect of the road marking based on a captured image acquired by the photographing unit; an illuminance estimation unit that estimates an illuminance of the road marking based on the illuminance detected by the illuminance detector; a brightness ratio estimation unit that estimates a brightness ratio between a first region inside the road marking and a second region outside the road marking based on the degree of defect of the road marking recognized by the first recognition unit; a reflectance estimation unit that estimates a reflectance of the first region based on the brightness ratio estimated by the brightness ratio estimation unit; a brightness estimation unit that estimates a brightness of the first region based on the illuminance of the road marking estimated by the illuminance estimation unit and the reflectance of the first region estimated by the reflectance estimation unit; and a second recognition unit that recognizes a position of an end portion of the road marking based on the brightness of the first region estimated by the brightness estimation unit. BRIEF DESCRIPTION OF DRAWINGS

[0008] Objects, features, and advantages of the present application will be further clarified by the following description of embodiments with reference to the accompanying drawings.

[0009] Figure 1 is a block diagram showing a main part structure of a vehicle control device according to an embodiment of the present application.

[0010] Figure 2 This is an example of a photograph taken from the front of the vehicle.

[0011] Figure 3 Viewed from above Figure 2 A map showing part of the road.

[0012] Figure 4 It is a diagram used to illustrate the degree of damage to road markings.

[0013] Figure 5A This is a diagram showing an example of a brightness ratio table.

[0014] Figure 5B This is a diagram showing an example of a reflectance table.

[0015] Figure 6 This is a diagram used to illustrate the calculation of brightness.

[0016] Figure 7 It is shown by Figure 1 A flowchart of an example of the processing performed by the controller. Detailed Implementation

[0017] The following is for reference Figures 1 to 7 Embodiments of the present invention will be described. The road marking recognition device of the present invention is applicable to vehicles with autonomous driving capabilities, i.e., autonomous vehicles. It should be noted that sometimes the vehicle using the road marking recognition device of this embodiment is distinguished from other vehicles and referred to as "this vehicle". This vehicle can be any of the following: an engine vehicle with an internal combustion engine as the driving source, an electric vehicle with a drive motor as the driving source, or a hybrid vehicle with both an engine and a drive motor as driving sources. This vehicle can not only operate in an autonomous driving mode that does not require driver operation, but also in a manual driving mode based on driver operation.

[0018] When an autonomous vehicle is driving in autonomous driving mode (hereinafter referred to as automatic driving or autonomous driving), it identifies the lane markings of a designated road based on image data obtained by a camera unit located in a designated part of the vehicle (e.g., above the windshield). Based on the identified lane marking information, the autonomous vehicle controls its driving actuators to keep the vehicle traveling near the center of its lane.

[0019] However, because of the sunlight condition, water spread on the road surface (hereinafter referred to as water film), wear or stain of the road marking, and the like, sometimes the road marking cannot be recognized with high accuracy from the image data obtained by the imaging unit. In this case, it can be impossible to automatically travel well. By additionally mounting various high-accuracy sensors on the vehicle to detect the situation around the vehicle, the road surface state, and the like, the above problem can be addressed. However, it can complicate the structure of the device and cause an increase in cost. Therefore, in the present embodiment, the road marking recognition device is configured as follows.

[0020] Figure 1 is a block diagram showing the main part structure of a vehicle control device having the road marking recognition device of the embodiment of the present application. The vehicle control device 100 has a controller 10, a communication unit 1, a positioning unit 2, an internal sensor group 3, a camera 4, actuators AC for travel. In addition, the vehicle control device 100 has a road marking recognition device 50 that constitutes a part of the vehicle control device 100. The road marking recognition device 50 recognizes the road marking of the road on which the host vehicle travels, based on the captured image data of the camera 4.

[0021] The communication unit 1 communicates with various servers not shown via a network including a wireless communication network typified by the Internet, a mobile phone network, and the like, and acquires map information, travel history information, traffic information, and the like from the servers periodically or at an arbitrary timing. The network includes not only a public wireless communication network but also a closed communication network such as a wireless LAN, Wi-Fi (registered trademark), Bluetooth (registered trademark), and the like provided for each prescribed management area. The acquired map information is output to a storage 12, and the map information is updated. The positioning unit (GNSS unit) 2 has a positioning sensor that receives a positioning signal transmitted from a positioning satellite. The positioning satellite is a GPS (Global Positioning System) satellite, a quasi-Zenith satellite, or the like. The positioning unit 2 measures the current position (latitude, longitude, altitude) of the host vehicle using the positioning information received by the positioning sensor.

[0022] The internal sensor group 3 is a general term for a plurality of sensors (internal sensors) that detect the travel state of the host vehicle. For example, the internal sensor group 3 includes a vehicle speed sensor that detects the vehicle speed of the host vehicle, an acceleration sensor that detects the acceleration in the front-rear direction and the acceleration in the left-right direction (lateral acceleration) of the host vehicle, respectively, a rotation speed sensor that detects the rotation speed of the travel driving source, a yaw rate sensor that detects the rotation angular velocity of the center of gravity of the host vehicle around the vertical axis, and the like. Sensors that detect the driver's driving operation in a manual driving mode, such as the operation of an accelerator pedal, the operation of a brake pedal, the operation of a steering wheel, and the like, are also included in the internal sensor group 3.

[0023] The camera 4 has an imaging element such as a CCD (Charge-Coupled Device), a CMOS (Complementary Metal-Oxide Semiconductor), or the like, and captures the surroundings (front, rear, and side) of the host vehicle. The illuminance sensor 5 has a light-receiving element, and detects the brightness (illuminance) of light incident on the light-receiving element. The illuminance sensor 5 is disposed on the outside of the host vehicle (for example, on the roof) or on the inside of the host vehicle (on the instrument panel) in a manner that enables detection of the illuminance of the surroundings of the host vehicle. Note that, since light that has passed through the windshield is slightly attenuated by the glass, in the case where the illuminance sensor 5 is disposed on the inside of the host vehicle, the sensor value can also be corrected taking this amount of attenuation into account.

[0024] The actuator AC is a travel actuator for controlling the travel of the host vehicle. In the case where the travel drive source is an engine, the actuator AC includes a throttle actuator that adjusts the opening degree of a throttle valve of the engine. In the case where the travel drive source is a travel motor, the actuator AC includes the travel motor. A brake actuator that causes a brake device of the host vehicle to act, and a steering actuator that drives a steering device, are also included in the actuator AC.

[0025] The controller 10 is constituted by an electronic control unit (ECU). More specifically, the controller 10 is constituted by a computer that includes an arithmetic unit 11 having a CPU (microprocessor) or the like, a storage unit 12 such as a ROM (Read Only Memory), a RAM (Random Access Memory), and other peripheral circuits such as an I / O interface, and the like, which are not shown. Note that, a plurality of ECUs that differ in function, such as an engine control ECU, a travel motor control ECU, a brake device ECU, and the like, can be provided separately, but in the present embodiment, the controller 10 is shown as a collection of these ECUs for the sake of convenience. Figure 1

[0026] High-precision detailed map information (referred to as high-precision map information) is stored in the storage unit 12. The high-precision map information includes position information of a road, information of a road shape (curvature, etc.), information of a slope of a road, position information of an intersection or a fork, information of the number of lanes, the width of a lane, and the position information of each lane (the center position of a lane, information of a boundary line of a lane position), position information of a landmark (a signal, a sign, a building, etc.) as a marker on a map, information of a road profile such as a concave-convex of a road surface, and the like. Various control programs, information of threshold values used in the programs, and the like are also stored in the storage unit 12.

[0027] The arithmetic unit 11 has, as a functional structure, an information acquisition unit 111, a measurement point setting unit 112, a missing rate recognition unit 113, an illuminance estimation unit 114, a brightness ratio estimation unit 115, a reflectance estimation unit 116, a brightness estimation unit 117, an end portion recognition unit 118, and a travel control unit 119. Note that, as shown in FIG. 1, the information acquisition unit 111, the measurement point setting unit 112, the missing rate recognition unit 113, the illuminance estimation unit 114, the brightness ratio estimation unit 115, the reflectance estimation unit 116, the brightness estimation unit 117, and the end portion recognition unit 118 are constituted by the controller 10. Figure 1 ​As shown, the information acquisition unit 111, the measurement point setting unit 112, the defect rate recognition unit 113, the illuminance estimation unit 114, the brightness ratio estimation unit 115, the reflectance estimation unit 116, the brightness estimation unit 117, and the end recognition unit 118 are included in the road marking recognition device 50.

[0028] The information acquisition unit 111 acquires environmental information. In detail, the information acquisition unit 111 estimates the position of the sun based on the current position of the host vehicle and the current date and time measured by the positioning unit 2. The estimated position of the sun can be an absolute position or a relative position with respect to the host vehicle. The information acquisition unit 111 acquires information showing the estimated position of the sun as environmental information. In addition, the information acquisition unit 111 acquires weather-related information (hereinafter referred to as weather information) such as rainfall from an external server (not shown) via the communication unit 1 as environmental information.

[0029] The measurement point setting unit 112 sets a measurement point on the road surface. Figure 2 is an example of a captured image of the front of the host vehicle during travel of the host vehicle on the road RD, which is acquired by the camera 4. In Figure 2 , the measurement point MP set on the road surface of the road RD is shown in schematic coincidence with the captured image. Figure 3 is a view of a portion of the road RD viewed from above Figure 2 . The measurement point setting unit 112 recognizes a region on the road surface corresponding to the road marking LL, LC, and LR based on the captured image acquired by the camera 4. This estimation can use machine learning or other image recognition techniques. Hereinafter, the region recognized at this time is referred to as a road marking recognition region or simply a road marking region. The measurement point setting unit 112 sets a measurement point MP0 at a position far ahead in the travel direction (a position separated by a certain distance from the host vehicle). The measurement point MP0 is a region having a prescribed number of pixels on the captured image, as shown in Figure 2 , is configured to include the road marking recognition region. In Figure 2 , the solid line measurement point MP0 schematically indicates the measurement point MP0 set at the current time (the time of Figure 2 ), and the dashed line measurement point MP0 schematically indicates the measurement point MP0 set at a past time (when the host vehicle was traveling at a position closer to the front in the travel direction than the current travel position). Note that Figure 2 , the measurement point MP0 is set by a method that is an example, and the shape, size, and position at which the measurement point MP0 is set are not limited thereto.

[0030] The defect rate recognition unit 113 recognizes the degree of defect of the road marking at the measurement point MP0 (hereinafter also referred to as road marking defect rate) on the basis of the captured image of the camera 4. Specifically, the defect rate recognition unit 113 calculates the degree of defect of the road marking in the target region within the measurement point MP0, using the maximum luminance of a region (for example, a region adjacent in the lane width direction) near the measurement point MP0 and a region (hereinafter referred to as the vicinity region) outside the road marking recognition region as a boundary value. In more detail, the defect rate recognition unit 113 calculates the road marking defect rate of the target region by dividing the area of a region (pixel group) having a luminance lower than the boundary value in the target region by the total area (total number of pixels) of the target region. Note that the entire region of the measurement point MP0 can be used as the target region, or a part of the region near the center of the measurement point MP0 can be used as the target region in such a manner that the target region does not include the region outside the road marking recognition region. Figure 4 is a graph for explaining the road marking defect rate. The characteristics f1, f2 represent the luminance distribution of the target region and the vicinity region obtained on the basis of the captured image of the camera 4, and specifically represent the number of pixels for each luminance. The line BD in the graph represents the maximum luminance of the vicinity region, and the region FA represents the pixel group of the target region having a luminance lower than the maximum luminance BD. The road marking defect rate of the target region is a value obtained by dividing the number of pixels within the region FA by the total number of pixels of the characteristic f1. The defect rate recognition unit 113 obtains the road marking defect rate of the target region as the road marking defect rate of the measurement point MP0.

[0031] The illuminance estimation unit 114 estimates the illuminance of the road surface in front of the host vehicle. In detail, first, the illuminance estimation unit 114 obtains the sensor value (illuminance) detected by the illuminance sensor 5. At this time, the illuminance estimation unit 114 obtains the illuminance detected by the illuminance sensor 5 under the condition that the influence of the shadow is excluded. In addition, on the basis of the relationship between the position of the sun and the position of the illuminance sensor 5, the illuminance of the direct light of the sun and the illuminance of the sky light are separated according to the oblique incidence light characteristic (light receiving angle characteristic) possessed by the illuminance sensor 5, and the respective illuminances are stored for the sun height and the date and time. That is, the illuminance estimation unit 114 obtains the illuminance of the direct light of the sun and the illuminance of the sky light (ambient light), respectively. Next, the illuminance estimation unit 114 recognizes the position, size, and shape of the structure (building BL) in the vicinity of the host vehicle on the basis of the map information stored in the storage unit 12, and estimates the shadow (shaded region) generated by the structure on the road in front of the host vehicle on the basis of the position information of the sun included in the environment information obtained by the information obtaining unit 111. Figure 2 Figure 2 ​The location, size, and shape of the area (SD) are determined. At this time, the illuminance estimation unit 114 can also estimate the location, size, and shape of the shadow cast by the structure on the road surface in front of the vehicle based on the image captured by the camera 4. The illuminance estimation unit 114 determines whether the measurement point MP0 (more specifically, the aforementioned object area) is covered by a shadow based on the estimated location, size, and shape of the structure's shadow. When the measurement point MP0 is covered by a shadow, the illuminance estimation unit 114 estimates the attenuation rate of the shadow on skylight based on the sky rate, and corrects the illuminance detected by the illuminance sensor 5 based on the estimation result.

[0032] The luminance ratio estimation unit 115 estimates the luminance ratio between measurement points MP1 and MP2 based on the road marking defect rate of measurement point MP0 identified by the defect rate identification unit 113. For example... Figure 2 As shown, measurement points MP1 and MP2 are set by the measurement point setting unit 112 at a distance from measurement point MP0 ( Figure 2 The solid line indicates the position of the measuring point MP0 (closely forward of the direction of travel). More specifically, the measuring point MP0 (set at a past time, when the vehicle was traveling at a position closer to the direction of travel than its current position). Figure 2 The dotted line indicates the setting position of the measurement point MP0, and measurement points MP1 and MP2 corresponding to that measurement point MP0 are set. Thus, when the measurement point setting unit 112 is somewhat close to the vehicle at the measurement point MP0 used to obtain illuminance and damage rate, it sets measurement points MP1 and MP2 to that measurement point MP0. Furthermore, at this time, if... Figure 2 and Figure 3 As shown, the measurement point setting unit 112 sets a measurement point MP1 within the road marking recognition area and sets a measurement point MP2 corresponding to MP1 outside the road marking recognition area. Measurement points MP1 and MP2 are rectangular areas with a width smaller than the width of the road marking, for example, areas the size of a postcard. It should be noted that measurement points MP1 and MP2 may also have shapes other than rectangles. Furthermore, the size of measurement points MP1 and MP2, and the positions where they are set, are not limited to these.

[0033] In detail, the luminance ratio estimation unit 115 uses a luminance ratio table (hereinafter also referred to as luminance ratio information) to estimate the luminance ratio (luminance of MP1 / luminance of MP2) of the measuring point MP2 relative to the measuring point MP1 based on the road marking defect rate of the measuring point MP1. Figure 5A This is a diagram illustrating an example of a brightness ratio table. (For example...) Figure 5AThe luminance ratio table is information showing a characteristic of a luminance ratio BR (BR1 to BRn) of the inner side and the outer side of the road marking recognition region with respect to a road marking defect rate CF (CF1 to CFn). The road marking defect rate CF1 is a state in which the road marking is not worn or stained, and corresponds to a state in which the visibility of the road marking is the highest. The road marking defect rate CFn is a state in which the road marking is worn and disappears, and corresponds to a state in which the visibility of the road marking is the lowest. Note that CFn-1 < CFn. The luminance ratio BR decreases as the road marking defect rate CF increases. Note that the luminance ratio BR can have a linear characteristic with respect to the road marking defect rate CF, or can have another characteristic. In this way, by measuring the degree of defect of the road marking and quantifying the data as the luminance ratio, it is possible to quantify the degree of defect of the road marking. As a result, it is possible to handle the degree of defect of the road marking as a numerical value.

[0034] Note that, Figure 5A The luminance ratio table is generated in advance based on the luminance of the inner side and the outer side of the road marking, which is obtained based on the captured image, and is stored in the storage section 12, by capturing a plurality of road marking samples having different road marking defect rates using the camera 4 or an imaging device having the same performance as the camera 4 in the same environment (light source, rainfall, state of the road).

[0035] The reflectance estimation section 116 estimates the reflectance of the measurement point MP1 based on the luminance ratio of the measurement point MP1 and the measurement point MP2 estimated by the luminance ratio estimation section 115, from the reflectance table (hereinafter also referred to as reflectance information). Figure 5B is a diagram showing an example of a reflectance table. As Figure 5B shown, the reflectance table is information showing a characteristic of reflectances RF1 to RFn in the road marking recognition region with respect to luminance ratios BR1 to BRn of the inner side and the outer side of the road marking recognition region. The reflectance RF is a reflectance predicted from the road marking defect rate CF corresponding to the luminance ratio. The reflectance RF increases as the luminance ratio BR increases. Note that the reflectance RF can have a linear characteristic with respect to the luminance ratio BR, or can have another characteristic. Figure 5A

[0036] In the reflectance table of Figure 5B , a reflectance table is generated and stored in the storage section 12 in advance for each angle formed by the light source (the sun), the reflection point (the observation point), and the camera 4 (more specifically, the position of the lens of the camera 4). The reflectance estimation section 116 calculates the angle formed by the light source (the sun), the reflection point, and the camera 4 from the position of the sun indicated by the environment information obtained by the information obtaining section 111 and the current position of the host vehicle measured by the positioning unit 2, reads out the reflectance table corresponding to the calculated angle from the storage section 12, and uses it. ​

[0037] The luminance estimation unit 117 estimates the luminance of the measurement point MP1 from the illuminance estimated by the illuminance estimation unit 114 and the reflectance estimated by the reflectance estimation unit 116. Specifically, the luminance estimation unit 117 calculates the luminance L(θ) of the measurement point MP1 using the following equations (i), (ii). Hereinafter, the luminance of the measurement point MP1 thus estimated is referred to as the estimated luminance. Figure 6 is a graph for explaining the calculation of the luminance. θ denotes the angle of the straight line connecting the measurement point MP1 and the camera 4 with respect to the vertical direction, i.e., the reflection angle. E(θ) denotes the illuminance when the reflection angle is θ. R denotes the reflectance. I(θ) denotes the luminosity of the light source S when the reflection angle is θ. r is the straight-line distance between the measurement point MP1 and the camera 4.

[0038]

[0039]

[0040] In addition, the luminance estimation unit 117 estimates the state of the road surface ahead of the host vehicle from the weather information included in the environmental information acquired by the information acquisition unit 111 and the current position of the host vehicle measured by the positioning unit 2. The luminance estimation unit 117 calculates the transmittance of the water film on the road surface on the basis of the estimated state of the road surface, more specifically, the state of the water film on the road surface. Note that the water film can also include water containing ice crystals or particles such as snow covering the road surface. The state of the water film refers to the depth of the accumulated water and the depth of the accumulated snow. In addition, in the case where the host vehicle is equipped with a rain sensor (raindrop sensor), the luminance estimation unit 117 can estimate the state of the water film on the road surface ahead of the host vehicle on the basis of the sensor value of the rain sensor. Also, the luminance estimation unit 117 can recognize the state of the rainfall or snowfall from the captured image of the camera 4 and estimate the state of the water film on the road surface.

[0041] The luminance estimation unit 117 also estimates the amount of attenuation of the luminance (the amount of attenuation from the measurement point MP1 to the camera 4) due to the water film on the basis of the calculated transmittance of the water film and corrects the estimated luminance of the measurement point MP1 on the basis of the estimated amount of attenuation. Thereby, it is possible to estimate the luminance of the measurement point MP1 on the captured image of the camera 4 with higher accuracy. Note that the light passing through the windshield is slightly attenuated by the glass, and therefore, in the case where the camera 4 is disposed on the inner side of the host vehicle, the estimated luminance of the measurement point MP1 can also be corrected taking this amount of attenuation into account.

[0042] The end recognition unit 118 identifies the position of the end (edge) of the road marking based on the estimated brightness calculated by the brightness estimation unit 117 (or, in the case of the above correction, the estimated brightness after correction). More specifically, the end recognition unit 118 identifies the position (pixel) of the end of the area (pixel group) in the lane width direction of the region (pixel group) in the image captured by the camera 4, which has a brightness difference from the measurement point MP1 within a predetermined range, as the position of the end of the road marking.

[0043] In automatic driving mode, the driving control unit 119 generates a target trajectory based on the road markings identified by the road marking recognition device 50, and controls the actuator AC to make the vehicle travel along the target trajectory. It should be noted that in manual driving mode, the driving control unit 119 controls the actuator AC based on driving commands (steering operations, etc.) from the driver obtained from the internal sensor group 3.

[0044] Figure 7 It shows that according to a predetermined procedure, by Figure 1 A flowchart of an example of the processing performed by controller 10. Figure 7 The process shown in the flowchart is repeated at predetermined intervals, for example, while the vehicle is driving in autonomous driving mode. Additionally, for example, this process is repeated every predetermined distance traveled while the vehicle is driving in autonomous driving mode.

[0045] First, in step S11, environmental information is obtained. Specifically, the position of the sun is estimated based on the current position of the vehicle determined by the positioning unit 2 and the current date and time, and the sun's position information (latitude and longitude information are shown) is obtained. Additionally, weather information is obtained from an external server (not shown) via the communication unit 1. In step S12, the area corresponding to the road markings is identified based on the image captured by the camera 4 in front of the vehicle, and a measurement point MP0 is set based on the identification result. In step S13, the image captured by the camera 4 is obtained, and the road marking defect rate of measurement point MP0 is identified based on the captured image. In step S14, the illuminance of measurement point MP0 is estimated based on the sensor value of the illuminance sensor 5.

[0046] In step S15, the position, size, shape of the structure around the host vehicle are recognized on the basis of the map information stored in the storage section 12. Then, on the basis of the recognized position, size, shape of the structure and the position information of the sun acquired in step Sll, the position, size, shape of the shadow generated on the road in front of the host vehicle by the structure are estimated. Further, on the basis of the estimation result, it is determined whether or not the shadow caused by the structure is generated on the measurement point MP0. When step S15 is affirmative (S15: YES), step S17 is reached. When step S15 is negative (S15: NO), in step S16, on the basis of the weather information acquired in step Sll, the state of the water film on the measurement point MP0 is determined. More specifically, it is determined whether or not the water film exists on the measurement point MP0. When step S16 is affirmative (S16: YES), in step S17, the illuminance estimated in step S14 is corrected. The set position of the measurement point MP0, the road marking defect rate, and the illuminance are stored in the storage section 12.

[0047] When step S16 is negative (S16: NO), in step S18, the measurement points MP1, MP2 are set, and the luminance ratio of the measurement points MP1, MP2 is estimated. More specifically, first, the measurement point MP0 set in the previous processing cycle in step S12 is decided as the measurement point to be processed (hereinafter referred to as the object measurement point). In addition, the method of deciding the object measurement point is not limited to this, and for example, the measurement point MP0 set in the previous previous processing cycle can be decided as the object measurement point. Next, the set position of the object measurement point is read out from the storage section 12, and on the basis of the read-out set position, the measurement points MP1, MP2 corresponding to the object measurement point are set. Thereby, after the measurement point MP0 for acquiring the illuminance, the road marking defect rate is somewhat close to the host vehicle, the measurement point MP1 and the measurement point MP2 are set to this measurement point MP0. Finally, the road marking defect rate of the object measurement point is read out from the storage section 12, and using the luminance ratio table of Figure 5A , on the basis of the read-out road marking defect rate, the luminance ratio of the measurement point MP1 to the measurement point MP2 is estimated. In step S19, using the reflectance table of Figure 5B , on the basis of the luminance ratio estimated in step S18, the reflectance of the measurement point MP1 is estimated. In step S20, using the above equations (i), (ii), on the basis of the illuminance estimated in step S14 and the reflectance estimated in step S19, the estimated luminance of the measurement point MP1 is calculated, and the end position in the lane width direction of the region in which the luminance difference between the estimated luminance is within a prescribed degree is recognized as the end position of the road marking.

[0048] With the present embodiment, the following effects can be obtained.

[0049] (1) The road marking recognition device 50 includes: the camera 4 that captures the situation around the host vehicle; the illuminance sensor 5 that detects the illuminance around the host vehicle; the defect rate recognition section 113 that recognizes the road marking of the lane of the prescribed road from the captured image acquired by the camera 4, and recognizes the defect degree of the road marking (specifically, the measurement point MP0); the illuminance estimation section 114 that estimates the illuminance of the road marking from the illuminance detected by the illuminance sensor 5; the brightness ratio estimation section 115 that estimates the brightness ratio between the first region (measurement point MP1) within the road marking and the second region (measurement point MP2) outside the road marking from the defect degree of the road marking recognized by the defect rate recognition section 113; the reflectance estimation section 116 that estimates the reflectance of the first region from the brightness ratio estimated by the brightness ratio estimation section 115; the brightness estimation section 117 that estimates the brightness of the first region from the illuminance of the road marking estimated by the illuminance estimation section 114 and the reflectance of the first region estimated by the reflectance estimation section 116; and the end portion recognition section 118 that recognizes the position of the end portion of the road marking from the brightness of the first region estimated by the brightness estimation section 117.

[0050] Thus, instead of recognizing only the road marking using the captured image of the camera 4, the position of the end portion of the road marking is recognized also considering the defect degree of the road marking based on the captured image of the camera 4, whereby the road marking can be recognized with high accuracy. As a result, unnecessary handover of driving authority or emergency handover of driving authority to the driver due to misrecognition or unrecognizability of the road marking can be suppressed. In addition, since the target trajectory is generated based on the road marking recognized with high accuracy, stable automatic travel can be performed, and the ride comfort of the occupant can be improved. Moreover, since the road marking can be recognized with high accuracy from the captured image of the camera 4, other sensors such as a laser radar or a radar are not needed, and the device structure can be simplified.

[0051] (2) The end portion recognition section 118 recognizes the position of the end portion of the road marking as the position of the end portion of the lane in the lane width direction of the region including the first region and having a brightness difference from the first region within a prescribed degree from the brightness of the first region estimated by the brightness estimation section 117. Thus, the boundary between the road marking of the lane of the prescribed road and the region where the road marking is present can be recognized with high accuracy.

[0052] (3) The road marking recognition device 50 also includes an information acquisition unit 111, which acquires environmental information including the position of the sun. An illuminance estimation unit 114 determines whether the road markings are covered by the shadows of structures surrounding the vehicle based on the sun's position indicated by the environmental information, and corrects the illuminance detected by the illuminance sensor 5 based on the determination result, thereby estimating the illuminance of the road markings. At this time, the illuminance estimation unit 114 identifies the position, size, and shape of structures based on images captured by the camera 4, and determines whether the road markings are covered by the shadows of structures surrounding the vehicle based on the identification results. Therefore, even when shadows are cast on the road due to buildings or vegetation beside the road, road markings can be recognized with high accuracy.

[0053] (4) The environmental information also includes weather-related information. The brightness estimation unit 115 further estimates the amount of brightness attenuation from the first area to the camera 4 based on the weather information shown by the environmental information, and corrects the brightness of the first area based on this attenuation amount. As a result, road markings can be identified with high accuracy regardless of the weather.

[0054] (5) The road marking recognition device 50 also includes a storage unit 12, which stores brightness ratio information showing the characteristics of the brightness ratio between the area inside the road marking and the area outside the road marking relative to the degree of damage to the road marking. Figure 5A ) and reflectance information showing the reflectance of road markings relative to this brightness ratio ( Figure 5B The luminance ratio estimation unit 115 estimates the luminance ratio between the first and second regions based on the degree of damage and luminance ratio information of the road markings identified by the damage rate identification unit 113. The reflectance estimation unit 116 estimates the reflectance of the first region based on the luminance ratio and reflectance information estimated by the luminance ratio estimation unit 115. Therefore, even when the road markings are worn or soiled, the reduction in road marking recognition accuracy can be suppressed. Furthermore, since a pre-prepared table (…) is used… Figure 5A , 5B The estimated value of reflectivity can be obtained, so no complicated calculation is required, which can reduce the processing load.

[0055] The above-described embodiments can be modified in various ways. Several modifications will be described below. In the above-described embodiment, the measurement point setting unit 112 identifies the road marking area (road marking recognition area) corresponding to the road markings based on the captured image obtained by the camera 4, which serves as the imaging unit. Additionally, the defect rate recognition unit 113 identifies the degree of damage to the road markings within the road marking area. However, the defect rate recognition unit may also function as a first recognition unit, identifying the road marking area corresponding to the road markings based on the captured image obtained by the imaging unit, and identifying the degree of damage to the road markings within that road marking area.

[0056] In addition, in the above-described embodiment, the illuminance estimation section 114 determines whether the measurement point MP0 is covered by a shadow of a structure around the host vehicle according to the position of the sun shown by the environmental information, and corrects the illuminance detected by the illuminance sensor 5 as the illuminance detector according to the determination result. However, the environmental information can also include map information including information of the structure, and the illuminance estimation section can determine whether the measurement point MP0 is covered by a shadow of a structure around the host vehicle according to the map information included in the environmental information.

[0057] In the above-described embodiment, the brightness estimation section 117 calculates the transmittance of the water film on the road surface based on the state of the water film on the road surface, and estimates the amount of attenuation of the brightness caused by the water film based on the transmittance. However, the brightness estimation section can calculate the refractive index instead of or together with the transmittance of the water film on the road surface. In this case, the brightness estimation section can estimate the amount of attenuation of the brightness caused by the water film based on the transmittance of the water film and the refractive index of the water film instead of or together with the transmittance of the water film.

[0058] In addition, in the above-described embodiment, the brightness ratio estimation section 115 estimates the brightness ratio of the first region to the second region according to the road marking defect rate of the measurement point MP0 using the brightness ratio table of Figure 5A However, the brightness ratio can be too large in backlight or dark portions at times. Therefore, in the brightness ratio table of Figure 5A , upper and lower limit values can be set to the brightness ratio according to the angle formed by the light source (the sun), the reflection point, and the camera 4, and the illuminance of the measurement point MP0 estimated by the illuminance estimation section 114.

[0059] In addition, the shutter speed of the camera 4 can be accelerated in an environment in which light exceeding the upper limit of the image sensor function of the camera 4 is incident to the camera 4 due to reflection of sunlight by the backlight of the sun, road surface reflection, an oncoming vehicle or a vehicle traveling ahead, another object, or the like. Thereby, it is possible to suppress the occurrence of so-called blooming in which a high-light portion (a bright portion) of the captured image of the camera 4 whitens due to the backlight of the sun or the like, and it is possible to further improve the recognition accuracy of the road marking.

[0060] Further, in the above-described embodiment, the end portion recognition section 118 as the second recognition section recognizes the position (pixel) of the end portion of the region (pixel group) having a luminance within a prescribed degree from the luminance of the first region estimated by the luminance ratio estimation section 115 as the end portion of the road marking. However, the end portion recognition section can recognize the position (pixel) of the end portion of the extension direction in addition to the end portion of the lane width direction of the above-described pixel group as the position of the end portion of the road marking. Thus, it is possible to recognize a broken line like a lane boundary line, a line on the road surface that defines a pedestrian crossing, and the like with high precision. Further, the end portion recognition section can search for a pixel having a luminance difference from the first region that is equal to or greater than a prescribed threshold value in the lane width direction, and recognize the pixel preceding the first detected pixel as the position (pixel) of the end portion of the road marking. The prescribed threshold value is determined in accordance with the luminance ratio BR corresponding to the road marking defect rate CF of the first region obtained from the table. Figure 5A In more detail, the end portion recognition section obtains the luminance of the second region by multiplying the luminance of the first region by the luminance ratio BR, and determines the luminance difference between the first region and the second region to be the prescribed threshold value.

[0061] Further, in the above-described embodiment, the road marking recognition device 50 is applied to an autonomous vehicle, but the road marking recognition device 50 can also be applied to a vehicle other than an autonomous vehicle. For example, the road marking recognition device 50 can also be applied to a manual driving vehicle equipped with an ADAS (Advanced driver-assistance systems).

[0062] The above description is merely an example, and the present application is not limited to the above-described embodiments and modified examples as long as the characteristics of the present application are not impaired. One or more of the above-described embodiments and modified examples can be arbitrarily combined, and each modified example can be combined with each other.

[0063] With the present application, it is possible to recognize road markings of a lane of a prescribed road with high precision.

[0064] The present application has been described above with reference to preferred embodiments. However, it will be appreciated by persons skilled in the art that various modifications and changes can be made without departing from the scope of the disclosure of the following claims.

Claims

1. A road marking recognition apparatus characterized by comprising: Possess: a photographing section (4) that photographs a situation of the surroundings of the host vehicle; an illuminance detector (5) that detects illuminance of the surroundings of the host vehicle; a first recognition section (113) that recognizes a road marking of a lane of a prescribed road from a photographed image acquired by the photographing section (4), and recognizes a degree of defect of the road marking in a manner that a ratio of a pixel group of a region having a luminance of less than a boundary value in a target region within the road marking to an area of the target region is recognized; an illuminance estimation section (114) that estimates illuminance of the road marking from the illuminance detected by the illuminance detector (5); a luminance ratio estimation section (115) that estimates a luminance ratio between a first region within the road marking and a second region outside the road marking in a manner that the luminance ratio decreases as the degree of defect of the road marking increases, from the degree of defect of the road marking recognized by the first recognition section (113); a reflectance estimation section (116) that estimates reflectance of the first region from the luminance ratio estimated by the luminance ratio estimation section (115); a luminance estimation section (117) that estimates luminance of the first region from the illuminance of the road marking estimated by the illuminance estimation section (114) and the reflectance of the first region estimated by the reflectance estimation section (116); and a second recognition section (118) that recognizes a position of an end portion of the road marking from the luminance of the first region estimated by the luminance estimation section (117).

2. The road marking recognition device according to claim 1, wherein the second recognition section (118) recognizes a position of an end portion in a lane width direction of a region containing the first region and having a luminance difference from the first region within a prescribed degree as the position of the end portion of the road marking, from the luminance of the first region estimated by the luminance estimation section (117).

3. The road marking recognition device according to claim 1, further comprising an information acquisition section (111) that acquires environmental information containing a position of the sun, the illuminance estimation section (114) determines whether the road marking is covered by a shadow of a structure in the surroundings of the host vehicle from the position of the sun shown by the environmental information, corrects the illuminance detected by the illuminance detector (5) according to a result of the determination, and estimates the illuminance of the road marking.

4. The road marking recognition device according to claim 3, wherein the illuminance estimation section (114) recognizes a position, size, and shape of the structure from a photographed image acquired by the photographing section (4), and determines whether the road marking is covered by the shadow of the structure in the surroundings of the host vehicle from a result of the recognition.

5. The road marking recognition device according to claim 3, wherein the environmental information further includes map information containing information of the structure, ​ ​ The illuminance estimation unit (114) determines whether the road marking is covered with a shadow of the structure around the host vehicle, based on the environmental information.

6. The road marking recognition device according to claim 3, wherein The environmental information further includes weather-related information, The luminance ratio estimation unit (115) further estimates an amount of attenuation of luminance from the first region to the photographing unit (4) based on the weather indicated by the environmental information, and corrects the luminance of the first region based on the amount of attenuation.

7. The road marking recognition device according to any one of claims 1 to 6, wherein Further provided is a storage unit (12) that stores luminance ratio information indicating a characteristic of a luminance ratio between a region within the road marking and a region outside the road marking with respect to a degree of damage of the road marking, and reflectance information indicating a characteristic of reflectance of the road marking with respect to the luminance ratio, The luminance ratio estimation unit (115) estimates a luminance ratio between the first region and the second region based on the degree of damage of the road marking recognized by the first recognition unit (113) and the luminance ratio information, The reflectance estimation unit (116) estimates reflectance of the first region based on the luminance ratio estimated by the luminance ratio estimation unit (115) and the reflectance information.

Images