Lane marking recognition device

Abstract

To accurately recognize a division line for regulating a lane of a road.SOLUTION: A division line recognition device 50 comprises: an information acquisition unit 111 which acquires environment information including a position of the sun; a scrape rate recognition unit 113 which recognizes a division line on the basis of a pickup image of a camera 4 and recognizes a scrape degree of the division line; an illuminance estimation unit 114 which estimates illuminance of the division line on the basis of illuminance detected by an illuminance sensor 5; a luminance ratio estimation unit 115 which estimates a luminance ratio between a first area in the division line and a second area outside the division line on the basis of the scrape degree of the division line; a reflectance estimation unit 116 which estimates reflectance of the first area on the basis of the estimated luminance ratio; a luminance estimation unit 117 which estimates luminance of the first area on the basis of the illuminance of the division line and the reflectance of the first area which are estimated; and an end recognition unit 118 which recognizes a position of an end of the division line on the basis of the estimated luminance of the first area.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a lane marking recognition device for recognizing lane markings on roads. [Background technology]

[0002] A known example of this type of device is one that recognizes lane markings on a road surface based on the brightness of an image of the road surface (see, for example, Patent Document 1). The device described in Patent Document 1 calculates the reflectance of the road surface based on the image of the road surface, and extracts, as candidates for lane markings, pixels whose brightness difference with adjacent pixels is equal to or greater than a threshold set based on the reflectance of the road surface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4023333 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in a captured image of a road surface, there may be pixels whose brightness difference with adjacent pixels is equal to or exceeds a threshold even in areas where lane markings do not exist. Therefore, if the device described in Patent Document 1 extracts pixels that are candidates for lane markings based on brightness difference, it may not be possible to accurately recognize lane markings. [Means for solving the problem]

[0005] A lane marking recognition device according to one aspect of the present invention includes an imaging unit that captures an image of the surroundings of the vehicle, an illuminance detector that detects illuminance around the vehicle, a first recognition unit that recognizes lane markings that define road lanes and recognizes the degree of friction of the lane markings based on the image captured by the imaging unit, an illuminance estimation unit that estimates the illuminance of the lane markings based on the illuminance detected by the illuminance detector, and a first region within the lane markings based on the degree of friction of the lane markings recognized by the first recognition unit. adjacent to the first region The system includes a brightness ratio estimation unit that estimates the brightness ratio with a second area outside the demarcation line, a reflectance estimation unit that estimates the reflectance of the first area based on the brightness ratio estimated by the brightness ratio estimation unit, a brightness estimation unit that estimates the brightness of the first area based on the illuminance of the demarcation line estimated by the illuminance estimation unit and the reflectance of the first area estimated by the reflectance estimation unit, and a second recognition unit that recognizes the position of the end of the demarcation line based on the brightness of the first area estimated by the brightness estimation unit. The lane marking recognition device further includes a memory unit that stores reflectance information indicating characteristics of the reflectance of the lane marking relative to the brightness ratio between the area inside the lane marking and the area outside the lane marking adjacent to the area. The reflectance estimation unit estimates the reflectance of the first area based on the brightness ratio estimated by the brightness ratio estimation unit and the reflectance information. [Effects of the Invention]

[0006] According to the present invention, it is possible to accurately recognize the dividing lines that define the lanes of a road. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a block diagram showing a configuration of a main part of a vehicle control device according to an embodiment of the present invention; [Figure 2] FIG. 2 is a diagram showing an example of a captured image of the area ahead of the vehicle; [Figure 3] A top view of part of the road in Figure 2. [Figure 4] FIG. 10 is a diagram for explaining the degree of friction of the lane markings. [Figure 5A] FIG. 10 is a diagram showing an example of a luminance ratio table. [Figure 5B] FIG. 10 is a diagram showing an example of a reflectance table. [Figure 6] FIG. 10 is a diagram for explaining calculation of brightness. [Figure 7] 2 is a flowchart showing an example of processing executed by the controller of FIG. 1; DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, an embodiment of the present invention will be described with reference to FIGS. 1 to 7. A lane marking recognition device according to an embodiment of the present invention can be applied to a vehicle having an automatic driving function, i.e., an automatically driven vehicle. Note that a vehicle to which a lane marking recognition device according to this embodiment is applied may be referred to as the host vehicle to distinguish it from other vehicles. The host vehicle may be an engine vehicle having an internal combustion engine (engine) as a driving source, an electric vehicle having a traction motor as a driving source, or a hybrid vehicle having an engine and a traction motor as driving sources. The host vehicle can be driven not only in an automatic driving mode in which no driving operation by the driver is required, but also in a manual driving mode in which the driver operates the vehicle.

[0009] When an autonomous vehicle is driving in autonomous driving mode (hereinafter referred to as "autonomous driving"), it recognizes the lane markings that define the road lanes based on image data obtained by an imaging unit installed in a predetermined location on the vehicle (for example, above the windshield).The autonomous vehicle controls its driving actuators according to the information on the recognized lane markings so that the vehicle drives near the center of its lane.

[0010] However, due to factors such as sunlight conditions, water on the road surface (hereinafter referred to as a water film), wear or dirt on the lane markings, etc., it may not be possible to accurately recognize lane markings based on image data obtained by the imaging unit. In such cases, there is a risk that the vehicle will not be able to travel automatically. It is possible to address the above-mentioned problems by separately installing various high-precision sensors in the vehicle to detect the conditions around the vehicle and the road surface. However, this may complicate the device configuration and increase costs. Therefore, in this embodiment, the lane marking recognition device is configured as follows.

[0011] 1 is a block diagram showing the main configuration of a vehicle control device having a lane marking recognition device according to an embodiment of the present invention. The vehicle control device 100 has a controller 10, a communication unit 1, a positioning unit 2, an internal sensor group 3, a camera 4, and a driving actuator AC. The vehicle control device 100 also has a lane marking recognition device 50 that forms part of the vehicle control device 100. The lane marking recognition device 50 recognizes lane markings on the road on which the vehicle is traveling, based on image data captured by the camera 4.

[0012] The communication unit 1 communicates with various servers (not shown) via networks including wireless communication networks such as the Internet and mobile phone networks, and acquires map information, driving history information, traffic information, and the like from the servers periodically or at any timing. Networks include not only public wireless communication networks but also closed communication networks established for each specific management area, such as wireless LAN, Wi-Fi (registered trademark), and Bluetooth (registered trademark). The acquired map information is output to the memory unit 12, where it is updated. The positioning unit (GNSS unit) 2 has a positioning sensor that receives positioning signals transmitted from positioning satellites. The positioning satellites are artificial satellites such as GPS satellites and quasi-zenith satellites. The positioning unit 2 measures the current position (latitude, longitude, altitude) of the vehicle using the positioning information received by the positioning sensor.

[0013] The internal sensor group 3 is a collective term for multiple sensors (internal sensors) that detect the driving state of the host vehicle. For example, the internal sensor group 3 includes a vehicle speed sensor that detects the vehicle speed of the host vehicle, an acceleration sensor that detects the longitudinal acceleration and lateral acceleration (lateral acceleration) of the host vehicle, a rotation speed sensor that detects the rotation speed of the driving source, and a yaw rate sensor that detects the rotation angular velocity around the vertical axis of the center of gravity of the host vehicle. The internal sensor group 3 also includes sensors that detect the driver's driving operations in manual driving mode, such as operation of the accelerator pedal, operation of the brake pedal, and operation of the steering wheel.

[0014] The camera 4 has an imaging element such as a CCD or CMOS and captures images of the surroundings (front, rear, and sides) of the vehicle. The illuminance sensor 5 has a light-receiving element and detects the brightness (illuminance) of light incident on the light-receiving element. The illuminance sensor 5 is installed outside the vehicle (for example, on the roof) or inside the vehicle (on the dashboard) so that it can detect the illuminance around the vehicle. Note that light passing through the windshield is attenuated to some extent by the glass, so if the illuminance sensor 5 is installed inside the vehicle, the sensor value may be corrected taking this attenuation amount into account.

[0015] Actuators AC are driving actuators for controlling the driving of the host vehicle. When the driving source is an engine, actuators AC include a throttle actuator that adjusts the opening of the engine's throttle valve (throttle opening). When the driving source is a driving motor, actuators AC include the driving motor. Actuators AC also include a brake actuator that operates the host vehicle's braking device and a steering actuator that drives the steering device.

[0016] The controller 10 is composed of an electronic control unit (ECU). More specifically, the controller 10 includes a computer having an arithmetic unit 11 such as a CPU (microprocessor), a storage unit 12 such as a ROM and RAM, and other peripheral circuits (not shown) such as an I / O interface. Note that although multiple ECUs with different functions, such as an engine control ECU, a traction motor control ECU, and a braking device ECU, can be provided separately, for convenience, the controller 10 is shown in FIG. 1 as a collection of these ECUs.

[0017] The memory unit 12 stores highly accurate, detailed map information (referred to as high-accuracy map information). The high-accuracy map information includes road position information, road shape (curvature, etc.) information, road gradient information, intersection and branch point position information, number of lanes information, lane width and lane position information (information on lane center positions and lane boundary lines), position information of landmarks (traffic lights, signs, buildings, etc.) on the map, and road surface profile information such as road surface irregularities. The memory unit 12 also stores information on various control programs, thresholds used in the programs, and other information.

[0018] The calculation unit 11 has, as functional components, an information acquisition unit 111, a measurement point setting unit 112, a friction rate recognition unit 113, an illuminance estimation unit 114, a luminance ratio estimation unit 115, a reflectance estimation unit 116, a luminance estimation unit 117, an edge recognition unit 118, and a driving control unit 119. As shown in FIG. 1 , the information acquisition unit 111, the measurement point setting unit 112, the friction rate recognition unit 113, the illuminance estimation unit 114, the luminance ratio estimation unit 115, the reflectance estimation unit 116, the luminance estimation unit 117, and the edge recognition unit 118 are included in the lane marking recognition device 50.

[0019] The information acquisition unit 111 acquires environmental information. Specifically, the information acquisition unit 111 estimates the position of the sun based on the current position of the vehicle measured by the positioning unit 2 and the current date and time. The estimated position of the sun may be an absolute position or a relative position with respect to the vehicle. The information acquisition unit 111 acquires information indicating the estimated position of the sun as environmental information. In addition, the information acquisition unit 111 acquires information relating to weather such as rainfall (hereinafter referred to as weather information) from an external server (not shown) via the communication unit 1 as environmental information.

[0020] The measurement point setting unit 112 sets measurement points on the road surface. FIG. 2 is a diagram showing an example of a captured image of the area ahead of the vehicle, captured by the camera 4 while the vehicle is traveling on the road RD. In FIG. 2, measurement points MP set on the road surface of the road RD are shown superimposed on the captured image. FIG. 3 is a diagram showing a portion of the road shown in FIG. 2 as viewed from above. The measurement point setting unit 112 recognizes areas on the road surface corresponding to the lane markings LL, LC, and LR based on the captured image captured by the camera 4. This estimation may be performed using machine learning or other image recognition techniques. Hereinafter, the recognized area will be referred to as the lane marking recognition area or simply the lane marking area. The measurement point setting unit 112 sets a measurement point MP0 at a distant position ahead in the traveling direction (a position a certain distance away from the vehicle). The measurement point MP0 is an area having a predetermined number of pixels on the captured image, and is located on the lane marking recognition area as shown in FIG. 2. In Fig. 2, the measurement point MP0 indicated by a solid line schematically represents the measurement point MP0 set at the current time (the time of Fig. 2), and the measurement point MP0 indicated by a dashed line schematically represents the measurement point MP0 set at a past time (when the host vehicle was traveling at a position earlier in the traveling direction than the current traveling position). Note that the method of setting the measurement point MP0 shown in Fig. 2 is one example, and the shape and size of the measurement point MP0 and the position at which the measurement point MP0 is set are not limited to this.

[0021] The friction rate recognition unit 113 recognizes the degree of friction of the lane markings at measurement point MP0 (hereinafter also referred to as the lane marking friction rate) based on the image captured by camera 4. Specifically, the friction rate recognition unit 113 sets the upper limit of brightness at measurement point MP0 outside the lane marking recognition area as a boundary value, and calculates the lane marking friction rate at measurement point MP0 by dividing the area (pixel group) at measurement point MP0 that has a brightness equal to or lower than the boundary value by the total area (total number of pixels) of measurement point MP1. Figure 4 is a diagram for explaining the lane marking friction rate. Characteristics f1 and f2 represent the brightness distribution of measurement points MP1 and MP2 acquired based on the image captured by camera 4, specifically, the number of pixels for each brightness. In the diagram, line BD represents the maximum brightness of measurement point MP2, and area FA represents the pixel group at measurement point MP1 that has a brightness equal to or lower than the maximum brightness BD. The lane marking friction rate at measurement point MP1 is calculated by dividing the number of pixels in area FA by the total number of pixels in characteristic f1.

[0022] The illuminance estimation unit 114 estimates the illuminance of the road surface ahead of the vehicle. In detail, first, the illuminance estimation unit 114 acquires a sensor value (illuminance) detected by the illuminance sensor 5. At this time, the illuminance estimation unit 114 acquires the illuminance detected by the illuminance sensor 5 under conditions excluding the influence of shadows. Furthermore, based on the relationship between the position of the sun and the position of the illuminance sensor 5, the illuminance of direct sunlight and the illuminance of sky light are separated according to the oblique incident light characteristics (light-receiving angle characteristics) of the illuminance sensor 5, and the respective illuminances are stored for the solar altitude and the date and time. That is, the illuminance estimation unit 114 acquires the illuminance of direct sunlight and the illuminance of sky light (ambient light). Next, the illuminance estimation unit 114 recognizes the position, size, and shape of structures around the vehicle based on the map information stored in the memory unit 12, and estimates the position, size, and shape of a shadow (area SD in FIG. 2) cast on the road ahead of the vehicle by a structure (building BL in FIG. 2) based on the position information of the sun included in the environmental information acquired by the information acquisition unit 111. At this time, the illuminance estimation unit 114 may estimate the position, size, and shape of the shadow cast on the road ahead of the vehicle by a structure based on an image acquired by the camera 4. The illuminance estimation unit 114 determines whether the measurement point MP0 is covered by a shadow based on the estimated position, size, and shape of the shadow of the structure. If the measurement point MP0 is covered by a shadow, the illuminance estimation unit 114 estimates the attenuation rate of the shadow's effect on sky light based on the sky factor, and corrects the illuminance detected by the illuminance sensor 5 based on the estimation result.

[0023] The luminance ratio estimation unit 115 estimates the luminance ratio between the measurement points MP1 and MP2 based on the lane marking friction rate of the measurement point MP0 recognized by the friction rate recognition unit 113. The measurement point setting unit 112 sets the measurement points MP1 and MP2 at positions earlier in the traveling direction than the measurement point MP0 (measurement point MP0 represented by a solid line in FIG. 2), as shown in FIG. 2. More specifically, the measurement points MP1 and MP2 corresponding to the measurement point MP0 (measurement point MP0 represented by a dashed line in FIG. 2) are set at the setting position of the measurement point MP0 that was set at a time in the past (when the host vehicle was traveling at a position earlier in the traveling direction than the host vehicle's current traveling position). In this way, the measurement point setting unit 112 sets the measurement points MP1 and MP2 in the target area (measurement point MP0) used to acquire the illuminance and friction rate after the target area (measurement point MP0) approaches the host vehicle to a certain extent. At this time, the measurement point setting unit 112 sets a measurement point MP1 within the lane line recognition area, and sets a measurement point MP2 corresponding to the measurement point MP1 outside the lane line recognition area, as shown in Figures 2 and 3. The measurement points MP1 and MP2 are rectangular areas with a width smaller than the width of the lane line, for example, an area the size of a postcard. Note that the measurement points MP1 and MP2 may have shapes other than rectangular. Furthermore, the sizes of the measurement points MP1 and MP2 and the positions at which the measurement points MP1 and MP2 are set are not limited to these.

[0024] In detail, the brightness ratio estimation unit 115 uses a brightness ratio table (hereinafter also referred to as brightness ratio information) to estimate the brightness ratio of measurement point MP2 to measurement point MP1 (brightness of MP1 / brightness of MP2) based on the lane marking abrasion rate of measurement point MP1. FIG. 5A is a diagram showing an example of the brightness ratio table. As shown in FIG. 5A, the brightness ratio table estimates the lane marking abrasion rate CF (CF1 to CF n ) the brightness ratio BR (BR1 to BR n ) The lane marking friction rate CF1 is the state where the lane markings are not worn or soiled, and corresponds to the state where the lane markings are most visible. nCF indicates that the lane markings have worn away and disappeared, and corresponds to the lowest visibility of lane markings. n-1 <CF n The brightness ratio BR decreases as the lane marking abrasion rate CF increases. The brightness ratio BR may have a linear characteristic with respect to the lane marking abrasion rate CF, or it may have other characteristics. In this way, by measuring and quantizing the degree of lane marking abrasion as brightness ratio data, the degree of lane marking abrasion can be quantified. As a result, it becomes possible to treat the degree of lane marking abrasion as a numerical value.

[0025] The brightness ratio table of Figure 5A is generated in advance based on the brightness of the inside and outside of the lane markings obtained from the captured images of multiple lane marking samples with different lane marking abrasion rates under the same environment (light source, rainfall, road conditions) using camera 4 or an imaging device with performance equivalent to camera 4, and is stored in memory unit 12.

[0026] The reflectance estimation unit 116 uses a reflectance table (hereinafter also referred to as reflectance information) to estimate the reflectance of the measurement point MP1 based on the luminance ratio between the measurement point MP1 and the measurement point MP2 estimated by the luminance ratio estimation unit 115. FIG. 5B is a diagram showing an example of the reflectance table. As shown in FIG. 5B, the reflectance table includes luminance ratios BR1 to BR2 between the inside and outside of the lane marking recognition area. n Reflectances RF1 to RF2 within the lane marking recognition area n The reflectance RF is information indicating the characteristics of the luminance ratio BR. The reflectance RF is a reflectance predicted based on the lane marking friction rate CF corresponding to the luminance ratio in FIG. 5A. The reflectance RF increases as the luminance ratio BR increases. The reflectance RF may have a linear characteristic with respect to the luminance ratio BR, or may have other characteristics.

[0027] 5B is generated in advance for each angle formed by the light source (sun), the reflection point (observation point), and the camera 4 (more specifically, the position of the lens of the camera 4) and stored in the storage unit 12. The reflectance estimation unit 116 calculates the angle formed by the light source (sun), the reflection point, and the camera 4 based on the position of the sun indicated by the environmental information acquired by the information acquisition unit 111 and the current position of the vehicle measured by the positioning unit 2, and reads out the reflectance table corresponding to the calculated angle from the storage unit 12 and uses it.

[0028] The luminance estimation unit 117 estimates the luminance of the measurement point MP1 based on the illuminance estimated by the illuminance estimation unit 114 and the reflectance estimated by the reflectance estimation unit 116. Specifically, the luminance estimation unit 117 calculates the luminance L(θ) of the measurement point MP1 using the following equations (i) and (ii). Hereinafter, the luminance of the measurement point MP1 estimated in this manner will be referred to as the estimated luminance. FIG. 6 is a diagram for explaining the calculation of the luminance. θ represents the angle of the line connecting the measurement point MP1 and the camera 4 with respect to the vertical direction, i.e., the reflection angle. E(θ) represents the illuminance when the reflection angle is θ. R represents the reflectance. I(θ) represents the luminous intensity of the light source S when the reflection angle is θ. r represents the linear distance between the measurement point MP1 and the camera 4.

number

number

[0029] Furthermore, the brightness estimation unit 117 estimates the condition of the road surface ahead of the vehicle based on the weather information included in the environmental information acquired by the information acquisition unit 111 and the current position of the vehicle measured by the positioning unit 2. The brightness estimation unit 117 calculates the transmittance of the water film on the road surface based on the estimated road surface condition, more specifically, the condition of the water film on the road surface. The water film may include water containing ice crystals or particles, such as snow covering the road surface. The condition of the water film refers to the depth of puddles or the depth of accumulated snow. Furthermore, if the vehicle is equipped with a rain sensor (raindrop sensor), the brightness estimation unit 117 may estimate the condition of the water film on the road surface based on the sensor value of the rain sensor. Furthermore, the brightness estimation unit 117 may recognize the conditions of rainfall or snowfall based on images captured by the camera 4 and estimate the condition of the water film on the road surface.

[0030] The brightness estimation unit 117 further estimates the amount of brightness attenuation caused by the water film (the amount of brightness attenuation from measurement point MP1 to camera 4) based on the calculated transmittance of the water film, and corrects the estimated brightness of measurement point MP1 based on the estimated amount of attenuation. This allows for more accurate estimation of the brightness of measurement point MP1 on the image captured by camera 4. Note that, since light passing through the windshield is attenuated to some extent by the glass, if camera 4 is installed inside the vehicle, the estimated brightness of measurement point MP1 may be corrected taking this amount of attenuation into account.

[0031] The edge recognition unit 118 recognizes the position of the edge (edge) of the lane marking based on the estimated brightness calculated by the brightness estimation unit 117 (or the estimated brightness after the correction, if the correction has been performed). More specifically, the edge recognition unit 118 recognizes the position (pixel) of the edge in the lane width direction of an area (pixel group) that includes the measurement point MP1 on the image captured by the camera 4 and whose brightness difference from the measurement point MP1 is within a predetermined range, as the position of the edge of the lane marking.

[0032] In the automatic driving mode, the driving control unit 119 generates a target trajectory based on the lane markings recognized by the lane marking recognition device 50, and controls the actuator AC so that the vehicle travels along the target trajectory. In the manual driving mode, the driving control unit 119 controls the actuator AC in response to a driving command (such as a steering operation) from the driver acquired by the internal sensor group 3.

[0033] Fig. 7 is a flowchart showing an example of processing executed by the controller 10 of Fig. 1 in accordance with a predetermined program. The processing shown in the flowchart of Fig. 7 is repeated, for example, at predetermined intervals while the host vehicle is traveling in autonomous driving mode. Also, for example, the processing is repeated every time the host vehicle travels a predetermined distance while traveling in autonomous driving mode.

[0034] First, in step S11, environmental information is acquired. Specifically, the position of the sun is estimated based on the current position of the vehicle measured by the positioning unit 2 and the current date and time, and sun position information (information indicating latitude and longitude) is acquired. Weather information is also acquired from an external server (not shown) via the communication unit 1. In step S12, an area corresponding to the lane markings is recognized based on an image of the area in front of the vehicle acquired by the camera 4, and a measurement point MP0 is set based on the recognition result. In step S13, an image acquired by the camera 4 is acquired, and the lane marking friction rate at the measurement point MP0 is recognized based on the image. In step S14, the illuminance at the measurement point MP0 is estimated based on the sensor value of the illuminance sensor 5.

[0035] In step S15, the positions, sizes, and shapes of structures around the vehicle are recognized based on the map information stored in the memory unit 12. Then, based on the recognized positions, sizes, and shapes of the structures and the sun position information acquired in step S11, the position, size, and shape of the shadow cast by the structures on the road ahead of the vehicle are estimated. Furthermore, based on the estimation results, it is determined whether a shadow cast by the structure is cast on measurement point MP0. If the result in step S15 is positive, the process proceeds to step S17. If the result in step S15 is negative, the state of the water film on measurement point MP0 is determined in step S16 based on the weather information acquired in step S11. More specifically, it is determined whether a water film exists on measurement point MP0. If the result in step S16 is positive, the illuminance estimated in step S14 is corrected in step S17.

[0036] If the result in step S16 is negative, in step S18, measurement points MP1 and MP2 corresponding to measurement point MP0 set in step S12 are set using the brightness ratio table of Figure 5A, and the brightness ratio between measurement points MP1 and MP2 is estimated based on the lane marking abrasion rate of measurement point MP0 recognized in step S13. In step S19, the reflectance of measurement point MP1 is estimated based on the brightness ratio estimated in step S18 using the reflectance table of Figure 5B. In step S20, the estimated brightness of measurement point MP1 is calculated using the above equations (i) and (ii) based on the illuminance estimated in step S14 and the reflectance estimated in step S19, and the position of the end of the lane width direction of the area where the brightness difference from the estimated brightness is within a predetermined range is recognized as the position of the end of the lane marking.

[0037] According to this embodiment, the following effects can be achieved. (1) The lane marking recognition device 50 includes a camera 4 that captures an image of the surroundings of the vehicle, an illuminance sensor 5 that detects the illuminance around the vehicle, a friction rate recognition unit 113 that recognizes lane markings that define road lanes and recognizes the degree of friction of the lane markings (more specifically, measurement point MP0) based on the captured image acquired by the camera 4, an illuminance estimation unit 114 that estimates the illuminance of the lane markings based on the illuminance detected by the illuminance sensor 5, and a friction rate recognition unit 113 that recognizes the degree of friction of the lane markings based on the degree of friction of the lane markings recognized by the friction rate recognition unit 113. The system includes a brightness ratio estimation unit 115 that estimates the brightness ratio between the first area and the second area outside the dividing line, a reflectance estimation unit 116 that estimates the reflectance of the first area based on the brightness ratio estimated by the brightness ratio estimation unit 115, a brightness estimation unit 117 that estimates the brightness of the first area based on the illuminance of the dividing line estimated by the illuminance estimation unit 114 and the reflectance of the first area estimated by the reflectance estimation unit 116, and an end recognition unit 118 that recognizes the position of the end of the dividing line based on the brightness of the first area estimated by the brightness estimation unit 117.

[0038] In this way, rather than simply recognizing the lane markings using the image captured by camera 4, the position of the end of the lane markings is recognized by taking into account the degree of friction of the lane markings obtained based on the image captured by camera 4, thereby enabling accurate recognition of the lane markings. As a result, unnecessary or sudden delegation of driving authority to the driver, which occurs due to incorrect or inability to recognize lane markings, can be suppressed. In addition, since the target trajectory is generated based on accurately recognized lane markings, stable automated driving becomes possible, improving the riding comfort of passengers. Furthermore, since lane markings can be accurately recognized based on the image captured by camera 4, other sensors such as lidar or radar are not required, and the device configuration can be simplified.

[0039] (2) The edge recognition unit 118 recognizes the position of the edge of the lane width direction of an area that includes the first area and whose brightness difference from the first area is within a predetermined range as the position of the edge of the lane marking, based on the brightness of the first area estimated by the brightness estimation unit 117. This allows for accurate recognition of the boundary between the lane marking that defines the lane of the road and the area beyond the lane marking.

[0040] (3) The lane marking recognition device 50 further includes an information acquisition unit 111 that acquires environmental information including the position of the sun. The illuminance estimation unit 114 determines whether the lane marking is obscured by the shadow of a structure surrounding the vehicle based on the position of the sun indicated by the environmental information, and estimates the illuminance of the lane marking by correcting the illuminance detected by the illuminance sensor 5 based on the determination result. At this time, the illuminance estimation unit 114 recognizes the position, size, and shape of the structure based on the captured image acquired by the camera 4, and determines whether the lane marking is obscured by the shadow of a structure surrounding the vehicle based on the recognition result. This allows the lane marking to be recognized accurately even in situations where shadows are cast on the road by buildings or plants on the side of the road.

[0041] (4) The environmental information further includes information about the weather. The brightness ratio estimation unit 115 further estimates the amount of brightness attenuation between the first region and the camera 4 based on the weather indicated by the environmental information, and corrects the brightness of the first region based on the amount of attenuation. This allows the lane markings to be recognized accurately regardless of the weather.

[0042] (5) The lane marking recognition device 50 further includes a memory unit 12 that stores brightness ratio information ( FIG. 5A ) that indicates the characteristics of the brightness ratio between the area inside the lane marking and the area outside the lane marking, relative to the degree of lane marking friction, and reflectance information ( FIG. 5B ) that indicates the characteristics of the lane marking's reflectance relative to the brightness ratio. The brightness ratio estimation unit 115 estimates the brightness ratio between the first area and the second area based on the degree of lane marking friction recognized by the friction ratio recognition unit 113 and the brightness ratio information. The reflectance estimation unit 116 estimates the reflectance of the first area based on the brightness ratio estimated by the brightness ratio estimation unit 115 and the reflectance information. This prevents a decrease in lane marking recognition accuracy even when the lane marking is worn or soiled. Furthermore, because the estimated reflectance is obtained using pre-prepared tables ( FIGS. 5A and 5B ), complex calculations are not required, reducing the processing load.

[0043] The above embodiment can be modified in various ways. Some modified examples will be described below. In the above embodiment, the measurement point setting unit 112 recognizes a demarcation line area (demarcation line recognition area) corresponding to the demarcation line based on an image captured by the camera 4 as an imaging unit. Furthermore, the friction rate recognition unit 113 recognizes the degree of friction of the demarcation line in a first area within the demarcation line area. However, the friction rate recognition unit may function as a first recognition unit and recognize the demarcation line area corresponding to the demarcation line and the degree of friction of the demarcation line in the first area within the demarcation line area based on an image captured by the imaging unit.

[0044] In the above embodiment, the illuminance estimation unit 114 determines whether the first area is covered by the shadow of a structure around the vehicle based on the position of the sun indicated by the environmental information, and corrects the illuminance detected by the illuminance sensor 5 serving as an illuminance detector based on the determination result. However, the environmental information may further include map information including information about the structure, and the illuminance estimation unit may determine whether the first area is covered by the shadow of a structure around the vehicle based on the map information included in the environmental information.

[0045] In the above embodiment, the brightness estimation unit 117 calculates the transmittance of the water film on the road surface based on the state of the water film on the road surface, and estimates the amount of luminance attenuation due to the water film based on the transmittance. However, the brightness estimation unit may calculate the refractive index instead of or in addition to the transmittance of the water film on the road surface. In this case, the brightness estimation unit may estimate the amount of luminance attenuation due to the water film based on the refractive index of the water film instead of or in addition to the transmittance of the water film.

[0046] In the above embodiment, the brightness ratio estimation unit 115 estimates the brightness ratio between the first and second regions based on the division line rub rate of the first region using the brightness ratio table of Fig. 5A. However, the brightness ratio may be excessively large in backlit or dark areas. Therefore, upper and lower limits may be set for the brightness ratio in the brightness ratio table of Fig. 5A depending on the angle between the light source (sun), the reflection point, and the camera 4, and the illuminance of the first region estimated by the illuminance estimation unit 114.

[0047] Furthermore, in an environment where light exceeding the upper limit of the image sensor function of camera 4 enters camera 4 due to backlighting from the sun, glare from the road surface, or reflection of sunlight from oncoming vehicles, vehicles ahead, or other objects, the shutter speed of camera 4 may be increased. This makes it possible to prevent highlights (bright areas) in the image captured by camera 4 from becoming washed out due to backlighting from the sun or the like, which is known as whiteout, and further improves the accuracy of lane marking recognition.

[0048] In the above embodiment, the edge recognition unit 118, which serves as the second recognition unit, recognizes the position (pixel) of the edge in the lane width direction of a region (pixel group) having a luminance within a predetermined range of difference from the luminance of the first region estimated by the luminance ratio estimation unit 115 as the edge of the lane line. However, the edge recognition unit may recognize the position (pixel) of the edge in the extension direction of the pixel group as the position of the edge of the lane line, in addition to the edge in the lane width direction. This allows for accurate recognition of dashed lane boundary lines, lines on the road surface defining pedestrian crossings, and the like. Alternatively, the edge recognition unit may search the lane width direction for pixels whose luminance difference from the first region is equal to or greater than a predetermined threshold, and recognize the pixel immediately preceding the first detected pixel as the position (pixel) of the edge of the lane line. The predetermined threshold is determined based on the luminance ratio BR corresponding to the lane line abrasion rate CF of the first region, obtained from the table of FIG. 5A. More specifically, the edge recognition unit obtains the luminance of the second region by multiplying the luminance of the first region by the luminance ratio BR, and determines the luminance difference between the first region and the second region as a predetermined threshold value.

[0049] Furthermore, in the above embodiment, the lane marking recognition device 50 is applied to an autonomous vehicle, but the lane marking recognition device 50 can also be applied to vehicles other than autonomous vehicles. For example, the lane marking recognition device 50 can also be applied to manually driven vehicles equipped with ADAS (Advanced Driver-Assistance Systems).

[0050] The above description is merely an example, and the present invention is not limited to the above-described embodiment and modifications as long as the features of the present invention are not impaired. One or more of the above-described embodiment and modifications can be arbitrarily combined, and modifications can also be combined with each other. [Explanation of symbols]

[0051] 1 communication unit, 2 positioning unit, 3 internal sensor group, 4 camera, 5 illuminance sensor, 10 controller, 12 memory unit, 111 information acquisition unit, 112 measurement point setting unit, 113 friction rate recognition unit, 114 illuminance estimation unit, 115 luminance ratio estimation unit, 116 reflectance estimation unit, 117 luminance estimation unit, 118 edge recognition unit, 119 travel control unit, AC actuator

Claims

1. an imaging unit that captures an image of the surroundings of the vehicle; an illuminance detector that detects illuminance around the vehicle; a first recognition unit that recognizes lane markings that define road lanes and recognizes the degree of friction of the lane markings based on the captured image acquired by the imaging unit; an illuminance estimation unit that estimates the illuminance of the lane marking based on the illuminance detected by the illuminance detector; a brightness ratio estimation unit that estimates a brightness ratio between a first region within the demarcation line and a second region outside the demarcation line adjacent to the first region, based on the degree of friction of the demarcation line recognized by the first recognition unit; a reflectance estimation unit that estimates the reflectance of the first region based on the luminance ratio estimated by the luminance ratio estimation unit; a brightness estimation unit that estimates a brightness of the first area based on the illuminance of the lane marking estimated by the illuminance estimation unit and the reflectance of the first area estimated by the reflectance estimation unit; a second recognition unit that recognizes the position of an end of the demarcation line based on the luminance of the first area estimated by the luminance estimation unit, The method further includes a storage unit that stores reflectance information indicating a characteristic of the reflectance of the demarcation line relative to a luminance ratio between an area inside the demarcation line and an area outside the demarcation line adjacent to the area, The lane marking recognition device is characterized in that the reflectance estimation unit estimates the reflectance of the first area based on the brightness ratio estimated by the brightness ratio estimation unit and the reflectance information.

2. The lane marking recognition device according to claim 1, The second recognition unit recognizes the position of the end of the lane width direction of an area that includes the first area and whose brightness difference from the first area is within a predetermined range based on the brightness of the first area estimated by the brightness estimation unit, as the position of the end of the lane width direction of the lane marking.

3. The lane marking recognition device according to claim 1 or 2, further comprising an information acquisition unit that acquires environmental information including the position of the sun; The illuminance estimation unit determines whether the lane marking is covered by the shadow of a structure surrounding the vehicle based on the position of the sun indicated by the environmental information, and corrects the illuminance detected by the illuminance detector based on the determination result to estimate the illuminance of the lane marking.

4. The lane marking recognition device according to claim 3, The illuminance estimation unit recognizes the position, size, and shape of the structure based on the captured image acquired by the imaging unit, and determines, based on the recognition result, whether the lane marking line is covered by the shadow of the structure surrounding the vehicle.

5. The lane marking recognition device according to claim 3 or 4, the environmental information further includes map information including information about the structure; The lane marking recognition device is characterized in that the illuminance estimation unit determines, based on the environmental information, whether the lane marking is covered by the shadow of the structure surrounding the vehicle.

6. The lane marking recognition device according to any one of claims 3 to 5, The environmental information further includes information about weather, The brightness ratio estimation unit further estimates the amount of brightness attenuation between the first area and the imaging unit based on the weather indicated by the environmental information, and corrects the brightness of the first area based on the amount of attenuation.

7. The lane marking recognition device according to any one of claims 1 to 6, The storage unit further stores brightness ratio information indicating a characteristic of a brightness ratio between an area inside the demarcation line and an area outside the demarcation line with respect to a degree of friction of the demarcation line, A lane marking recognition device characterized in that the brightness ratio estimation unit estimates the brightness ratio between the first area and the second area based on the degree of friction of the lane marking recognized by the first recognition unit and the brightness ratio information.

Images