Road marking detection method and road marking detection device

Abstract

The simple configuration suppresses the influence of sunlight on road marking detection. [Solution] In the road marking detection method, an ambient image is generated by an imaging device installed on the vehicle, capturing the area around the vehicle (S2). The brightness value of a portion of the road surface image included in the ambient image that extends along the direction of travel of the vehicle is detected (S5). If the brightness value changes at a gradient such that the change in brightness value between adjacent positions in the direction of travel of the portion image is less than a predetermined value, the ambient image is corrected by brightness correction, and road markings are detected from the corrected ambient image (S13, S17). If the change in brightness value between adjacent positions in the direction of travel of the portion image is greater than or equal to a predetermined value, the ambient image is corrected by contrast correction, and road markings are detected from the corrected ambient image (S14, S17).

Description

Technical Field

[0001] The present invention relates to a road marking detection method and a road marking detection device.

Background Art

[0002] In Patent Document 1, a lane marking recognition device that performs robust road lane white line detection by estimating the reflectance of a road lane white line in consideration of the influence of shadows such as buildings obtained from the sun position, illuminance, and map has been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the lane marking recognition device described in Patent Document 1, in order to grasp the light environment of the driving scene, illuminance measurement considering the attenuation of light by clouds or the like in the optical path from the sun as a light source to the road surface, and light propagation simulation considering the light shielding region by structures or the like were performed. For this reason, there was a problem that a large number of dedicated sensors and high-performance arithmetic processing devices were required, resulting in high costs. An object of the present invention is to suppress the influence of sunlight in the detection of road markings with a simple configuration.

Means for Solving the Problems

[0005] In a road marking detection method according to one aspect of the present invention, an ambient image is generated by an imaging device installed on the vehicle, the brightness value of a portion of the road surface image included in the ambient image that extends along the direction of travel of the vehicle is detected, and if the brightness value changes at a gradient such that the change in brightness value between adjacent positions in the direction of travel of the portion image is less than a predetermined value, the ambient image is corrected by brightness correction, and road markings are detected from the corrected ambient image. If the change in brightness value between adjacent positions in the direction of travel of the portion image is greater than or equal to a predetermined value, the ambient image is corrected by contrast correction, and road markings are detected from the corrected ambient image. [Effects of the Invention]

[0006] According to the present invention, the influence of sunlight on the detection of road markings can be suppressed with a simple configuration. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram of an example of a driving assistance device according to an embodiment. [Figure 2] This is a schematic diagram of an example of a surrounding image. [Figure 3] (a) to (d) are diagrams illustrating the effect of sunlight on images of the road surface around the vehicle. [Figure 4] (a) to (c) are characteristic diagrams showing the conversion characteristics from incident light intensity to brightness value for pixels of the image sensor. [Figure 5] This is a flowchart of an example of a driving assistance method in an embodiment. [Figure 6] Figure 1 is a block diagram showing an example of the controller's functional configuration. [Figure 7] (a) and (b) are schematic diagrams of an example of setting the observation area. [Figure 8] This is an explanatory diagram for image integration of surrounding images. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. Note that the drawings are schematic and may differ from actual ones. Furthermore, the embodiments of the present invention described below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the structure, arrangement, etc., of the components described below. The technical concept of the present invention can be modified in various ways within the technical scope defined by the claims described in the patent claims.

[0009] (composition) Figure 1 is a schematic diagram of an example of a driving support device according to an embodiment. The vehicle 1 is equipped with a driving support device 10 that assists in the driving of the vehicle 1. The driving support control by the driving support device 10 may be, for example, autonomous driving control that automatically drives the vehicle 1 without driver intervention based on the driving environment around the vehicle 1. In autonomous driving control, the driving support device 10 detects the position of road markings (e.g., white lines, etc.) on the road on which the vehicle 1 is traveling, and controls the steering angle, driving force, and braking force of the vehicle 1 so that the vehicle 1 does not deviate from the driving lane by crossing the road markings that represent the boundary of the lane (i.e., the road on which the vehicle 1 is traveling). The driving support device 10 is an example of the "road marking detection device" described in the claims.

[0010] For example, the driving support control by the driving support device 10 includes driver assistance control that controls at least the yaw behavior and vehicle speed of the vehicle 1 to assist the driver in driving the vehicle 1. In the driver assistance control, the driving support device 10 detects the position of the road markings on the road on which the vehicle 1 is traveling, and controls the steering angle, driving force, and braking force of the vehicle 1 so that the vehicle 1 does not deviate from the driving lane by crossing the road markings that represent the boundary of the driving lane in which the vehicle 1 is traveling. For example, the driving support device 10 may control the yaw behavior of the vehicle 1 by controlling the steering angle of the vehicle 1 or the difference in braking and driving force between the left and right wheels.

[0011] The driving support system 10 includes a camera 11, a vehicle sensor 12, a positioning device 13, a map database (map DB) 14, an actuator 15, and a controller 16. Camera 11 is an imaging device installed on the vehicle 1 that captures images of the area around the vehicle 1. The mounting position, optical axis, and field of view of camera 11 are set so that it can capture images of the road surface around the vehicle 1, so that the driving support system 10 can detect road markings on the road surface around the vehicle 1 from the images captured by camera 11.

[0012] For example, camera 11 may include a front camera that photographs the area in front of the vehicle 1, and the driving support device 10 may detect road markings in front of the vehicle 1 from the image captured by the front camera. The optical axis of the front camera may be directed forward in the longitudinal direction of the vehicle body of the vehicle 1. The camera 11 may also include a rear camera, a right-side camera, and a left-side camera that photograph the rear, right, and left sides of the vehicle 1, respectively. The optical axes of the rear camera, right-side camera, and left-side camera may be directed towards the rear in the longitudinal direction and to the right and left sides in the lateral direction of the vehicle 1, respectively. The driving support device 10 may detect road markings on the rear, right, and left sides of the vehicle 1 from the images captured by the rear camera, right-side camera, and left-side camera.

[0013] The vehicle sensor 12 is mounted on the vehicle 1 and detects various information obtained from the vehicle 1 (hereinafter sometimes referred to as "vehicle information") and outputs it to the controller 16. The vehicle sensor 12 may include, for example, a vehicle speed sensor for detecting the vehicle speed of the vehicle 1, a three-axis acceleration sensor for detecting the acceleration and deceleration of the vehicle 1 in the three axes, a steering angle sensor for detecting the steering angle of the steering wheel, a steering angle sensor for detecting the turning angle of the steering wheels, a gyro sensor for detecting the angular velocity of the vehicle 1, a yaw rate sensor for detecting the yaw rate, an accelerator sensor for detecting the accelerator opening, and a brake sensor for detecting the amount of brake operation.

[0014] The positioning device 13 is equipped with a Global Navigation Satellite System (GNSS) receiver and receives radio waves from multiple navigation satellites to measure the current position and direction of travel of the vehicle 1. The GNSS receiver may be, for example, a GPS receiver. The positioning device 13 may also be, for example, an inertial navigation device. Map DB 14 stores map data. The map data stored in Map DB 14 may be, for example, map data for navigation and highly accurate map data suitable as a map for autonomous driving. Map DB 14 may also include information on the road structure of the road as map data. The information on the road structure may include, for example, information on road shape and lane shape.

[0015] Actuator 15 operates the steering device, accelerator opening, and braking device of host vehicle 1 in response to a control signal from Controller 16 to generate the vehicle behavior of host vehicle 1. Actuator 15 includes a steering actuator, an accelerator opening actuator, and a brake control actuator. The steering actuator controls the steering direction and amount of the steering device. The accelerator opening actuator controls the accelerator opening. The brake control actuator controls the braking operation of the braking device.

[0016] Controller 16 is an electronic control unit that performs driving support control of host vehicle 1. Controller 16 includes a processor 16a and peripheral components such as a storage device 16b. Processor 16a may be, for example, a CPU or MPU. Storage device 16b may include a semiconductor storage device, a magnetic storage device, an optical storage device, etc. Storage device 16b may include memories such as ROM and RAM used as a main storage device, registers, and cache memories. The functions of Controller 16 described below are realized, for example, when processor 16a executes a computer program stored in storage device 16b. Controller 16 may be formed by dedicated hardware for executing each information processing described below. For example, Controller 16 may include a functional logic circuit set in a general-purpose semiconductor integrated circuit. For example, Controller 16 may have a PLD such as an FPGA.

[0017] Subsequently, an outline of the road lane line detection method of the embodiment will be described. Controller 16 acquires an image captured by camera 11 of the surroundings of host vehicle 1 as a surrounding image. An example of the surrounding image is schematically shown in FIG. 2. FIG. 2 shows an example of the surrounding image IM captured by the front camera that captures the front of the host vehicle 1 (that is, the surrounding image captured by the camera 11 whose optical axis is directed forward in the front-rear direction of the host vehicle 1).

[0018] The example of the surrounding image IM in FIG. 2 includes an image of the road RD on which the host vehicle 1 travels, images of the road marking lines LN1 to LN4 laid on the road surface of the road RD, and images of the buildings BD1 to BD4 beside the road RD. The same applies to the examples of the surrounding image IM in FIGS. 7(a) and 7(b) described later and the examples of the surrounding images IM1 to IM3 in FIG. 8. The area surrounded by the dashed-dotted line Rv indicates a partial area extending along the vertical direction of the surrounding image IM among the images of the road surface included in the surrounding image IM, and the dashed-dotted line Rv is not a part of the surrounding image IM.

[0019] The controller 16 detects the road marking lines LN2 and LN3 by detecting the boundary (edge) between the road marking lines LN2 and LN3 representing the lane in which the host vehicle 1 travels and the road surface other than the marking lines based on the luminance change of the image of the road RD in front of the host vehicle 1 in the surrounding image IM.

[0020] Here, the luminance characteristics in the surrounding image IM change depending on the positional relationship between the sun 6 and the host vehicle 1. FIG. 3(a) shows a state in which the sun 6 is located in front of the host vehicle 1 and has a relatively low altitude. In this case, since the sunlight from the sun 6 travels toward the host vehicle 1 as the reflected light RY that is specularly reflected by the road surface, the light specularly reflected by the road surface enters the camera 11, and the luminance characteristic is such that the luminance value gradually decreases as it moves away from the point where the incident angle and the reflection angle on the road surface are the same, with the luminance at that point being a predetermined maximum value. The predetermined maximum value of the luminance that the surrounding image IM can take is, for example, "255" when the luminance is represented by 8-bit data, and the minimum value of the luminance that the surrounding image IM can take may be "0". Although FIG. 3(a) has described an example of capturing the surrounding image with the front camera, the same phenomenon occurs when capturing the surrounding image with the rear camera if the sun 6 is behind the host vehicle 1 and has a relatively low altitude.

[0021] Figure 3(b) shows a situation where the sun 6 is located to the side of the vehicle 1 and at a relatively low altitude. In this case, the sunlight from the sun 6 is reflected light RY from the road surface and heads toward the side of the vehicle 1. Therefore, the light that is specularly reflected from the road surface does not enter the camera 11, and only the light that is diffusely reflected from the road surface enters the camera 11, resulting in a uniform brightness value for the road surface image. However, if there are buildings BD1 and BD2 next to the road, their shadows S will be cast in places on the road surface, causing road markings to exist in both sunny and shaded areas, and the contrast difference between the sunny and shaded areas of the road marking image may become large.

[0022] Figure 3(c) shows a state where the sun 6 is relatively high in the sky. In this case, the sunlight from the sun 6 is specularly reflected from the road surface, and the reflected light RY is directed upward. Therefore, the light that is specularly reflected from the road surface does not enter the camera 11, and the influence of the contrast difference in the image of the road markings between the sunlit and shaded areas due to the shadows S of buildings BD1 and BD2 next to the road is small. Therefore, by determining the luminance characteristics of these ambient images (IM), the positional relationship between the sun (6) and the vehicle (1) can be estimated.

[0023] Figure 3(d) shows an example of the vertical luminance change in a subregion Rv of the road surface image included in the surrounding image IM of Figure 2. The dashed line shows the change in brightness when the sun 6 is located in front of the vehicle 1 and at a relatively low altitude (Figure 3(a)), the dotted line shows the change in brightness when the sun 6 is located to the side of the vehicle 1 and at a relatively low altitude (Figure 3(b)), and the solid line shows the change in brightness when the sun 6 is at a relatively high altitude (Figure 3(c)).

[0024] When the sun 6 is in front of and low to the vehicle 1 (dashed line), the point where the angle of incidence and the angle of reflection on the road surface are equal is defined as the point of maximum brightness, and the brightness value gradually decreases as you move away from that point. Therefore, in the vertical direction of the surrounding image IM, if the brightness value changes approximately linearly with a gradient such that the change in brightness value between adjacent positions is less than a predetermined value, it can be estimated that the sun 6 is in front of and low to the vehicle 1 (Figure 3(a)).

[0025] When the sun 6 is to the side and low to the vehicle 1 (dotted line), a large contrast difference occurs between the sunlit and shaded areas due to the shadow of the building next to the road. Therefore, if the change in brightness value between adjacent positions in the vertical direction of the surrounding image IM exceeds a predetermined value, it can be estimated that the sun 6 is to the side and low to the vehicle 1 (Figure 3(b)).

[0026] When the altitude of the sun 6 is relatively high (solid line), the change in brightness values ​​in the vertical direction of the surrounding image IM is small, and the brightness values ​​are almost uniform. Therefore, if the change in brightness values ​​in the road surface image is below a threshold smaller than a predetermined value, it can be estimated that the altitude of the sun 6 is relatively high (Figure 3(c)).

[0027] In the above positional relationships, when the altitude of the sun (6) is relatively low (Figures 3(a) and 3(b)), the influence of sunlight on the brightness of the road surface image can make it difficult to detect the edges of the road markings. The reason for this is explained below. As shown in Figure 3(a), when the sun 6 is in front of the vehicle 1 and at a low position, specular reflection of sunlight may occur, causing saturation of the brightness of the highlight portion of the surrounding image IM (so-called "overexposure"). This can prevent the detection of the brightness difference between the road surface and the road markings, which is necessary for detecting road markings, from being detected due to brightness saturation.

[0028] The reason for this is explained below. Figures 4(a) and 4(b) show examples of the conversion characteristics from the incident light intensity to the pixels of the image sensor of camera 11 to the brightness values ​​of the ambient image IM. Figure 4(a) is an example of the conversion characteristics in a normal state where specular reflection is not captured, and Figure 4(b) is an example of the conversion characteristics in a state where specular reflection is captured.

[0029] When the camera 11 generates the surrounding image IM, the exposure of the camera 11 is adjusted so that the brightness levels BL1 and BL2 corresponding to the incident light intensity IL1 and IL2 from the road division lines (hereinafter referred to as "division line incident light intensity") exceed the saturation brightness value BS (corresponding to the "predetermined maximum value" in the claim). When specular reflection is captured (Figure 4(b)), the incident light intensity IL2 of the segmental line in the area with specular reflection is significantly higher than the incident light intensity IL1 of the segmental line in the state where specular reflection is not captured (Figure 4(a)).

[0030] As a result, the slope of the conversion characteristic from incident light intensity to luminance value is steeper when specular reflection is not captured (Figure 4(a)) than when specular reflection is captured (Figure 4(b)). On the other hand, when specular reflection is captured (Figure 4(b)), a brightness problem occurs in which the luminance value outside the specular reflection area becomes generally smaller (i.e., darker).

[0031] Therefore, in a state where specular reflection is not captured (Figure 4(a)), a certain degree of brightness difference ΔB1 is generated between the brightness BR1 (hereinafter referred to as "road surface brightness") with respect to the incident light intensity IR1 from the road surface other than the road marking lines (hereinafter referred to as "road surface incident light intensity") and the marking line brightness BL1, so the boundaries (edges) of the road marking lines used for detecting road marking lines can be observed.

[0032] On the other hand, in the state where specular reflection is captured (Figure 4(b)), the incident light intensity IL2 of the road markings and the incident light intensity IR2 of the road surface at the location where specular reflection occurs are intensities that are multiplied by a certain factor compared to the incident light intensity IL1 of the road markings and the incident light intensity IR1 of the road surface at locations without specular reflection.

[0033] Therefore, a certain brightness difference ΔB2 occurs between the luminance BL2 of the dividing line at the location where specular reflection occurs and the road surface luminance BR2, allowing the edges of the road markings used for detecting road markings to be observed. On the other hand, in areas where specular reflection does not occur, the luminance of the dividing line BL1 and the road surface luminance BR1 themselves are small, so the difference in luminance between them ΔB1 is also small, resulting in a brightness problem where it becomes difficult to observe the edge components of the road markings.

[0034] In the following explanation, the problem of specular reflection of sunlight being captured in the surrounding image (IM) causing areas other than the specular reflection to become dark, making it difficult to observe the edge components of road markings, will be referred to as the "brightness problem due to specular reflection."

[0035] Furthermore, as shown in Figure 3(b), when the sun 6 is to the side and low to the vehicle 1, shadows S of buildings BD1 and BD2 next to the road are cast on the road surface in places. This can lead to a large contrast difference between the brightness values ​​of the sunlit areas and the shaded areas, resulting in a problem where the edge components of the road markings can be detected in the sun but not in the shaded areas (referred to as the "contrast problem due to shadows" in the following explanation).

[0036] Therefore, in the road lane marking detection method of this embodiment, the positional relationship between the sun 6 and the vehicle 1 is estimated based on the luminance characteristics of the ambient image IM. If it is determined that a brightness problem due to specular reflection occurs based on the estimated positional relationship, the ambient image IM is corrected by brightness correction, and the road lane markings are detected from the corrected ambient image IM. For example, the brightness correction of the ambient image IM may include gamma correction using a gamma value of less than 1.

[0037] Figure 4(c) shows an example of the conversion characteristics from the incident light intensity of the image sensor to the brightness value of the ambient image when the ambient image is gamma corrected with a gamma value of less than 1 while specular reflection is captured. Gamma correction increases the brightness gradation of pixels with low incident light intensity, allowing for the observation of a certain degree of brightness difference ΔB1 and ΔB2 in both areas with and without specular reflection. As a result, the influence of specular reflection caused by the position of the sun is reduced, enabling the detection of road markings.

[0038] Furthermore, in the road lane marking detection method of this embodiment, the positional relationship between the sun 6 and the vehicle 1 is estimated based on the luminance characteristics of the surrounding image IM. If it is determined that a contrast problem due to shadows occurs based on the estimated positional relationship, the surrounding image IM is corrected by contrast correction, and the road lane markings are detected from the corrected surrounding image IM.

[0039] For example, contrast correction of the ambient image IM may include logarithmic correction. In this specification, "logarithmic correction" means correcting the luminance value of the ambient image IM by the following formula (1) so that the natural logarithm or common logarithm of the luminance value before correction becomes the luminance value after correction. Corrected brightness value = LOG(uncorrected brightness value) …(1)

[0040] Here, if we denote the luminance of the dividing lines in sunny and shaded areas as "BLs" and "BLd", the luminance of the road surface in sunny and shaded areas as "BRs" and "BRd", the solar illuminance in sunny and shaded areas as "SLs" and "SLd", the reflectance of the road dividing lines as "RL", and the reflectance of the road surface area other than the road dividing lines as "RR", then the luminance of the dividing lines BLs and the road surface luminance BRs in sunny areas, and the luminance of the dividing lines BLd and the road surface luminance BRd in shaded areas can be expressed as shown in the following equations (2) to (5).

[0041] BLs = RL × SLs …(2) BRs = RR × SLs …(3) BLd = RL × SLd …(4) BRd = RR × SLd …(5)

[0042] Therefore, the luminance difference ΔBs between the road surface area other than the road markings and the road markings in sunny areas, and the luminance difference ΔBd between the road surface area other than the road markings and the road markings in shaded areas, can be expressed as shown in equations (6) and (7) below. ΔBs=BLs-BRs=(RL-RR)×SLs…(6) ΔBd=BLd-BRd=(RL-RR)×SLd…(7)

[0043] In other words, the luminance difference ΔBs and ΔBd are expressed as the product of the difference in reflectance and the amount of sunlight; therefore, the luminance difference decreases in sunlight and shade depending on the amount of sunlight. Therefore, when the luminance values ​​of the surrounding image IM are logarithmically corrected, the luminance differences ΔBs and ΔBd can be expressed as shown in equations (8) and (9) below.

[0044] LOG(BLs) = LOG(RL × SLs) LOG(BRs) = LOG(RR × SLs) ΔBs = LOG(BLs) - LOG(BRs) =LOG(RL×SLs)-LOG(RR×SLs) =LOG(RL / RR) …(8)

[0045] LOG(BLd) = LOG(RL × SLd) LOG(BRd) = LOG(RR × SLd) ΔBd = LOG(BLd) - LOG(BRd) =LOG(RL×SLd)-LOG(RR×SLd) =LOG(RL / RR)…(9)

[0046] As can be seen from equations (8) and (9) above, the effect of solar irradiance on the luminance differences ΔBs and ΔBd is eliminated. As a result, even in scenes with a large contrast difference between sunny and shaded areas, the luminance difference between the dividing line luminance and the road surface luminance can be observed without any difference between the two, so road markings can be detected stably.

[0047] The operation and functions of the driving support device 10 of this embodiment will be described in detail below. Figure 5 is a flowchart of an example of a driving support method performed by the controller 16 of the driving support device 10, and Figure 6 is a block diagram of an example of the functional configuration of the controller 16. The controller 16 includes a camera control unit 20, an observation area setting unit 21, a brightness measurement unit 22, a brightness change determination unit 23, a detection area extraction unit 24, an image integration unit 25, a brightness value correction unit 26, a white line detection unit 27, and a vehicle control unit 28.

[0048] In step S1, the camera control unit 20 adjusts the exposure so that the road markings image captured by the camera 11 does not become overexposed (i.e., the brightness of the road markings image does not become saturated). Here, "brightness saturation" means that the gradation of the image data is lost when the ambient image IM reaches a predetermined maximum brightness value that it can take. The predetermined maximum brightness value that the ambient image IM can take is, for example, "255" when brightness is represented by 8-bit data, and may be the minimum brightness value that the ambient image IM can take, which is "0".

[0049] For example, the camera control unit 20 may adjust the exposure amount of the camera 11 so that the maximum brightness of all pixels in the image output from the camera 11 is less than or equal to a predetermined maximum brightness that the surrounding image IM can take. For example, the camera control unit 20 may adjust the exposure amount of the camera 11 by adjusting the exposure time from when the image sensor of the camera 11 starts to finish accumulating charge due to the photoelectric effect. Alternatively, for example, the camera control unit 20 may adjust the exposure amount of the camera 11 by adjusting the aperture of the optical system of the camera 11.

[0050] In step S2, the controller 16 reads the surrounding image IM captured by the camera 11. In this embodiment, the case in which an image taken in front of the vehicle 1 is read as the surrounding image IM, as shown in Figure 2, will be described. The controller 16 stores a predetermined number of surrounding image IMs of frames continuously captured by the camera 11 in the storage device 16b. When a new surrounding image IM is read, the old surrounding image IM is discarded, thereby sequentially updating the predetermined number of surrounding image IMs stored in the storage device 16b.

[0051] In step S3, the observation area setting unit 21 obtains information on the current position and direction of travel of the vehicle 1 from the positioning device 13, and also obtains information on the road structure of the roads surrounding the vehicle 1 from the map DB 14. Based on the current position and direction of travel of the vehicle 1 obtained from the positioning device 13 and the road structure information (e.g., road shape and lane shape) obtained from the map DB 14, the observation area setting unit 21 calculates the direction in which the vehicle 1 should travel. For example, the observation area setting unit 21 may calculate the direction in which the vehicle 1 should travel from its current position along the road shape and lane shape.

[0052] In step S4, the observation area setting unit 21 projects the range of the road surface in the direction the vehicle 1 is traveling onto the surrounding image IM in two dimensions, based on the road shape information in front of the vehicle 1 obtained from the map DB 14. For example, the observation area setting unit 21 projects the range of the road surface of the road RD in the direction the vehicle 1 is traveling onto the surrounding image IM in two dimensions, as shown in the area enclosed by the dashed line on the surrounding image IM in Figure 7(a).

[0053] Refer to Figure 5. In step S5, the observation area setting unit 21 sets a partial image Ro as the observation area, which extends along the direction of travel of the vehicle 1 within the area of ​​the road RD projected in two dimensions on the surrounding image IM. The observation area Ro is an example of the "partial image" described in the claims. For example, the observation area setting unit 21 may set the area Ro occupied by the image of the lane in which the vehicle 1 is traveling (i.e., the road of the vehicle 1) as a band-shaped observation area extending along the direction of travel of the vehicle 1, as shown in the area enclosed by the dashed line on the surrounding image IM in Figure 7(b).

[0054] For example, the observation area setting unit 21 may calculate the observation area Ro by projecting the range of the vehicle's lane onto the surrounding image IM in two dimensions, based on the lane shape information of the vehicle's lane obtained from the map DB 14. The observation area setting unit 21 divides the observation area Ro into predetermined sub-regions Rd1, Rd2, Rd3, Rd4, Rd5, etc. along the direction of travel of the vehicle 1.

[0055] Refer to Figure 5. In step S6, the luminance measurement unit 22 detects the luminance within the divided regions Rd1, Rd2, Rd3, Rd4, Rd5... and calculates the average luminance within each divided region Rd1, Rd2, Rd3, Rd4, Rd5.... This allows for the measurement of the luminance change in the observation region Ro along the direction of travel of the vehicle 1.

[0056] In step S7, the detection region extraction unit 24 calculates the amount of lateral displacement (image displacement) of the observation region Ro that occurs as the vehicle 1 moves, between the surrounding images IM of consecutive frames stored in step S2. Figure 8 schematically shows the surrounding images IM1, IM2, and IM3 of consecutive frames. Since the surrounding images IM1 to IM3 were taken at different times, a lateral shift in the observation area Ro occurs as the vehicle 1 moves. For example, the detection region extraction unit 24 may calculate the amount of displacement of the observation region Ro based on the change in the position of the observation region Ro set in each of the surrounding images IM1 to IM3 of consecutive frames.

[0057] In step S8, the detection region extraction unit 24 extracts partial images R1 to R3, which are used for detecting road division lines, from the surrounding images IM1 to IM3 of the consecutive frames stored in step S2, as detection region images. At this time, the detection region extraction unit 24 adjusts the position from which to cut out the detection region images R1 to R3 from the surrounding images IM1 to IM3, based on the amount of displacement calculated in step S7, so that the position and shape of the road division lines included in the detection region images R1 to R3 are approximately the same.

[0058] Since road markings extend along the road in the direction of travel ahead of vehicle 1, the road markings are captured at approximately the same position in consecutive frames even while vehicle 1 is moving. Therefore, by correcting the cropping position according to the lateral displacement of the lane on the surrounding image IM that occurs as vehicle 1 moves, detection region images R1 to R3 can be extracted so that the position and shape of the road markings contained in detection region images R1 to R3 are approximately the same.

[0059] In step S9, the image integration unit 25 integrates the multiple detection region images R1 to R3 extracted in step S8. The image integration unit 25 integrates the multiple detection region images R1 to R3 by adding the brightness values ​​of the same pixels for each pixel in the multiple detection region images R1 to R3. As described above, the position and shape of the road markings included in the detection region images R1 to R3 are almost identical. Therefore, even when integrating multiple detection region images R1 to R3, blurring of the road markings can be suppressed.

[0060] By integrating multiple images in this way, the luminance gradation can be increased. Therefore, even when gamma correction or logarithmic correction is applied to the integrated image, as described later, an image with sufficient resolution can be secured. Although Figure 8 illustrates an example of integrating three detection region images R1 to R3, the number of images integrated by the image integration unit 25 is not limited to three. The image integration unit 25 may integrate four or more images. For example, the image integration unit 25 may integrate 10 detection region images.

[0061] In step S10, the image integration unit 25 offsets the integrated image so that the minimum brightness value in the detected region image after image integration (hereinafter sometimes referred to as the "integrated image") becomes 0. For example, if the minimum brightness value of the integrated image is Bmin, the minimum value Bmin is subtracted from the brightness of all pixels in the integrated image.

[0062] In step S11, the brightness change determination unit 23 determines the brightness change in the observation area Ro along the direction of travel of the vehicle 1, based on the average brightness within the divided areas Rd1, Rd2, Rd3, Rd4, Rd5… calculated in step S6. If the brightness change in the observation area Ro is less than or equal to a threshold value smaller than a predetermined value (step S11: "less than or equal to the threshold"), the brightness change determination unit 23 determines that the altitude of the sun 6 is relatively high (Figure 3(c)), and the process proceeds to step S12.

[0063] If the average brightness changes gradually (for example, proportionally) along the direction of travel of the vehicle 1, with a gradient such that the brightness change of the average brightness in adjacent divided regions Rd1, Rd2, Rd3, Rd4, Rd5... is less than a predetermined value (step S11: "gradual change"), the brightness change determination unit 23 determines that the sun 6 is in front of the vehicle 1 and at a low position (Figure 3(a)), and the process proceeds to step S13. For example, the brightness change determination unit 23 may determine that the sun 6 is in front of and low to the vehicle 1 if the average brightness changes gradually at a gradient such that the brightness change in adjacent divided regions is less than a predetermined value, and the average brightness has a predetermined maximum value in any of the divided regions along the direction of travel.

[0064] If the change in average brightness in adjacent divided regions is greater than or equal to a predetermined value, and there is a step in the average brightness of adjacent divided regions (step S11: "step"), the brightness change determination unit 23 determines that the sun 6 is to the side and low to the vehicle 1 (Figure 3(b)), and the process proceeds to step S14.

[0065] In step S12, the brightness value correction unit 26 linearly corrects the integrated image by dividing the brightness of each pixel in the integrated image by the number of images integrated by the image integration unit 25, and acquires the corrected integrated image as the corrected image. The process then proceeds to step S15.

[0066] In step S13, the luminance value correction unit 26 generates a corrected image by performing brightness correction on the integrated image. For example, the luminance value correction unit 26 may gamma correct the integrated image with a gamma value γ of less than 1, and gain correct the integrated image after correction so that the maximum possible luminance of the integrated image is converted to a predetermined maximum possible luminance of the ambient image IM, and acquire the corrected integrated image as the corrected image. For example, the brightness value correction unit 26 may perform gamma correction of the integrated image based on the following equation (10).

[0067] BA = (BI / (N × Bmax)) γ ×Bmax …(10) In equation (10) above, Bmax represents the maximum brightness that the surrounding image IM can take (for example, "255"), BI represents the brightness of the integrated image, BA represents the brightness of the corrected image, and N represents the number of images integrated by the image integration unit 25. The process then proceeds to step S15.

[0068] In step S14, the luminance value correction unit 26 generates a corrected image by performing contrast correction on the integrated image. For example, the luminance value correction unit 26 performs logarithmic correction on the integrated image and gain correction so that the maximum value of the luminance after logarithmic correction is converted to a predetermined maximum value of the luminance that the surrounding image IM can take, and acquires the corrected integrated image as the corrected image. The process then proceeds to step S15. In step S15, the white line detection unit 27 extracts edge images by performing a predetermined edge calculation operation on the corrected image generated in step SS12, S13, or S14.

[0069] In step S16, the white line detection unit 27 extracts the linear component of the edge by performing a Hough transform on pixels in the edge image extracted in step S15 that have a value greater than or equal to a predetermined value. In step S17, the white line detection unit 27 detects a pair of parallel lines located in front of the vehicle 1 from among the extracted straight line components as road markings.

[0070] In step S18, when the extracted road lane markings are detected, the vehicle control unit 28 calculates the amount of vehicle speed and steering angle required to drive the vehicle 1 without crossing the detected road lane markings and deviating from the driving lane, and sends these amounts of operation to the actuator 15. The actuator 15 operates the accelerator, brake, and steering wheel connected to it to achieve the specified speed and steering angle, thereby driving the vehicle 1 automatically.

[0071] (modified version) In the above-described embodiment, the driving support device 10 detects the brightness value of the observation area Ro, which extends along the direction of travel of the vehicle 1, from the road surface image IM included in the surrounding image captured by the front camera that captures the area in front of the vehicle body 1 in the longitudinal direction, and determines whether or not to perform brightness correction or contrast correction on the integrated image based on the change in brightness value along the direction of travel of the vehicle 1.

[0072] By scanning the change in brightness value along the direction of travel of the vehicle 1 in this way, it is possible to prevent the detection of changes in brightness value that cross the boundary line between the road markings and the rest of the road surface. Therefore, when determining the position of the sun based on the change in brightness, it is possible to suppress the risk of false detection of a contrast problem caused by shadows, which would occur if changes in brightness value that cross the boundary line between the road markings and the rest of the road surface were detected.

[0073] However, the present invention is not limited to such examples, and the luminance value of a portion of the road surface image included in the surrounding image IM that extends along the vertical direction of the surrounding image IM may be detected, and a determination may be made as to whether or not to perform brightness correction or contrast correction on the integrated image based on the change in luminance value along the vertical direction of the surrounding image IM. Even when using these changes in brightness values ​​for determination, it is possible to determine whether or not brightness problems due to specular reflection or contrast problems due to shadows are occurring, as described above, by referring to Figures 3(a) to 3(d).

[0074] Furthermore, in the above-described embodiment, the driving support device 10 detected the road markings in front of the vehicle 1 based on the surrounding image IM captured by a front camera that photographs the area in front of the vehicle body in the longitudinal direction. However, the present invention is not limited to such examples, and the road markings behind, to the right, and to the left of the vehicle 1 may also be detected based on surrounding images captured by a rear camera that photographs the area behind the vehicle body in the longitudinal direction, or by a right-side camera and a left-side camera that photograph the right and left sides of the vehicle body.

[0075] (Effects of the embodiment) (1) In the road marking detection method of the embodiment, an ambient image is generated by an imaging device installed on the vehicle, an ambient image is generated of the area around the vehicle, the brightness value of a portion of the road surface image included in the ambient image that extends along the direction of travel of the vehicle is detected, if the brightness value changes at a gradient such that the change in brightness value between adjacent positions in the direction of travel of the portion image is less than a predetermined value, the ambient image is corrected by brightness correction, and the road marking is detected from the corrected ambient image, if the change in brightness value between adjacent positions in the direction of travel of the portion image is greater than or equal to a predetermined value, the ambient image is corrected by contrast correction, and the road marking is detected from the corrected ambient image. This makes it possible to suppress the influence of sunlight when detecting road markings from ambient images with a simple configuration.

[0076] (2) In the direction of travel of a partial image, if the brightness value changes with a gradient such that the change in brightness value between adjacent positions is less than a predetermined value, and the brightness value of the partial image has a predetermined maximum value, the surrounding image may be corrected by brightness correction, and road markings may be detected from the corrected surrounding image. This makes it possible to more accurately determine whether or not a brightness problem due to specular reflection is occurring.

[0077] (3) Partial images of the same area of ​​the road surface around the vehicle may be extracted from multiple ambient images taken at different times, and image integration may be performed. The integrated image obtained from image integration may be corrected by brightness correction or contrast correction, and road markings may be detected from the corrected integrated image. This increases the brightness gradation of the image in which road markings are detected, so that an image with sufficient resolution can be secured even after brightness correction or contrast correction.

[0078] (4) Brightness correction may include gamma correction, gain correction, and offset correction. This can eliminate brightness problems caused by specular reflection. (5) The road marking detection method according to claim 1, characterized in that the contrast correction includes logarithmic correction. This eliminates the contrast problem caused by shadows. (6) If the change in brightness value in the road surface image included in the surrounding image is below a threshold value smaller than a predetermined value, brightness correction and contrast correction do not need to be performed. In this case, since there are no brightness problems due to specular reflection or contrast problems due to shadows, road markings can be detected well from the surrounding image without performing these corrections. [Explanation of symbols]

[0079] 1...Vehicle, 6...Sun, 10...Driving support system, 11...Camera, 12...Vehicle sensor, 13...Positioning device, 14...Map database (Map DB), 15...Actuator, 16...Controller, 16a...Processor, 16b...Storage device, 20...Camera control unit, 21...Observation area setting unit, 22...Brightness measurement unit, 23...Brightness change determination unit, 24...Detection area extraction unit, 25...Image integration unit, 26...Brightness value correction unit, 27...White line detection unit, 28...Vehicle control unit,

Claims

1. An imaging device installed on the vehicle generates an ambient image of the area surrounding the vehicle, The brightness value of a portion of the road surface image included in the surrounding image that extends along the direction of travel of the vehicle is detected. In the direction of travel of the aforementioned partial image, if the brightness value changes at a gradient such that the change in brightness value between adjacent positions is less than a predetermined value, the surrounding image is corrected by brightness correction, and road markings are detected from the corrected surrounding image. If, in the direction of travel of the aforementioned partial image, the change in brightness value between adjacent positions is greater than or equal to a predetermined value, the surrounding image corrected by contrast correction is corrected, and road markings are detected from the corrected surrounding image. A method for detecting road lane markings, characterized by the features described herein.

2. The road marking detection method according to claim 1, characterized in that, in the direction of travel of the partial image, the brightness value changes with a gradient such that the change in brightness value between adjacent positions is less than a predetermined value, and the brightness value of the partial image has a predetermined maximum value, the surrounding image is corrected by the brightness correction, and road markings are detected from the corrected surrounding image.

3. From multiple surrounding images taken at different times, partial images capturing the same area of ​​the road surface around the vehicle are extracted, and image integration is performed. The integrated image obtained by the aforementioned image integration is corrected by the brightness correction or the contrast correction, and road markings are detected from the corrected integrated image. The road lane marking detection method according to feature 1.

4. The road marking detection method according to claim 1, characterized in that the brightness correction includes gamma correction.

5. The road marking detection method according to claim 1, characterized in that the brightness correction includes gain correction and offset correction.

6. The road marking detection method according to claim 1, characterized in that the contrast correction includes logarithmic correction.

7. The road marking detection method according to claim 1, characterized in that if the change in the brightness value in the road surface image included in the surrounding image is less than or equal to a threshold smaller than the predetermined value, the brightness correction and contrast correction are not performed.

8. An imaging device installed on the vehicle to photograph the area around the vehicle, A controller that performs the following processes: acquiring an ambient image which is an image of the area around the vehicle captured by the imaging device; detecting the brightness value of a portion of the road surface image included in the ambient image that extends along the direction of travel of the vehicle; correcting the ambient image by brightness correction and detecting road markings from the corrected ambient image when the brightness value changes at a gradient such that the change in brightness value between adjacent positions in the direction of travel of the portion image is less than a predetermined value; and correcting the ambient image corrected by contrast correction and detecting road markings from the corrected ambient image when the change in brightness value between adjacent positions in the direction of travel of the portion image is greater than or equal to a predetermined value. A road lane marking detection device characterized by comprising the following features.

Images