Vehicle lane recognition apparatus

Abstract

To provide a vehicle lane recognition apparatus that can recognize a lane by using a small-size estimation model without being affected by a disturbance factor.SOLUTION: An image recognition ECU 13 is configured to: compare a traveling environment image with a comparison image obtained by imaging the same traveling environment as the traveling environment image in estimating a lane based on the traveling environment image using an estimation model 50 constructed by machine learning; determine, when a brightness average value in the traveling environment image is lower than a brightness average value in the comparison image and a difference between the brightness average value in the traveling environment image and the brightness average value in the comparison image is equal to or larger than a threshold, that a disturbance factor that blocks a field of view is included in the traveling environment image; and replace information on the traveling environment image determined to include the disturbance factor with information on the traveling environment image that is different by one or several frames.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a lane recognition device for a vehicle that recognizes the driving lane by using an image of the driving environment outside the vehicle. [Background technology]

[0002] Driving assistance devices installed in vehicles such as automobiles achieve driving assistance control by appropriately combining, for example, adaptive cruise control (ACC), active lane keep centering control (ALKC), and emergency lane departure prevention control (ELKA: Emergency Lane Keep Assist), etc.

[0003] In such driving assistance control, it is important to recognize lanes such as the lane in which the vehicle is traveling and adjacent lanes in real time. In recent years, a technology has been proposed that uses an estimation model such as a machine-learned neural network to recognize lanes from an image of the driving environment in front of the vehicle. For example, Patent Document 1 discloses a technology that detects lane boundaries while maintaining robustness against noise (disturbance) in each frame caused by windshield wipers, etc., by inputting a series of multiple frames of images into a neural network. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2023-503536 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when countermeasures against disturbance elements such as wipers are machine-learned into an estimation model as in the technology disclosed in the above-mentioned Patent Document 1, there is a risk that the model size of the estimation model will become large.

[0006] An object of the present invention is to provide a lane recognition device for a vehicle that can recognize lanes using an estimation model with a small model size without being affected by external disturbance factors. [Means for solving the problem]

[0007] A lane recognition device for a vehicle according to one embodiment of the present invention comprises an imaging means for imaging the driving environment outside the vehicle, a disturbance determination means for comparing a driving environment image obtained by imaging the driving environment with a comparison image obtained by imaging the driving environment equivalent to the driving environment image, and determining that an external disturbance element that obstructs visibility is captured in the driving environment image when the average brightness value of the driving environment image is lower than the average brightness value of the comparison image and the difference in average brightness between the driving environment image and the comparison image is equal to or greater than a threshold value, and a lane estimation means for estimating a lane based on the driving environment image using an estimation model constructed by machine learning, wherein when estimating the lane, the lane estimation means replaces information relating to the driving environment image determined to include the disturbance element with information relating to the driving environment image that differs by one or several frames.

[0008] In addition, a lane recognition device for a vehicle according to another aspect of the present invention includes an imaging means for imaging the driving environment outside the vehicle, and a processor. The processor compares a driving environment image obtained by imaging the driving environment with a comparison image obtained by imaging the driving environment equivalent to the driving environment image, and when the average brightness value of the driving environment image is lower than the average brightness value of the comparison image and the difference in average brightness between the driving environment image and the comparison image is equal to or greater than a threshold value, determines that an external disturbance element that obstructs visibility is captured in the driving environment image, estimates a lane based on the driving environment image using an estimation model constructed by machine learning, and when estimating the lane, replaces information about the driving environment image determined to include the disturbance element with information about the driving environment image that differs by one or several frames. Effect of the Invention

[0009] According to the vehicle lane recognition device of the present invention, it is possible to recognize lanes using an estimation model with a small model size without being affected by external disturbance factors. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a schematic diagram showing a vehicle driving assistance device; [Diagram 2] Functional configuration diagram of the estimation model [Diagram 3] FIG. 1 is an explanatory diagram illustrating a convolution process for a driving environment image; [Figure 4] FIG. 1 is an explanatory diagram illustrating a convolution process for a driving environment image; [Diagram 5] FIG. 13 is an explanatory diagram illustrating an example of an area where average luminance values ​​are compared in a disturbance determination; [Figure 6] FIG. 13 is an explanatory diagram illustrating an example of an area where average luminance values ​​are compared in a disturbance determination; [Figure 7] Flowchart showing lane recognition routine [Figure 8] FIG. 2 is an explanatory diagram showing a driving state of a host vehicle relative to a host vehicle driving lane; [Figure 9] FIG. 11 is an explanatory diagram illustrating an example of an area where average luminance values ​​are compared in a disturbance determination according to a modified example. [Figure 10] FIG. 11 is an explanatory diagram illustrating an example of an area where average luminance values ​​are compared in a disturbance determination according to a modified example. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings relate to one embodiment of the present invention, and Fig. 1 is a schematic diagram showing a vehicle driving support device.

[0012] 1, the driving assistance device 1 is configured to include a camera unit 10. The camera unit 10 is fixed to the center of the upper front part of the interior of a vehicle (host vehicle) O, for example.

[0013] The camera unit 10 includes a stereo camera 11 as an imaging means, an image processing unit (IPU) 12, an image recognition unit (image recognition_ECU) 13, and a driving control unit (driving_ECU) 14.

[0014] The stereo camera 11 has a main camera 11a as a first camera and a sub camera 11b as a second camera as sensors. The main camera 11a and the sub camera 11b are configured with imaging elements such as CMOS. The main camera 11a and the sub camera 11b are arranged, for example, at symmetrical positions on either side of the center in the vehicle width direction. As a result, the main camera 11a and the sub camera 11b stereoscopically capture images of the driving environment in the area in front of the vehicle from different viewpoints at imaging periods (set timings) synchronized with each other.

[0015] The IPU 12 performs predetermined image processing on the driving environment images captured by the stereo camera 11. As a result, the IPU 12 detects edges of various objects such as three-dimensional objects and lane markings on the road surface displayed on the images. The IPU 12 also obtains distance information from the positional deviation amount of corresponding edges on the left and right images. As a result, the IPU 12 generates image information including distance information (distance image information).

[0016] All or a part of the image recognition_ECU 13 is configured by a processor including hardware. The processor is configured by a known configuration including, for example, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a non-volatile memory, a non-volatile storage, a non-transitory computer readable medium, and the like, and peripheral devices thereof. Furthermore, the processor constituting the image recognition_ECU 13 of this embodiment includes a data processing unit (DPU). The ROM, the non-volatile memory, the non-volatile storage, and the like store software programs executed by the CPU, etc., fixed data such as data tables, and the like. The CPU, etc. reads out the software programs stored in the ROM, etc., expands them in the RAM, and executes them, and the software programs refer to various data, etc. as appropriate, thereby realizing the functions of the above-mentioned components and units, etc.

[0017] The image recognition_ECU 13 estimates a lane based on a driving environment image by using an estimation model 50 such as a neural network constructed by machine learning. Here, as the driving environment image, for example, an image captured by the main camera 11a of the stereo camera 11 is used.

[0018] In this embodiment, the lane estimation process using the estimation model 50 is mainly executed by the DPU provided in the image recognition_ECU 13. As a result, the image recognition_ECU 13 recognizes lanes on a road including the lane on which the host vehicle O is traveling (host vehicle traveling lane). The lane estimation process using the estimation model 50 will be described in detail later.

[0019] Furthermore, the image recognition_ECU 13 performs predetermined pattern matching on the distance image information. As a result, the image recognition_ECU 13 recognizes three-dimensional objects such as guardrails and curbs extending along the road, and surrounding vehicles traveling on the road. Here, in the recognition of three-dimensional objects by the image recognition_ECU 13, for example, the type of the three-dimensional object, the distance to the three-dimensional object, the speed of the three-dimensional object, and the relative speed between the three-dimensional object and the vehicle O are recognized. In this embodiment, the recognition process of the three-dimensional object using pattern matching is mainly executed by a CPU provided in the image recognition_ECU 13.

[0020] Various pieces of information recognized by the image recognition_ECU 13 are output to the driving_ECU 14 as driving environment information.

[0021] The driving_ECU 14 is a control unit for overall control of the driving assistance device 1.

[0022] Various control units, such as an engine control unit (E / G_ECU) 22, a transmission control unit (T / M_ECU) 23, a brake control unit (BK_ECU) 24, and a power steering control unit (PS_ECU) 25, are connected to the driving_ECU 14 via an in-vehicle communication line such as a CAN (Controller Area Network).

[0023] In this embodiment, all or part of the traveling_ECU 14, the E / G_ECU 22, the T / M_ECU 23, the BK_ECU 24, the PS_ECU 25, etc. are configured by a processor including hardware. Here, the processor is configured by a well-known configuration including, for example, a central processing unit (CPU), a random access memory (RAM), a read only memory (ROM), a non-volatile memory, a non-volatile storage, etc., as well as a non-transitory computer readable medium, etc., and peripheral devices thereof. The ROM, the non-volatile memory, the non-volatile storage, etc. store in advance software programs executed by the CPU, etc., fixed data such as data tables, etc. Then, the CPU reads out the software programs stored in the ROM, etc., expands them in the RAM, and executes them, and the software programs appropriately refer to various data, etc., thereby realizing the functions of the above-mentioned components and components.

[0024] A throttle actuator 32 for an electronically controlled throttle and the like are connected to an output side of the E / G_ECU 22. In addition, various sensors such as an accelerator sensor (not shown) are connected to an input side of the E / G_ECU 22.

[0025] The E / G_ECU 22 controls the throttle actuator 32 etc. based on control signals from the travel_ECU 14 or detection signals from various sensors. In this way, the E / G_ECU 22 adjusts the amount of intake air for the engine to generate a desired engine output. The E / G_ECU 22 also outputs signals such as the accelerator opening degree detected by the various sensors to the travel_ECU 14.

[0026] An output side of the T / M_ECU 23 is connected to a hydraulic control circuit 33. In addition, an input side of the T / M_ECU 23 is connected to various sensors such as a shift position sensor (not shown).

[0027] The T / M_ECU 23 controls the hydraulic control circuit 33 and the like based on the engine torque signal estimated by the E / G_ECU 22 and detection signals from various sensors. As a result, the T / M_ECU 23 operates friction engagement elements, pulleys, and the like provided in the automatic transmission to shift the engine output at a desired gear ratio. The T / M_ECU 23 also outputs signals such as the shift position detected by the various sensors to the travel_ECU 14.

[0028] A brake actuator 34 for adjusting the brake fluid pressure output to the brake wheel cylinders provided on the respective wheels is connected to the output side of the BK_ECU 24. In addition, various sensors such as a vehicle speed sensor, a yaw rate sensor, a brake pedal sensor, and a longitudinal acceleration sensor are connected to the input side of the BK_ECU 24.

[0029] The BK_ECU 24 controls the brake actuator 34 and the like based on control signals from the travel_ECU 14 or detection signals from various sensors. As a result, the BK_ECU 24 appropriately generates braking force on each wheel for performing forced braking control, yaw rate control, etc. on the host vehicle O. In addition, the BK_ECU 24 outputs signals of the brake operation state, yaw rate, longitudinal acceleration, host vehicle speed, etc. detected by the various sensors to the travel_ECU 14.

[0030] An electric power steering motor 35 that applies a steering torque to the steering mechanism by the rotational force of the motor is connected to the output side of the PS_ECU 25. In addition, various sensors such as a steering torque sensor and a steering angle sensor are connected to the input side of the PS_ECU 25.

[0031] The PS_ECU 25 controls the electric power steering motor 35 and the like based on control signals from the travel_ECU 14 or detection signals from various sensors. In this way, the PS_ECU 25 generates a steering torque for the steering mechanism. The PS_ECU 25 also outputs signals of the steering torque, steering angle, and the like detected by the various sensors to the travel_ECU 14.

[0032] The traveling_ECU 14 performs driving assistance control by, for example, outputting various control signals to the E / G_ECU 22, the T / M_ECU 23, the BK_ECU 24, and the PS_ECU 25.

[0033] This driving assistance control is realized by appropriately combining adaptive cruise control (ACC), active lane keep centering control (ALKC), emergency lane keep assist control (ELKA), and auto lane changing control (ALC), etc. For example, the driving_ECU 14 realizes driving assistance control according to the driving assistance mode selected by the driver by appropriately combining each control.

[0034] The following vehicle distance control is realized by selectively executing following running control and constant speed running control. In this following vehicle distance control, the traveling_ECU 14 performs a registration process of the preceding vehicle based on the traveling environment information. That is, when a preceding vehicle is registered ahead of the host vehicle O, the traveling_ECU 14 performs following running control. In this following running control, the traveling_ECU 14 sets a target vehicle distance according to the vehicle speed of the preceding vehicle, and performs acceleration / deceleration control to maintain the set target vehicle distance. On the other hand, when the preceding vehicle L is not registered ahead of the host vehicle O, the traveling_ECU 14 performs constant speed running control. In this constant speed running control, the traveling_ECU 14 performs acceleration / deceleration control for the host vehicle O with the set vehicle speed Vset input by the driver as the target vehicle speed.

[0035] The lane center keeping control and lane departure suppression control are performed based on lane marking information and the like included in the driving environment information. That is, the driving_ECU 14 sets a target travel path along the left and right lane markings, for example, in the center of the host vehicle's lane. Then, the driving_ECU 14 performs feedforward control and feedback control for steering based on the target travel path. For example, as shown in FIG. 8, if the road curvature of the host vehicle's lane is "a", the yaw angle of the host vehicle O with respect to the host vehicle's lane is "b", and the lateral position of the host vehicle O in the host vehicle's lane is "c", the target steering angle θ for the lane center keeping control is calculated by the following formula (1).

[0036] θ=G p ×c+G i ×∫cdt+G d ×b+G ff ×a …(1) Here, in equation (1), G p ,G i ,G d is the gain in feedback control (PID control), and G ff is a gain in the feedforward control. In this way, the driving_ECU 14 keeps the host vehicle O in the center of the lane.

[0037] The automated lane change control is performed based on lane marking information and the like included in the driving environment information. That is, the driving_ECU 14 sets a target lateral position in a lane adjacent to the lane in which the host vehicle is traveling. Furthermore, the driving_ECU 14 sets a target route from the target route of the host vehicle O to the target lateral position. Then, the driving_ECU 14 performs feedforward control and feedback control of steering along the target route. As a result, the driving_ECU 14 changes the lane to the adjacent lane.

[0038] Next, the lane recognition process performed by the image recognition_ECU 13 will be described in detail.

[0039] 2, when the estimation model 50 is functionally shown, the estimation model 50 has a convolutional layer 51, a storage unit 52, and a fully connected layer 53. Furthermore, the estimation model 50 is provided with a disturbance determination unit 54.

[0040] The disturbance determination unit 54 receives, for example, a driving environment image captured by the main camera 11a of the stereo camera 11. Furthermore, the disturbance determination unit 54 receives, for example, a driving environment image captured by the sub-camera 11b as a comparison image.

[0041] Here, the main camera 11a and the sub camera 11b capture stereo images of the driving environment from different viewpoints at mutually synchronized imaging timings. Therefore, the comparative image corresponds to an image of a driving environment equivalent to the driving environment image captured by the main camera 11a.

[0042] The disturbance determination unit 54 compares the average luminance value of the driving environment image with the average luminance value of the comparison image. When the average luminance value of the driving environment image is lower than the average luminance value of the comparison image and the difference in the average luminance value between the driving environment image and the comparison image is equal to or greater than a threshold value, the disturbance determination unit 54 determines that a disturbance element obstructing visibility is captured in the driving environment image.

[0043] In this case, it is preferable that the disturbance determination unit 54 compares the average brightness values ​​by limiting the comparison to areas set in correspondence with the lanes on the driving environment image and the comparative image, respectively (see, for example, FIGS. 5 and 6). For example, the disturbance determination unit 54 can set areas in the driving environment image and the comparative image, respectively, that correspond to the lanes previously estimated by the estimation model 50, as areas for comparing the average brightness values.

[0044] Possible disturbance elements include, for example, windshield wipers, raindrops during rainfall, snow during snowfall, and insects that temporarily fly into the vicinity of the windshield.

[0045] The convolution layer 51 performs a convolution process on the driving environment image. For example, the driving environment image captured by the main camera 11a is input to the convolution layer 51. For example, the convolution layer 51 extracts a pixel of interest from the input driving environment image, and generates an image (feature map: see, for example, FIGS. 3 and 4) that captures position information of the pixel of interest and the relationship between the pixel of interest and surrounding pixels as features. In this embodiment, the convolution layer 51 extracts, for example, a pixel indicating a lane marking or the like as the pixel of interest. Then, the convolution layer 51 stores the generated feature map in the storage unit 52.

[0046] Here, Fig. 3 illustrates an example of a feature map generated from a driving environment image in which no disturbance elements are captured. Fig. 4 illustrates an example of a feature map generated from a driving environment image in which, for example, a disturbance element is captured. In Fig. 4, for example, a wiper is captured as the disturbance element.

[0047] The storage unit 52 stores, in chronological order, information on driving environment images captured at set timings in a travel section of a past set range until the vehicle O reaches its current position. In this embodiment, the storage unit 52 stores, in chronological order, feature maps calculated for each imaging period of the stereo camera 11 (main camera 11a) in a travel section 20 [m] behind the vehicle O, for example.

[0048] In this case, the storage unit 52 stores the determination results of the disturbance determination unit 54 on the driving environment image in chronological order in association with the feature map.

[0049] Thus, in this embodiment, the storage unit 52 corresponds to a specific example of a storage means.

[0050] The fully connected layer 53 estimates the lane by combining, for example, a feature map based on the current driving environment image and a plurality of past feature maps extracted from the memory unit 52.

[0051] For this reason, the fully connected layer 53 extracts, as input information, feature maps for each point in time when the host vehicle O was traveling 5 [m] behind, 10 [m] behind, 15 [m] behind, and 20 [m] behind the current host vehicle position from among a plurality of feature maps stored in chronological order in the memory unit 52, for example.

[0052] In extracting this feature map, if a feature map based on a driving environment image including a disturbance element is extracted, the fully connected layer 53 replaces the feature map including the disturbance element with a feature map from one or several frames before (or one or several frames after).

[0053] Furthermore, if the current driving environment image contains a disturbance element, the fully connected layer 53 replaces the current feature map with a feature map from one or several frames ago.

[0054] As a result, the fully connected layer 53 estimates the lane using the time series of the feature map with the disturbance elements removed.

[0055] Thus, in this embodiment, the convolution layer 51, the storage unit 52, and the fully connected layer 53 correspond to a specific example of a lane estimation means.

[0056] Next, lane recognition performed by the image recognition_ECU 13 will be described with reference to a flowchart of a lane recognition routine shown in Fig. 7. This routine is repeatedly executed by the image recognition_ECU 13 at set time intervals (for example, at each image capture period of the stereo camera 11).

[0057] 7, the processes of steps S101 to S104 are mainly performed by a CPU provided in the image recognition_ECU 13. The processes of steps S105 to S110 are mainly performed by a DPU provided in the image recognition_ECU 13.

[0058] When the routine starts, in step S101, the image recognition_ECU 13 reads an image captured by the stereo camera 11. That is, the image recognition_ECU 13 reads an image captured by the main camera 11a as a driving environment image. The image recognition_ECU 13 also reads an image captured by the sub camera 11b as a comparison image.

[0059] In the next step S102, the image recognition_ECU 13 reads the estimation result of the lane previously estimated using the estimation model 50.

[0060] In the next step S103, the image recognition_ECU 13 uses the previous lane estimation result to set an average brightness calculation area for the driving environment image and the comparative image (see, for example, FIGS. 5 and 6).

[0061] In the next step S104, the image recognition_ECU 13 calculates the average luminance value in the area set in the driving environment image and the average luminance value in the area set in the comparison image, and then compares the average luminance values ​​in the driving environment image and the comparison image to determine whether the driving environment image includes a disturbance factor.

[0062] In the next step S105, the image recognition_ECU 13 performs a convolution process on the driving environment image currently read from the main camera 11a using the convolution layer 51. That is, the image recognition_ECU 13 inputs the driving environment image to the convolution layer 51 of the estimation model 50 to generate a feature map of the driving environment image.

[0063] In the next step S106, the image recognition_ECU 13 updates the time series of the feature map stored in the storage unit 52. That is, the image recognition_ECU 13 stores the feature map based on the currently read driving environment image together with the reflection determination result of the disturbance factor in the driving environment image in the storage unit 52. Furthermore, the image recognition_ECU 13 deletes from the storage unit 52 the feature map from before the time when the host vehicle O was traveling 20 [m] behind the current host vehicle position.

[0064] In the following step S107, the image recognition_ECU 13 extracts, from the multiple feature maps stored in chronological order in the memory unit 52, feature maps for each point in time when the vehicle O was traveling 5 [m] behind, 10 [m] behind, 15 [m] behind, and 20 [m] behind the current vehicle position.

[0065] In the following step S108, the image recognition_ECU 14 checks whether or not a disturbance element is included in the current feature map and the extracted feature map.

[0066] Then, if it is determined in step S108 that no disturbance element is included in any of the feature maps, the image recognition_ECU 13 proceeds to step S110.

[0067] On the other hand, if it is determined in step S108 that at least one of the feature maps includes a disturbance element, the image recognition_ECU 13 proceeds to step S109.

[0068] In step S109, the image recognition_ECU 13 replaces the feature map including the disturbance element with a feature map from one or several frames before (or one or several frames after).

[0069] When the process proceeds to step S108 or step S109 to step S110, the image recognition_ECU 13 performs lane estimation using the fully connected layer 53, and then exits the routine. That is, the image recognition_ECU 13 performs lane estimation by inputting the current feature map and the feature map extracted from the storage unit 52 (the replaced feature map in the case where a disturbance element is included) to the fully connected layer 53.

[0070] According to this embodiment, when estimating a lane based on a driving environment image using the estimation model 50 constructed by machine learning, the image recognition_ECU 13 compares the driving environment image with a comparison image captured of a driving environment equivalent to the driving environment image, and determines that a disturbance element obstructing visibility is captured in the driving environment image when the average brightness value of the driving environment image is lower than the average brightness value of the comparison image and the difference in average brightness value between the driving environment image and the comparison image is equal to or greater than a threshold value. Then, the image recognition_ECU 13 replaces information about the driving environment image determined to include a disturbance element with information about a driving environment image that differs by one or several frames. This allows the lane to be recognized using the estimation model 50 with a small model size without being affected by disturbance elements.

[0071] That is, the image recognition_ECU 13 performs a process for dealing with a case where a disturbance element is included in the driving environment image, which is different from the process using the estimation model 50. Therefore, there is no need to make the estimation model 50 learn a measure for a disturbance element such as a wiper, and it is possible to realize accurate lane estimation using a small estimation model 50.

[0072] Here, it is difficult to assume that disturbance elements such as wipers are continuously captured across multiple frames of the driving environment image. Also, information on stationary objects such as lanes on the driving environment image varies only slightly between one or several frames. By focusing on these points, the image recognition_ECU 13 can realize good lane estimation by a simple measure of simply replacing information on a driving environment image that is determined to include a disturbance element with information on a driving environment image that differs by one or several frames.

[0073] In addition, the brightness of an area on the driving environment image where a disturbance element such as a wiper is reflected is generally much lower than the brightness of other areas. With this in mind, the determination of the reflection of the disturbance element is performed by comparing the average brightness values ​​of the driving environment image and the comparative image. This makes it possible to determine the reflection of the disturbance element by simple calculation processing.

[0074] Furthermore, the image recognition_ECU 13 stores in chronological order the feature map generated by the convolution process on the driving environment image in the storage unit 52. Then, lane estimation is performed by performing full-combination processing on the current feature map and the feature map extracted from the storage unit 52. This allows for accurate estimation of lanes in a wide range of areas in front of and behind the vehicle O. In this case, by storing the feature map in which the driving environment image is compressed by the convolution process, information on the driving environment image can be held in a smaller capacity than when the driving environment image itself is stored.

[0075] The image recognition_ECU 13 also sets the regions of the driving environment image and the comparison image that correspond to the previously estimated lane as regions for calculating the average brightness value. The image recognition_ECU 13 then compares the average brightness values ​​of these regions to determine whether or not a disturbance element is reflected in the driving environment image. As a result, even if a driving environment image contains disturbance elements, the driving environment image that contains disturbance elements in areas other than those overlapping with the lane can be effectively used for lane estimation.

[0076] Moreover, the image recognition_ECU 13 uses an image captured by one of the stereo cameras 11 (for example, the main camera 11a) as a driving environment image for lane estimation, and an image captured by the other camera (for example, the sub-camera 11b) as a comparison image. Here, it is difficult to imagine that disturbance elements such as wipers are captured by the main camera 11a and the sub-camera 11b at the same time. By focusing on this point, it is possible to easily obtain a driving environment image and a comparison image that capture the same driving environment at the same time.

[0077] In the above embodiment, a configuration has been described in which a driving environment image and a comparative image are simultaneously acquired by stereoscopically capturing the driving environment, but the acquisition of the comparative image is not limited to this. For example, as shown in Figs. 9 and 10, it is also possible to use a driving environment image from one or several frames back as a comparative image. Such use of a driving environment image from one or several frames back as a comparative image is effective in both cases where the driving environment is captured by a stereo camera and where the driving environment is captured by a monocular camera.

[0078] Here, the processors constituting the image recognition_ECU 13, the driving_ECU 14, the E / G_ECU 22, the T / M_ECU 23, the BK_ECU 24, the PS_ECU 25, etc. may be constituted by semiconductor chips such as an FPGA (Field Programmable Gate Array), etc. Furthermore, each of the above components and units may be constituted by an electronic circuit.

[0079] Furthermore, the software program may be in a form in which the whole or part is recorded as a computer program product on a portable disk medium such as a flexible disk, CD-ROM, or DVD-ROM, or on a non-transitory computer readable medium such as a card-type memory, an HDD (Hard Disk Drive) device, or an SSD (Solid State Drive) device.

[0080] The invention described in the above embodiments is not limited to those embodiments, and various modifications can be made in the implementation stage without departing from the gist of the invention. Furthermore, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriate combinations of the disclosed constituent elements.

[0081] For example, if the problem described can be solved and the effect described can be obtained even if some of the constituent elements are deleted from all the constituent elements shown in the above form, the configuration from which these constituent elements are deleted can be extracted as an invention. [Explanation of symbols]

[0082] 1. Driving assistance devices 10 … Camera unit 11 … Stereo camera 11a ... Main camera 11b … Sub camera 13...Image Recognition_ECU 14 … Driving_ECU 22 … E / G_ECU 23 … T / M_ECU 24 … BK_ECU 25 … PS_ECU 32 … Throttle actuator 33 ... Hydraulic control circuit 34 ... Brake actuator 35 ... Electric power steering motor 50 … Estimation model 51 ... Convolutional layer 52 … Storage section 53 … Fully connected layer 54 ... Disturbance judgment section

Claims

1. An imaging means for imaging the driving environment outside the vehicle; a disturbance determination means for comparing a driving environment image obtained by capturing an image of the driving environment with a comparative image obtained by capturing an image of the driving environment equivalent to the driving environment image, and determining that a disturbance element obstructing visibility is captured in the driving environment image when an average value of the luminance of the driving environment image is lower than an average value of the luminance of the comparative image and a difference in average value of the luminance of the driving environment image and the comparative image is equal to or greater than a threshold value; A lane estimation means for estimating a lane based on the driving environment image by using an estimation model constructed by machine learning, The lane estimation means is characterized in that, when estimating the lane, it replaces information about the driving environment image in which it is determined that the disturbance element is captured with information about the driving environment image that differs by one or several frames.

2. the lane estimation means includes a storage means for storing, in chronological order, information about the driving environment images captured at each set timing in a travel section of a past set range until the host vehicle reaches its current vehicle position; 2. The vehicle lane recognition device according to claim 1, wherein the lane estimation means stores a feature map generated by convolution processing of the driving environment image in the storage means, and further estimates the lane using the current feature map and the feature map extracted from the storage means.

3. 2. The lane recognition device for a vehicle according to claim 1, wherein the disturbance determination means compares the average luminance values ​​in areas set in the driving environment image and the comparison image in correspondence with the lane previously estimated by the lane estimation means.

4. The imaging means is a stereo camera that uses a first camera and a second camera to stereoscopically image the traveling environment from different viewpoints, The driving environment image is an image of the driving environment captured by the first camera, 2. The lane recognition device for a vehicle according to claim 1, wherein the comparison image is an image of the driving environment captured by the second camera in synchronization with the first camera.

5. 2. The lane recognition device for a vehicle according to claim 1, wherein the comparison image is an image of the driving environment captured one or several frames before the driving environment image in which the inclusion of the disturbance element is determined.

6. An imaging means for imaging the driving environment outside the vehicle; a processor; The processor, a driving environment image obtained by capturing the driving environment is compared with a comparison image obtained by capturing the driving environment equivalent to the driving environment image, and when an average value of the luminance of the driving environment image is lower than an average value of the luminance of the comparison image and a difference in the average values ​​of the luminance of the driving environment image and the comparison image is equal to or greater than a threshold value, it is determined that a disturbance element obstructing visibility is captured in the driving environment image; Estimating a lane based on the driving environment image using an estimation model constructed by machine learning; A lane recognition device for a vehicle, characterized in that, when estimating the lane, information regarding the driving environment image in which it is determined that the disturbance element is captured is replaced with information regarding the driving environment image that differs by one or several frames.

Images