Vehicle control device, vehicle control method, and computer program for vehicle control

Abstract

To provide a vehicle control device that suppresses switching between on and off of a control mode for controlling travel of a vehicle based on a lane boundary line.SOLUTION: The vehicle control device includes: a calculation part 21 that calculates a confidence level at which the lane boundary line is represented in an image generated by a camera 2 mounted on a vehicle 1; an application determination part 22 that enables application of a control mode for controlling travel of the vehicle 1 based on the lane boundary line when the confidence level is equal to or more than a predetermined detection threshold, and disables the application of the control mode when the confidence level is less than the detection threshold; a counting part 24 that counts a number of times during which an applicable period from when the application of the control mode becomes enabled to when the application becomes disabled is equal to or less than a predetermined time threshold; and a threshold adjustment part 25 that increases the detection threshold by a predetermined adjustment amount when the number of times becomes equal to or more than a predetermined number.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a computer program for vehicle control. [Background technology]

[0002] In order to reduce the burden on the driver, it has been proposed to control the vehicle so that even when switching from one section where automatic driving is recommended to the other where manual driving is recommended, the switching is not performed based on at least one of the driving distance and driving time of the section after the switch (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-95132 Summary of the Invention [Problem to be solved by the invention]

[0004] When a driving assistance mode, such as an Advanced Driver-Assistance System (ADAS), is applied to assist the driver in steering the vehicle, the vehicle control device may estimate the vehicle's position relative to lane markings and control the vehicle's driving based on the estimation result. Therefore, the driving assistance mode can be applied (turned on) only when lane markings are detected from an image acquired by an onboard camera. However, when lane markings are visible in places, such as when snow accumulates on parts of the road, or when patterns similar to lane markings are formed on the image, the lane markings may be detected and then not detected repeatedly within a relatively short period of time. As a result, the driving assistance mode may be frequently switched on and off, which may be annoying to the driver.

[0005] Therefore, an object of the present invention is to provide a vehicle control device that can prevent a control mode that controls vehicle travel based on lane markings from being switched on and off in a short period of time. [Means for solving the problem]

[0006] A vehicle control device provided in one embodiment has a calculation unit that calculates the reliability of lane markings being represented in an image generated by a camera mounted on the vehicle; an applicability determination unit that enables the application of a control mode that controls the vehicle's travel based on the lane markings when the reliability is equal to or greater than a predetermined detection threshold and disables the application of the control mode when the reliability is less than the detection threshold; a counting unit that counts the number of times the applicable period from when the control mode becomes applicable to when the control mode becomes unapplicable is equal to or less than a predetermined time threshold; and a threshold adjustment unit that increases the detection threshold by a predetermined adjustment amount when the number of times exceeds a predetermined number.

[0007] A vehicle control device provided in another embodiment has a calculation unit that calculates the reliability of lane markings being represented in an image generated by a camera mounted on the vehicle, and an applicability determination unit that enables the application of a control mode that controls vehicle travel based on lane markings when the reliability is equal to or greater than a predetermined detection threshold and the image is classified by a classifier related to the visibility of lane markings into a first class corresponding to a situation in which lane markings are visible, and that disables the application of the control mode when the reliability is less than the detection threshold or the classifier classifies the image into a second class corresponding to a situation in which lane markings are not visible, and a learning unit that trains the classifier based on images obtained within an applicable period from when the application of the control mode becomes possible to when the application of the control mode becomes impossible, when the applicable period is equal to or less than a predetermined time threshold.

[0008] A vehicle control method provided in yet another embodiment includes calculating the reliability of lane markings being represented in an image generated by a camera mounted on the vehicle, enabling the application of a control mode that controls vehicle travel based on the lane markings when the reliability is equal to or greater than a predetermined detection threshold, disabling the application of the control mode when the reliability is less than the detection threshold, counting the number of times the applicable period from when the control mode becomes applicable to when the control mode becomes unapplicable is equal to or less than a predetermined time threshold, and when the number of times exceeds the predetermined number, increasing the detection threshold by a predetermined adjustment amount.

[0009] A computer program for vehicle control provided in yet another embodiment causes a processor mounted on the vehicle to calculate the reliability of lane markings being represented in images generated by a camera mounted on the vehicle, enable the application of a control mode that controls the vehicle's travel based on the lane markings when the reliability is equal to or greater than a predetermined detection threshold, disable the application of the control mode when the reliability is less than the detection threshold, count the number of times the applicable period from when the control mode becomes applicable to when the control mode becomes unapplicable is equal to or less than a predetermined time threshold, and when the number of times exceeds the predetermined number, increase the detection threshold by a predetermined adjustment amount. [Effects of the Invention]

[0010] The vehicle control device according to the present disclosure has the advantage of being able to prevent a control mode that controls vehicle travel based on lane markings from being switched on and off in a short period of time. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic configuration diagram of a vehicle in which a vehicle control device is implemented; [Figure 2] FIG. 2 is a functional block diagram of a processor of an ECU related to vehicle control processing according to the first embodiment. [Figure 3] FIG. 10 illustrates the relationship between short term count and detection threshold. [Figure 4]4 is an operational flowchart of a vehicle control process according to the first embodiment. [Figure 5] FIG. 10 is a functional block diagram of a processor of an ECU related to vehicle control processing according to a second embodiment. [Figure 6] 10 is an operational flowchart of a vehicle control process according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] A vehicle control device, a vehicle control method, and a vehicle control computer program executed by the vehicle control device will be described below with reference to the drawings. The vehicle control device calculates a reliability indicating the likelihood that lane markings are present in an image generated by an onboard camera. Furthermore, when the reliability is equal to or greater than a predetermined detection threshold, the vehicle control device enables application of a control mode that controls vehicle travel based on the lane markings (hereinafter, for convenience of explanation, referred to as a lane keeping mode). On the other hand, when the reliability is less than the detection threshold, the vehicle control device disables application of the lane keeping mode. The vehicle control device then counts the number of times that the applicable period from when the lane keeping mode becomes applicable to when the lane keeping mode becomes unapplicable is equal to or less than a predetermined time threshold, and when the number of times exceeds the predetermined number, the vehicle control device increases the detection threshold by a predetermined adjustment amount.

[0013] 1 is a schematic configuration diagram of a vehicle in which an electronic control device, which is an example of a vehicle control device, is implemented. Vehicle 1 has a camera 2 and an electronic control unit (ECU) 3, which is an example of a vehicle control device. Camera 2 and ECU 3 are communicatively connected. Vehicle 1 may further have a distance measurement sensor (not shown) such as a LiDAR sensor or radar. Vehicle 1 may also have a wireless communication terminal (not shown) for wireless communication with other devices, a receiver (not shown) such as a GPS receiver that measures the position of vehicle 1 in accordance with a satellite positioning system, and a storage device (not shown) for storing map information.

[0014] Camera 2 is mounted inside the cabin of vehicle 1 so as to capture an image of a predetermined capture range around vehicle 1. Camera 2 captures an image of the predetermined capture range at predetermined capture intervals (for example, 1 / 30 to 1 / 10 seconds) and generates an image showing the capture range. Camera 2 outputs the generated image to ECU 3 every time it generates an image. In this embodiment, the capture range is the area ahead of vehicle 1, and camera 2 is attached so as to face the front of vehicle 1. Note that vehicle 1 may be provided with two or more cameras with different capture directions or focal lengths.

[0015] The ECU 3 is capable of executing driving control according to a lane keeping mode as an example of vehicle control processing for the vehicle 1. As driving control in the lane keeping mode, the ECU 3 performs, for example, departure prevention control to prevent the vehicle 1 from departing from the lane in which the vehicle 1 is traveling, lane centering assist control to cause the vehicle 1 to travel along the center of the lane in which the vehicle 1 is traveling, or lateral position control to adjust the position at which the vehicle 1 travels within the lane based on lane markings. Driving control according to the lane keeping mode may be executed as one function of driving assist control or autonomous driving control. In other words, the driving assist mode and the autonomous driving mode are examples of lane keeping modes.

[0016] The ECU 3 includes a communication interface 11, a memory 12, and a processor 13. The communication interface 11, the memory 12, and the processor 13 may be configured as separate circuits, or may be integrated into a single integrated circuit.

[0017] The communication interface 11 has an interface circuit for connecting the ECU 3 to other devices, and passes the image received from the camera 2 to the processor 13 .

[0018] The memory 12 is an example of a storage unit and includes, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 12 stores various data used in the vehicle control process executed by the processor 13 of the ECU 3. For example, the memory 12 stores various information and thresholds used to calculate the reliability of lane markings, such as a parameter set that defines a classifier for detecting lane markings. The memory 12 also stores parameters of the camera 2, such as the focal length, shooting direction, and installation position (including installation height) of the camera 2. The memory 12 also temporarily stores images received from the camera 2. The memory 12 also temporarily stores various data generated during the vehicle control process. Such data includes a short-term count that indicates the applicability period from when the lane keeping mode becomes applicable to when the lane keeping mode becomes inapplicable and the number of times that applicability period is equal to or less than a predetermined time threshold.

[0019] The processor 13 includes one or more central processing units (CPUs) and their peripheral circuits. The processor 13 may further include other arithmetic circuits such as a logic operation unit, a numerical operation unit, or a graphics processing unit. The processor 13 executes vehicle control processing for the vehicle 1.

[0020] (First embodiment) The following describes the vehicle control process according to the first embodiment. In this embodiment, the processor 13 adjusts the detection threshold value that is compared with the reliability required for detecting lane markings, depending on the number of times that the duration during which the lane keeping mode can be applied is equal to or greater than a predetermined time threshold.

[0021] 2 is a functional block diagram of the processor 13 relating to the vehicle control process according to the first embodiment. The processor 13 includes a calculation unit 21, an applicability determination unit 22, a control unit 23, a counting unit 24, and a threshold adjustment unit 25. Each of these units included in the processor 13 is a functional module implemented by a computer program running on the processor 13, for example. Alternatively, each of these units included in the processor 13 may be a dedicated arithmetic circuit provided in the processor 13.

[0022] The calculation unit 21 periodically calculates a reliability indicating the likelihood that a lane marking is represented in the latest image generated by the camera 2. To do this, the calculation unit 21 detects the reliability by inputting the image acquired from the camera 2 into a classifier that has been trained in advance to calculate the reliability of the lane marking. Such a classifier is configured as a deep neural network (DNN) for semantic segmentation, such as a fully convolutional network or U-net. Such a classifier is trained in advance according to a predetermined learning method, such as backpropagation, using a large number of training images depicting objects to be identified, including lane markings, so as to calculate the reliability of each object that may be represented in that pixel. The calculation unit 21 determines, among the reliability calculated for each object, a set of pixels with the highest reliability for the lane marking as a candidate area that may represent a lane marking. The calculation unit 21 then determines the statistical representative value (e.g., average, median, or mode) of the reliability calculated for the lane markings at each pixel included in the candidate area as the reliability that the lane markings are represented in the image.

[0023] Alternatively, the calculation unit 21 may apply an edge detection filter such as a Sobel filter to the image to calculate the edge strength for each pixel, and detect pixels whose edge strength is equal to or greater than a predetermined threshold as edge pixels. For each horizontal scan line that is located at a different vertical position, the calculation unit 21 determines a pair of edge pixels on that scan line that are separated by a distance within a predetermined tolerance range from the width or lane width of the lane marking. The width and lane width of the lane marking for each vertical position on the image are pre-stored in the memory 12. Furthermore, the calculation unit 21 may determine the continuity of pairs of edge pixels in a direction on the image that corresponds to the traveling direction of the vehicle 1, or the continuity of pairs of edge pixels at the same position across images acquired in a time series. The calculation unit 21 may then determine a set of pairs of edge pixels with the longest continuity on the image as a candidate region. The calculation unit 21 calculates the reliability as a weighted sum of the average edge strength of each edge pixel included in the candidate area, the average deviation from the lane marking line width or lane width for each pair of edge pixels, the degree of continuity of pairs of edge pixels on the image, and the degree of continuity of pairs of edge pixels at the same position between images in a time series.

[0024] Every time the calculation unit 21 calculates a reliability, it notifies the applicability determination unit 22 of the calculated reliability and information indicating the candidate region.

[0025] The applicability determination unit 22 compares the reliability with a predetermined detection threshold. If the reliability is equal to or greater than the detection threshold, the applicability determination unit 22 enables application of the lane keeping mode. On the other hand, if the reliability is less than the detection threshold, the applicability determination unit 22 disables application of the lane keeping mode.

[0026] When the applicability determination unit 22 determines that the lane keeping mode is applicable when the lane keeping mode is not actually applied, the applicability determination unit 22 notifies the driver via a notification device (e.g., a display device or a speaker, not shown) provided in the cabin of the vehicle 1 that a driving mode (an autonomous driving mode or a driving assistance mode) equivalent to the lane keeping mode is applicable. The applicability determination unit 22 also starts timing an applicable period. After the notification, when the ECU 3 receives an operation signal to turn on an applicable driving mode from an operation device (not shown) provided in the cabin, the applicability determination unit 22 starts applying the driving mode. Note that the applicability determination unit 22 may start timing an applicable period at the time when application of the driving mode is started. Alternatively, the applicability determination unit 22 may count an applicable period even if no driving mode equivalent to the lane keeping mode is actually applied. Furthermore, a driving mode may be determined to be applicable only when the driving situation of the vehicle 1 is a predetermined situation. For example, it may be determined that the autonomous driving mode is applicable only when the vehicle 1 is traveling on a motorway within an area represented in the map information. Furthermore, when the traveling situation of the vehicle 1 is a predetermined situation, the applicability determination unit 22 may automatically start applying the driving mode.

[0027] When application of a driving mode equivalent to the lane keeping mode begins, the applicability determination unit 22 notifies the control unit 23 of the start of application of the driving mode. Furthermore, from the start of application of the driving mode until the lane keeping mode becomes inapplicable, the applicability determination unit 22 regards the candidate area received from the calculation unit 21 as a lane marking area in which lane markings are displayed, and notifies the control unit 23 of information representing the lane marking area. Furthermore, after starting to measure the applicable period, when it is determined that the lane keeping mode is inapplicable, the applicability determination unit 22 ends measuring the applicable period and notifies the counting unit 24 of the applicable period. Furthermore, the applicability determination unit 22 notifies the control unit 23 of the end of the applied driving mode, and also notifies the driver of the end of the applied driving mode via a notification device.

[0028] The control unit 23 controls the driving of the vehicle 1 according to the applied driving mode. When a driving assistance mode, which is an example of a lane keeping mode, is applied, the control unit 23 calculates the distance between the left and right lane markings that separate the vehicle 1 from the own lane (hereinafter referred to as the lateral distance) at predetermined intervals based on the latest image.

[0029] Therefore, the control unit 23 determines that the two regions of the marking area closest to the position of the vehicle 1 are regions representing the lane markings that define the vehicle's own lane. The control unit 23 also determines the horizontal positions of the pixels representing the lane markings that define the vehicle's own lane, respectively, at the positions closest to the bottom of the image as reference positions. Here, the positions on the image correspond one-to-one with the orientations as viewed from the camera 2, and the lane markings are located on the road surface. Based on the reference positions on the image for the left and right lane markings, the control unit 23 calculates the distances from the camera 2 to the left and right lane markings for each lane marking, based on the camera 2 parameters such as the reference positions on the image, the shooting direction, and the installation height. The control unit 23 then calculates the lateral distance for each lane marking on the left and right sides of the vehicle 1 by subtracting the distance from the camera 2 to the side of the vehicle 1 from the distance from the camera 2 to the lane marking.

[0030] The control unit 23 determines that the vehicle 1 may deviate from its own lane when either the left or right lateral distance falls below a first distance threshold (e.g., several centimeters to tens of centimeters) and the predicted trajectory of the vehicle 1 from that point onward is predicted to intersect with a lane marking that divides the own lane. Therefore, the control unit 23 compares the left and right lateral distances with the first distance threshold each time the control unit 23 calculates the left and right lateral distances. The control unit 23 then detects the time when either the left or right lateral distance falls below the first distance threshold as a caution time. The control unit 23 predicts the change in the lateral distance over a predetermined future period after the caution time based on the change in the lateral distance over a certain period of time immediately preceding the current time. In this case, the control unit 23 predicts the change in the lateral distance over a predetermined future period by applying a prediction filter, such as a Kalman filter, to the change in the lateral distance over a certain period of time immediately preceding the current time. Alternatively, the control unit 23 may predict the change in the lateral distance over a predetermined future period by applying a predetermined extrapolation process to the change in the lateral distance over a certain period of time immediately preceding the current time. If the predicted value of the lateral distance becomes zero at any point within a predetermined period, the control unit 23 determines that the predicted trajectory of the vehicle 1 will intersect with a lane marking that divides the own lane.

[0031] Furthermore, after the caution point, when the lateral distances on the left and right become greater than the distance obtained by adding a predetermined offset distance to the first distance threshold, the control unit 23 may determine that the vehicle 1 will not deviate from its own lane.

[0032] The control unit 23 may determine that there is a possibility that the vehicle 1 will deviate from its own lane if the lateral distance on either the left or right side remains equal to or less than the first distance threshold for a predetermined period of time or more.

[0033] When it is determined that there is a possibility that the vehicle 1 will deviate from its own lane, the control unit 23 performs deviation prevention control by correcting the current steering angle of the steering by a predetermined steering amount in a direction in which the vehicle 1 will move toward the center of the own lane. The control unit 23 then controls the steering according to the corrected steering angle, thereby preventing the vehicle 1 from deviating from its own lane.

[0034] After starting deviation prevention control, when the lateral distances on the left and right sides become greater than the distance obtained by adding a predetermined offset distance to the first distance threshold, the control unit 23 ends deviation prevention control.

[0035] Furthermore, when an autonomous driving mode, which is another example of a lane keeping mode, is applied, the control unit 23 sets, at predetermined intervals, the center line between the left and right lane markings that define the vehicle's own lane, detected from the latest image, as the planned travel trajectory of the vehicle 1. Then, the control unit 23 controls the steering so that the vehicle 1 travels along the planned travel trajectory.

[0036] Counting unit 24 compares the applicable period notified by applicability determination unit 22 with a predetermined time threshold (for example, several tens of seconds). If the applicable period is equal to or less than the time threshold, counting unit 24 increments the value of the short-term count stored in memory 12 by 1. On the other hand, if the applicable period is longer than the time threshold, counting unit 24 does not change the value of the short-term count.

[0037] When the value of the short-term count stored in memory 12 is changed, threshold adjustment unit 25 determines whether the value of the short-term count has reached a predetermined number of times (for example, several times to several dozen times). If the value of the short-term count has reached the predetermined number of times, threshold adjustment unit 25 increases the value of the detection threshold to be compared with the reliability by a predetermined adjustment amount (for example, a value equivalent to several percent of the original detection threshold, or a preset fixed value). Threshold adjustment unit 25 then resets the value of the short-term count to 0. By adjusting the detection threshold in this way, erroneous recognition of lane markings is less likely to occur even in situations where erroneous recognition of lane markings is likely to occur, such as when part of the road surface is covered with snow, and as a result, switching between the applicability of the lane keeping mode and the like is prevented from occurring in a short period of time.

[0038] Figure 3 shows the relationship between the short-term count and the detection threshold. In both the upper and lower charts of Figure 3, the horizontal axis represents time. In the upper chart, the vertical axis represents the short-term count value, and in the lower chart, the vertical axis represents the detection threshold value. Graph 301 in the upper chart represents the transition of the short-term count value, and graph 302 in the upper chart represents the transition of the detection threshold value.

[0039] As shown in graphs 301 and 302, the detection threshold value is kept constant until the value of the short term count reaches a predetermined number N. When the value of the short term count reaches the predetermined number N at time t1, the detection threshold value is adjusted to be higher by a predetermined adjustment amount Δ, and the value of the short term count is reset to 0. Thereafter, the detection threshold value is kept constant until time t2. When the value of the short term count reaches the predetermined number N again at time t2, the detection threshold value is also readjusted to be higher by the predetermined adjustment amount Δ, and the value of the short term count is reset to 0. In this way, each time the value of the short term count reaches the predetermined number N, the detection threshold value becomes higher, making it more difficult to apply the lane keeping mode.

[0040] The threshold adjustment unit 25 may decrease the detection threshold value as the time elapsed since the short-term count last reached the predetermined number of times increases. Furthermore, the threshold adjustment unit 25 does not adjust the detection threshold value so that it exceeds a preset upper limit. This prevents the detection threshold value from being adjusted too high, which would otherwise disable the lane keeping mode even in a situation where the lane keeping mode should be applied.

[0041] 4 is an operational flowchart of the vehicle control process according to the first embodiment. The processor 13 executes the vehicle control process at predetermined intervals in accordance with the following operational flowchart.

[0042] The calculation unit 21 calculates the reliability that lane markings are displayed in the image generated by the camera 2 (step S101). The applicability determination unit 22 determines whether the reliability is equal to or greater than the detection threshold Th (step S102). If the reliability is equal to or greater than the detection threshold Th (step S102-Yes), the applicability determination unit 22 determines that the lane keeping mode is applicable. Then, the control unit 23 controls the traveling of the vehicle 1 in accordance with the driving mode corresponding to the lane keeping mode, either automatically or after a driver operation approving the application of the driving mode corresponding to the lane keeping mode (step S103).

[0043] On the other hand, if the reliability is less than the detection threshold Th (step S102—No), the applicability determination unit 22 determines that the lane keeping mode cannot be applied (step S104). Furthermore, if there is a driving mode that corresponds to the lane keeping mode and is currently being applied, the control unit 23 terminates the application of that driving mode. The counting unit 24 then determines whether the applicable period from when the lane keeping mode becomes applicable to when it becomes unapplicable is equal to or less than a predetermined time threshold (step S105). If the applicable period is equal to or less than the time threshold (step S105—Yes), the counting unit 24 increments the value of the short-term count C by 1 (step S106). Thereafter, the threshold adjustment unit 25 determines whether the value of the short-term count C has reached a predetermined number N (step S107). If the value of the short-term count C has reached the predetermined number N (step S107—Yes), the threshold adjustment unit 25 increases the detection threshold Th by an adjustment amount Δ and resets the value of the short-term count C to 0 (step S108). Then, processor 13 ends the vehicle control process. Processor 13 also ends the vehicle control process if the applicable period is longer than the time threshold in step S105 (No in step S105) or if the value of short-term count C has not reached the predetermined number of times N in step S107 (No in step S107).

[0044] As described above, the vehicle control device according to the first embodiment adjusts the threshold for detecting lane markings so that the detection of lane markings from an image becomes more difficult as the frequency of the period during which the lane keeping mode can be applied becomes shorter. Therefore, this vehicle control device can prevent frequent changes in whether or not the lane keeping mode can be applied.

[0045] (Second embodiment) Next, a vehicle control device according to a second embodiment will be described. In the second embodiment, the vehicle control device determines whether to apply the lane keeping mode based not only on the reliability of lane markings being displayed on the image but also on the classification results of the image by a classifier relating to the visibility of the lane markings on the image. Furthermore, this vehicle control device trains the classifier based on images obtained during a period when the period during which the lane keeping mode can be applied is equal to or shorter than a predetermined time threshold.

[0046] 5 is a functional block diagram of the processor 13 relating to the vehicle control processing according to the second embodiment. The processor 13 has a calculation unit 21, an applicability determination unit 22, a control unit 23, and a learning unit 26. Each of these units in the processor 13 is, for example, a functional module realized by a computer program running on the processor 13. Alternatively, each of these units in the processor 13 may be a dedicated arithmetic circuit provided in the processor 13. Note that differences from the first embodiment will be described below.

[0047] The applicability determination unit 22 enables application of the lane keeping mode when the reliability that lane markings are displayed on the image is equal to or greater than a predetermined detection threshold and the classifier for the visibility of lane markings classifies the image into a first class corresponding to a situation in which lane markings are visible. On the other hand, when the reliability is less than the detection threshold or the classifier classifies the image into a second class corresponding to a situation in which lane markings are not visible, the applicability determination unit 22 disables application of the lane keeping mode.

[0048] The classifier may be, for example, a classifier based on a support vector machine or a DNN. If the classifier is based on a DNN, the classifier may have, for example, from the input side, multiple convolutional layers, one or more fully connected layers, and an output layer that calculates a value indicating which class an image is classified into using a softmax operation. If the classifier classifies an image input to the classifier into the same class as a sample image known to be in a situation where lane markings are visible, the applicability determination unit 22 determines that the input image has been classified into the first class. Alternatively, if the classifier classifies an input image into a class different from the class into which a sample image known to be in a situation where lane markings are not visible is classified, the applicability determination unit 22 may determine that the input image has been classified into the first class. On the other hand, if the classifier classifies the input image into a class different from the class into which a sample image known to be in a situation where lane markings are visible is classified, the applicability determination unit 22 determines that the input image has been classified into the second class. Alternatively, if the classifier classifies the input image into the same class as the class into which a sample image known to be in a situation where lane markings are not visible is classified, the applicability determination unit 22 may determine that the input image has been classified into the second class. The above sample images are stored in memory 12 in advance.

[0049] As in the first embodiment, when it is determined that the lane keeping mode cannot be applied, the applicability determination unit 22 stops timing the applicable period from when the lane keeping mode becomes applicable. Then, the applicability determination unit 22 notifies the learning unit 26 of the time when the lane keeping mode becomes applicable and the time when timing ends, and the applicable period.

[0050] The learning unit 26 compares the applicable period from when the lane keeping mode becomes applicable until when the mode becomes inapplicable with a predetermined time threshold. If the applicable period is equal to or shorter than the time threshold, it is highly likely that the images acquired during the applicable period, particularly the images acquired when it is determined that the lane keeping mode is inapplicable, do not actually depict lane markings. Therefore, the learning unit 26 trains the classifier based on images acquired by the camera 2 during the applicable period (hereinafter referred to as additional training images). The learning unit 26 trains the classifier using a plurality of sample images, pre-stored in the memory 12, in which the visibility of lane markings is known, and the additional training images, according to a predetermined training method appropriate for the classifier. The additional training images are used to train the classifier as images representing situations in which lane markings are not visible. After the classifier has trained, the learning unit 26 stores in the memory 12 the center of gravity of the feature amounts of each image included in the class representing a situation in which lane markings are visible (e.g., a feature map output from the most output convolutional layer) and the center of gravity of each image included in the class representing a situation in which lane markings are not visible. Furthermore, the learning unit 26 may add, as sample images representing a situation in which lane markings are not visible, additional training images that are classified into a first class representing a situation in which lane markings are visible and for which the Mahalanobis distance between the feature amounts calculated by the classifier and the center of gravity of that class is less than a predetermined distance (e.g., 1). Similarly, the learning unit 26 may add, as sample images representing a situation in which lane markings are not visible, additional training images that are classified into a second class representing a situation in which lane markings are not visible and for which the Mahalanobis distance between the feature amounts calculated by the classifier and the center of gravity of that class is less than a predetermined distance.

[0051] FIG. 6 is an operational flowchart of the vehicle control process according to the second embodiment.

[0052] The calculation unit 21 calculates the reliability of the lane markings being displayed in the image generated by the camera 2 (step S201). The applicability determination unit 22 determines whether the calculated reliability is equal to or greater than the detection threshold Th (step S202). If the reliability is equal to or greater than the detection threshold Th (step S202—Yes), the applicability determination unit 22 determines whether the image has been classified by the classifier into a first class representing a situation in which the lane markings are visible (step S203). If the image has been classified into the first class (step S203—Yes), the applicability determination unit 22 determines that the lane keeping mode is applicable. Then, the control unit 23 controls the vehicle 1 automatically or after a driver's operation to approve the application of the driving mode corresponding to the lane keeping mode, in accordance with the driving mode corresponding to the lane keeping mode (step S204). Then, the processor 13 ends the vehicle control process.

[0053] On the other hand, if the reliability is less than the detection threshold Th (step S202—No), or if the image is classified into a second class representing a situation in which lane markings are not visible (step S203—No), the applicability determination unit 22 determines that the lane keeping mode is not applicable (step S205). Furthermore, if there is a driving mode that corresponds to the lane keeping mode and is currently being applied, the control unit 23 terminates the application of that driving mode. The learning unit 26 then determines whether the applicable period from when the lane keeping mode becomes applicable to when it becomes unavailable is equal to or less than a predetermined time threshold (step S206). If the applicable period is equal to or less than the time threshold (step S206—Yes), the learning unit 26 trains a classifier using images obtained during the applicable period (step S207). Thereafter, the processor 13 terminates the vehicle control process. Furthermore, if the applicable period is longer than the time threshold in step S206 (step S206—No), the processor 13 terminates the vehicle control process.

[0054] As described above, the vehicle control device according to the second embodiment enables application of the lane keeping mode when the reliability of the lane markings calculated from an image classified as one in which the lane markings are visible is equal to or greater than the detection threshold. Therefore, this vehicle control device can reduce the risk of applying the lane keeping mode because an object on the road surface is mistakenly recognized as a lane marking in a situation in which the lane markings are difficult to see. As a result, this vehicle control device can prevent frequent changes in whether or not the lane keeping mode is applicable. Furthermore, this vehicle control device trains the classifier using images taken during a short period in which the lane keeping mode is applicable, thereby improving the accuracy of the classifier's classification of images according to the visibility of the lane markings.

[0055] A computer program that realizes the functions of the processor 13 of the ECU 3 according to any of the above embodiments or variations may be provided in a form recorded on a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium or an optical recording medium.

[0056] As described above, those skilled in the art can make various modifications to the embodiments within the scope of the present invention. [Explanation of symbols]

[0057] REFERENCE SIGNS LIST 1 vehicle, 2 camera, 3 electronic control unit (ECU), 11 communication interface, 12 memory, 13 processor, 21 calculation unit, 22 applicability determination unit, 23 control unit, 24 counting unit, 25 threshold adjustment unit, 26 learning unit

Claims

1. a calculation unit that calculates a reliability of lane markings being represented in an image generated by a camera mounted on the vehicle; an applicability determination unit that enables application of a control mode that controls the vehicle's traveling based on lane markings when the reliability is equal to or greater than a predetermined detection threshold, and that disables application of the control mode when the reliability is less than the detection threshold; a counting unit that counts the number of times that an applicable period from when the control mode becomes applicable to when the control mode becomes inapplicable is equal to or less than a predetermined time threshold; a threshold adjustment unit that increases the detection threshold by a predetermined adjustment amount when the number of times reaches or exceeds a predetermined number of times; A vehicle control device having the above.

2. a calculation unit that calculates a reliability of lane markings being represented in an image generated by a camera mounted on the vehicle; an applicability determination unit that enables application of a control mode for controlling the vehicle's travel based on lane lines when the reliability is equal to or greater than a predetermined detection threshold and when a classifier for lane line visibility status classifies the image into a first class corresponding to a situation in which lane lines are visible, and that disables application of the control mode when the reliability is less than the detection threshold or when the classifier classifies the image into a second class corresponding to a situation in which lane lines are not visible; a learning unit that learns the classifier based on the images obtained within an applicable period from when the application of the control mode becomes possible to when the application of the control mode becomes impossible, when the applicable period is equal to or less than a predetermined time threshold; A vehicle control device having the above.

3. Calculating a reliability of lane markings being represented in an image generated by a camera mounted on the vehicle; a control mode for controlling the vehicle's running based on lane markings when the reliability is equal to or greater than a predetermined detection threshold is enabled; Disable application of the control mode when the reliability is less than the detection threshold; Counting the number of times that the applicable period from when the control mode becomes applicable to when the control mode becomes inapplicable is equal to or less than a predetermined time threshold; When the number of times is equal to or greater than a predetermined number of times, the detection threshold is increased by a predetermined adjustment amount. A vehicle control method comprising:

4. Calculating a reliability of lane markings being represented in an image generated by a camera mounted on the vehicle; a control mode for controlling the vehicle's running based on lane markings when the reliability is equal to or greater than a predetermined detection threshold is enabled; Disable application of the control mode when the reliability is less than the detection threshold; Counting the number of times that the applicable period from when the control mode becomes applicable to when the control mode becomes inapplicable is equal to or less than a predetermined time threshold; When the number of times is equal to or greater than a predetermined number of times, the detection threshold is increased by a predetermined adjustment amount. A computer program for vehicle control that causes a processor mounted on the vehicle to execute the above.

Images