Control device, control method, and program

Abstract

To accurately estimate the reliability of detecting road compartment lines.SOLUTION: A control device includes: a first processing unit that detects a first element corresponding to a road compartment line and a second element located outside the road compartment line from an image capturing the surroundings of a vehicle, and projects a point sequence corresponding to the first element onto a two-dimensional plane at predetermined distance intervals based on the actual distance; and a second processing unit that identifies inflection points using the point sequence and estimates the reliability of the detected road compartment line detected from the image based on whether the lateral distance between the first element and the second element, at the same longitudinal distance beyond the inflection point, is below a threshold.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] In recent years, efforts to provide sustainable transportation systems that take various situations into consideration have become more active. To achieve this, efforts are being focused on research and development to further improve traffic safety and convenience through research and development of driving assistance technologies. A lane mark detection device has been disclosed that includes a shape evaluation value calculation unit that calculates a shape evaluation value regarding whether an object has the shape of a lane mark, a distance evaluation value calculation unit that calculates a distance evaluation value regarding the field distance, and a reliability calculation unit that calculates the reliability of the lane mark based on the shape evaluation value and the distance evaluation value. By calculating the reliability based on the evaluation value regarding the field distance in addition to the evaluation value regarding the shape of the lane mark, it is possible to calculate the reliability according to the field distance, which is a factor that affects the accuracy of the detected position of the lane mark, and the calculation load for the reliability is reduced compared to a method that divides a captured image into regions and changes the reliability for each region (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-191040 Summary of the Invention [Problem to be solved by the invention]

[0004] In the prior art, the reliability of road marking detection has not been sufficiently considered.

[0005] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a control device, a control method, and a program that can accurately estimate the reliability of road-dividing line detection. For example, by utilizing this reliability, vehicles can be controlled more accurately. This will ultimately contribute to the development of sustainable transportation systems. [Means for solving the problem]

[0006] The control device, control method, and program according to the present invention employ the following configuration. (1): A control device according to one embodiment of the present invention includes a first processing unit that detects a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the vehicle's surroundings, and projects a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance; and a second processing unit that uses the sequence of points to identify an inflection point and estimates the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is below a threshold.

[0007] (2): In the above aspect (1), the second processing unit derives each angle between adjacent point sequences and finds the inflection point where the change in angle over distance crosses zero, or the inflection point where the change in angle obtained by quadratically differentiating the change in angle with respect to distance crosses zero.

[0008] (3): In the above aspect (2), the second processing unit estimates the reliability of the road dividing line detected from the image when the difference between the two largest angles before and after the inflection point is equal to or greater than a threshold value.

[0009] (4): In any of the above aspects (1) to (3), the second processing unit estimates that when the inflection point does not exist, the reliability of the road dividing line detected from the image in a distant area is higher than when the inflection point exists.

[0010] (5): In any one of the above aspects (1) to (3), the vehicle control unit further comprises a vehicle control unit that controls the vehicle by referring to the reliability.

[0011] (6) In any of the above aspects (1) to (3), the vehicle control unit further comprises a vehicle control unit that changes the level of control of the vehicle by referring to the reliability.

[0012] (7): In another aspect of the control method of the present invention, a computer executes the following processes: detecting a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the vehicle's surroundings; projecting a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance; identifying an inflection point using the sequence of points; and estimating the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is equal to or less than a threshold value.

[0013] (8): A program according to another aspect of the present invention causes a computer to perform the following processes: detect a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the vehicle's surroundings; project a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance; identify an inflection point using the sequence of points; and estimate the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is below a threshold. [Effects of the Invention]

[0014] According to aspects (1) to (8), the reliability of detection of road dividing lines can be estimated with high accuracy.

[0015] According to the aspect (5), the control unit can realize the control of the vehicle according to the reliability. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a configuration diagram of a vehicle system 1 that uses a vehicle control system according to an embodiment. [Figure 2] FIG. 2 is a diagram illustrating an example of the functional configuration of a determination processing unit 120. [Figure 3] 10 is a diagram for explaining the processing of a first conversion processing unit 122 and a second conversion processing unit 124. FIG. [Figure 4] FIG. 10 is a diagram for explaining processing using a projected point sequence. [Figure 5] FIG. 10 is a diagram showing changes in angle θ for each vertical distance. [Figure 6] FIG. 10 is a diagram showing the change in horizontal distance for each vertical distance. [Figure 7] 10 is a diagram showing the error between the position of a road dividing line that is actually measured and the position of a road dividing line that is obtained by image processing. FIG. [Figure 8] FIG. 10 is a diagram illustrating another example of an inflection point. [Figure 9] 3 is a flowchart showing an example of the flow of processing executed by the driving assistance device 100. [Figure 10] FIG. 10 is a diagram showing an example of an image IM2 in which a road that is not a curved road is captured. [Figure 11] FIG. 10 is a diagram showing a change in angle θ in image IM2. [Figure 12] FIG. 10 is a diagram showing changes in lateral distance in image IM2. [Figure 13] FIG. 10 is a diagram showing the error between the position of a road dividing line on a non-curved road and the position of an actually measured road dividing line. [Figure 14] 6 is a flowchart showing another example of the flow of the process executed by the driving assistance device 100. DETAILED DESCRIPTION OF THE INVENTION

[0017] [Overall configuration] FIG. 1 is a configuration diagram of a vehicle system 1 that uses a vehicle control system according to an embodiment. The vehicle on which the vehicle system 1 is mounted may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a secondary battery or a fuel cell. This embodiment will be described as being applied to a vehicle, but may also be applied to other moving objects instead of a vehicle.

[0018] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU 60, an operator 80, a direction indicator 90, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other via multiplex communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. The configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The driving assistance device 100 is an example of a "controller."

[0019] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to any location of a vehicle (hereinafter referred to as vehicle M) in which the vehicle system 1 is installed. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically captures images of the periphery of the vehicle M. The camera 10 may be a stereo camera.

[0020] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method.

[0021] The LIDAR 14 irradiates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 may be attached to any location on the vehicle M.

[0022] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, the radar device 12, and the LIDAR 14 to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 10, the radar device 12, and the LIDAR 14 directly to the driving assistance device 100. The object recognition device 16 may be omitted from the vehicle system 1.

[0023] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via a wireless base station.

[0024] The HMI 30 presents various information to the occupants of the vehicle M and accepts input operations by the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, keys, etc. The HMI 30 is equipped with a display device. The display device is, for example, a so-called multi-information display, provided in the center of the instrument panel of the vehicle M, and displays various information about the vehicle M, such as a speedometer that indicates the traveling speed of the vehicle M or a tachometer that indicates the rotation speed (rotational speed) of the internal combustion engine equipped in the vehicle M.

[0025] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, a direction sensor that detects the direction of the vehicle M, and the like.

[0026] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as a hard disk drive (HDD) or flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be identified or supplemented by an inertial navigation system (INS) that uses the output of the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. The navigation HMI 52 may share some or all of the components with the HMI 30 described above. The route determination unit 53 determines, for example, a route (hereinafter, a route on a map) from the position of the vehicle M identified by the GNSS receiver 51 (or any input position) to a destination input by the occupant using the navigation HMI 52, with reference to the first map information 54. The first map information 54 is information that represents road shapes using, for example, links indicating roads and nodes connected by the links. The first map information 54 may also include information such as road curvature and POI (Point of Interest) information. The route on the map is output to the MPU 60. The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized, for example, by the functions of a terminal device such as a smartphone or tablet device owned by the occupant. The navigation device 50 may transmit the current position and destination to a navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0027] The MPU 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of blocks (for example, every 100 m in the vehicle traveling direction) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines, for example, which lane from the left the vehicle M should take. When a branch point is present on the route on the map, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel on a reasonable route to the branch point. For example, when the vehicle M arrives a predetermined distance before the branch road on which the vehicle M is traveling, the recommended lane determination unit 61 determines a lane connecting to the branch road as the recommended lane. The recommended lane determination unit 61 and the second map information 62 may be functional units or information included in another device, such as the driving assistance device 100.

[0028] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. The second map information 62 may be updated as needed by the communication device 20 communicating with other devices.

[0029] The operators 80 include, for example, a steering wheel 82 as well as an accelerator pedal, a brake pedal, a shift lever, and other operators. The operators 80 are fitted with sensors that detect the amount of operation or whether or not an operation is performed, and the detection results are output to the driving assistance device 100 or some or all of the driving force output device 200, the brake device 210, and the steering device 220. The steering wheel 82 does not necessarily have to be annular, and may be in the form of an irregularly shaped steering wheel, a joystick, a button, or the like. The operators 80 include a first operator 84. The direction indicator 90 turns on or off in response to operation of the first operator 84.

[0030] The first operator 84 is, for example, a turn signal lever switch. For example, when the driver operates the first operator 84, the direction indicator 90 lights up in response to the operation. When the driver performs a predetermined operation on the first operator 84, the control unit 150 activates an ALC (Auto Lane Change) function and causes the vehicle M to perform an automatic lane change. The predetermined operation is an operation that triggers activation of the ALC function. The predetermined operation is, for example, operating the turn signal lever switch in the direction of a desired lane change for a predetermined period of time, or pressing the turn signal lever switch to a predetermined position. More specifically, the predetermined operation is operating the turn signal lever switch in the direction of a desired lane change while maintaining a predetermined position for a predetermined period of time. The first operator 84 may be another embodiment, such as a button, instead of a turn signal lever switch. The ALC function may also be activated by operating another operator, such as a predetermined button.

[0031] The driving assistance device 100 includes, for example, a recognition unit 110, a determination processing unit 120, and a control unit 150. The recognition unit 110 and the control unit 150 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or an SOC (System On Chip), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (storage device having a non-transitory storage medium) such as an HDD or flash memory of the driving assistance device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving assistance device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0032] The recognition unit 110 recognizes the position, speed, acceleration, and other states of objects around the vehicle M based on information input from the camera 10, the radar device 12, and the LIDAR 14 via the object recognition device 16. The position of an object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by an area. The "state" of an object may include the acceleration or jerk of the object, or the "behavioral state" (for example, whether or not the object is changing lanes or is about to change lanes).

[0033] The recognition unit 110 recognizes, for example, the lane in which the vehicle M is traveling (driving lane). For example, the recognition unit 110 recognizes the driving lane by comparing the pattern of road dividing lines (e.g., an arrangement of solid lines and dashed lines) obtained from the second map information 62 with the pattern of road dividing lines around the vehicle M recognized from an image captured by the camera 10. The recognition unit 110 may recognize the driving lane by recognizing road boundaries (road boundaries) including not only road dividing lines but also road dividing lines, shoulders, curbs, medians, guardrails, etc. In this recognition, the position of the vehicle M obtained from the navigation device 50 and the processing results by the INS may be taken into consideration. The recognition unit 110 recognizes stop lines, obstacles, red lights, toll booths, and other road phenomena.

[0034] When recognizing the driving lane, the recognition unit 110 recognizes the position and orientation of the vehicle M with respect to the driving lane. For example, the recognition unit 110 may recognize the deviation of the reference point of the vehicle M from the center of the lane and the angle it forms with a line connecting the centers of the lanes in the traveling direction of the vehicle M as the relative position and orientation of the vehicle M with respect to the driving lane. Alternatively, the recognition unit 110 may recognize the position of the reference point of the vehicle M with respect to either side edge of the driving lane (a road dividing line or a road boundary) as the relative position of the vehicle M with respect to the driving lane.

[0035] The determination processing unit 120 will be described later.

[0036] The control unit 150 executes driving assistance control. For example, the control unit 150 automatically controls the driving force output device 200 and the brake device 210 without relying on the driver's operation, thereby automatically controlling the speed of the vehicle M. The control unit 150 executes so-called ACC (Adaptive Cruise Control). The control unit 150 controls the vehicle M to travel at a set speed, or to travel by following a leading vehicle at a predetermined distance from the leading vehicle.

[0037] The control unit 150 controls the steering device 220 so that the vehicle M does not deviate from the driving lane. For example, the control unit 150 controls the steering device 220 so that the vehicle M travels in the center or near the center of the driving lane recognized by the recognition unit 110. Hereinafter, this control may be referred to as "lane keeping control." The control unit 150 executes hands-on lane keeping control and hands-off lane keeping control.

[0038] Hands-on lane keeping control is a control that is executed when the driver is gripping the steering wheel (when a steering grip sensor (not shown) detects that the driver is gripping the steering wheel). The conditions under which hands-on lane keeping control can be executed are less stringent than the conditions under which hands-off lane keeping control can be executed.

[0039] Hands-off lane keeping control is a control that is executed when the driver is not gripping the steering wheel (when a steering grip sensor, not shown, does not detect that the driver is gripping the steering wheel). Hands-off lane keeping control can be executed, for example, when the following conditions are met: the speed of vehicle M is equal to or greater than a predetermined speed, vehicle M is traveling on a predetermined road (for example, a road or type of road that has been set in advance as one for which hands-off lane keeping control can be executed), and the driver is monitoring the road ahead. When the driver is monitoring the road ahead, hands-off lane keeping control is executed, and when the driver is not monitoring the road ahead, hands-off lane keeping control is not executed or is stopped.

[0040] The above-described conditions under which the hands-on lane keeping control and the hands-off lane keeping control can be executed are merely examples, and other conditions (for example, vehicle M following a vehicle ahead) may be included, or some conditions may be omitted. The conditions under which the hands-on lane keeping control can be executed may be looser than the conditions under which the hands-off lane keeping control can be executed (the conditions under which the hands-off lane keeping control can be executed may be stricter than the conditions under which the hands-on lane keeping control can be executed). Whether the driver is monitoring the road ahead is recognized by driving assistance device 100 based on an image captured by a camera (not shown) that captures an image of the driver.

[0041] The driving force output device 200 outputs a driving force (torque) to the driving wheels for driving the vehicle M. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU that controls these. The ECU controls the above components in accordance with information input from the driving assistance device 100 or information input from the operator 80.

[0042] The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor in accordance with information input from the driving assistance device 100 or information input from the operator 80, so that a brake torque corresponding to the braking operation is output to each wheel.

[0043] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by applying a force to, for example, a rack and pinion mechanism. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the driving assistance device 100 or information input from the operator 80.

[0044] [Determination processing section] 2 is a diagram showing an example of the functional configuration of the determination processing unit 120. The determination processing unit 120 includes, for example, a first conversion processing unit 122, a second conversion processing unit 124, an angle processing unit 126, and a distance processing unit 128. The first conversion processing unit 122 is an example of a "first processing unit." The combined functional configuration of the angle processing unit 126 and the distance processing unit 128 is an example of a "second processing unit."

[0045] The first conversion processing unit 122 acquires an image of the periphery of the vehicle M captured by the camera 10. The first conversion processing unit 122 detects a first element corresponding to the road dividing line and a second element which is an element outside the road dividing line (an object such as an exterior wall or a curb) from the image of the periphery of the vehicle M. The first conversion processing unit 122 detects the first element corresponding to the road dividing line and the second element which is an element outside the road dividing line based on feature information extracted from the image. The second conversion processing unit 124 approximates the first element to a predetermined curve.

[0046] 3 is a diagram illustrating the processing of the first conversion processing unit 122 and the second conversion processing unit 124. The first conversion processing unit 122 acquires an image IM capturing an image of the periphery of the vehicle M. The first conversion processing unit 122 generates two-dimensional information IF1 by projecting road dividing lines, exterior walls, and the like from the image onto a two-dimensional plane. The two-dimensional information IF1 is an image that includes information indicating, for example, the road dividing line LR on the right side of the vehicle M (camera 10), the road dividing line LL on the left side, and the side wall C.

[0047] The first conversion processing unit 122, for example, performs image processing on the image to acquire feature information (e.g., brightness information) of the image, and identifies elements on the road (first elements corresponding to the road dividing lines, and second elements such as side walls and curbs outside the road dividing lines) based on the feature information. The first conversion processing unit 122 extracts portions of the image where the horizontal brightness changes by more than a threshold value. For example, a differential filter (Sobel filter) is used on the image to extract the features.

[0048] The above-mentioned transformation for projecting the information of the actually captured image onto a two-dimensional plane is performed based on, for example, the following equations (1) and (2). The projected point is defined as (d i ,d j ) Any point after transformation is (X, Y). "f" is the focal length between the camera 10, the image sensor, and the lens. "H" is the height of the camera 10. "λ" is the pixel size. "Z" is the position in the direction perpendicular to the XY plane. d i = Yf / Zλ formula (1) dj =Hf / Xλ formula...(2)

[0049] The second conversion processing unit 124 converts the above features into three-dimensional distances using, for example, the above-mentioned equations (1) and (2), and fits (regresses) the converted features to a numerical model of the road-dividing line. For example, the regression is performed using least squares of a cubic equation. As a result, three-dimensional information IF2 is generated.

[0050] For example, in the above processing, there is an area AR1 (an area corresponding to a position far from the camera 10) where road-dividing lines can be detected with high accuracy, and an area AR2 (an area corresponding to a position close to the camera 10) where road-dividing lines cannot be detected with high accuracy. However, it has sometimes been impossible to determine whether accuracy is guaranteed in the area between area AR1 and area AR2. In this embodiment, the reliability of road-dividing line recognition can be determined by performing the processing described below.

[0051] [Overview of the reliability determination process] The first conversion processing unit 122 projects a sequence of points corresponding to a first element (a road dividing line) onto a two-dimensional surface at a predetermined distance interval (for example, an arbitrary distance such as 0.5 m, 1 m, or 2 m) according to the actual distance. The first conversion processing unit 122 projects a sequence of points corresponding to a second element (an object such as an exterior wall or a curb) onto a two-dimensional surface at a predetermined distance interval according to the actual distance. The angle processing unit 126 identifies an inflection point using the sequence of points of the first element (or the second element). FIG. 4 is a diagram for explaining processing using the projected sequence of points.

[0052] The angle processing unit 126 calculates the angle θ between the sequence of points on the road-dividing line on the image (see FIG. 4), and calculates the inflection point where the transition of the angle θ for each longitudinal distance crosses zero. For example, the angle θ is calculated using equation (3). The angle processing unit 126 calculates θ between each sequence of points on the road-dividing line. (X NL ,Y NL ) and (X L ,Y L ) will be discussed later. θ=tan -1 (dy / dx) Equation (3)

[0053] FIG. 5 is a diagram showing changes in angle θ for each longitudinal distance. The longitudinal distance is the distance based on the point ahead or the point behind the point sequence for which angle θ was determined. In FIG. 5, angle processing unit 126 identifies an inflection point (zero crossing point) where the progression of changes in angle θ crosses zero. If an inflection point is present, it means that the road dividing line is a road dividing line provided on a curved road. The area after the inflection point is an area where it may not be possible to accurately recognize the road dividing line from the image. The inflection point may be determined from FIG. 8, which will be described later.

[0054] The distance processing unit 128 estimates the reliability of the road-dividing line detected from the image based on whether the horizontal distance between a first element and a second element that are the same vertical distance away from the inflection point is equal to or less than a threshold. The distance processing unit 128 derives the horizontal distance between the first element (road-dividing line) and the second element (an object such as an exterior wall or curb) for each distance away from the inflection point, and estimates the reliability of the road-dividing line detected from the image based on the distance. The distance is the horizontal distance between a point on the first element and a point on the second element that are the same vertical distance. Note that the projection of the point sequence of the second element may be omitted, and the distance between the first element and the second element may be calculated based on the position of the feature of the second element that is the same vertical distance as the first element. The distance processing unit 128 calculates the horizontal distance d using equation (4).

[0055]

number

[0056] 6 is a diagram showing changes in horizontal distance for each vertical distance. A threshold value is set in advance for the distance. Distance processing unit 128 determines that an area where the horizontal distance is equal to or greater than the threshold value is a highly reliable area, and an area where the horizontal distance is less than the threshold value is a low reliable area. The threshold value may be set, for example, for each type (pattern) of transition of distance change, or may be set according to the curvature of a curved road obtained from map information.

[0057] As described above, the determination processing unit 120 can estimate (or determine) that the reliability of detected road-dividing lines in the area after the inflection point and where the horizontal distance is equal to or greater than the threshold is high. Figure 7 shows the error between the actually measured position of the road-dividing line and the position of the road-dividing line obtained by image processing. The error increases as the vertical distance increases, but the error is small up to the area where the horizontal distance is equal to or greater than the threshold. Therefore, it can be confirmed that the reliability of the detection result of the road-dividing line up to the area where the horizontal distance is equal to or greater than the threshold is high.

[0058] 8 is a diagram showing another example of an inflection point. The angle processing unit 126 may obtain an inflection point where the transition of the result of quadratically differentiating the transition of the angle θ with respect to the distance crosses zero. The determination processing unit 120 may perform the above-described processing using an inflection point IP obtained as a result of the quadratically differentiated operation.

[0059] Furthermore, the determination processing unit 120 may calculate a value d1 that maximizes the difference between the two largest angles (the difference between a positive angle and a negative angle) before and after the inflection point IP, and if this value d1 is equal to or greater than a threshold, may determine that the road-dividing line in the image is a road-dividing line on a curved road. If the road-dividing line in the image is a road-dividing line on a curved road, the determination processing unit 120 may perform a process of estimating the reliability of the road-dividing line detected from the image using the lateral distance.

[0060] [flowchart] 9 is a flowchart showing an example of the flow of processing executed by the driving assistance device 100. First, the determination processing unit 120 acquires an image (step S100). Next, the determination processing unit 120 projects road dividing lines and exterior walls two-dimensionally based on the image (step S102). Next, the determination processing unit 120 sets a sequence of points for the projected road dividing lines and exterior walls (step S104). Note that the processing of step S102 may be omitted and the sequence of points may be set directly from the features of the image.

[0061] Next, the determination processing unit 120 derives the angle θ between the two sequences of points on the road-dividing line (step S106). Next, the determination processing unit 120 derives the lateral distance between the exterior wall and the road-dividing line (step S108). Next, the determination processing unit 120 identifies an area with low reliability based on the angle θ derived in step S106, a threshold value, and the lateral distance (step S110). This completes the processing of one routine of this flowchart.

[0062] Here, the angle θ, lateral distance, and error of road-dividing lines on non-curved roads will be explained. FIG. 10 is a diagram showing an example of image IM2 in which a road that is not curved is captured. FIG. 11 is a diagram showing changes in angle θ in image IM2. When the road is not curved, no inflection point is present. FIG. 12 is a diagram showing changes in lateral distance in image IM2. When the road is not curved, no threshold is set and the reliability of all areas is estimated to be high. A threshold may be set at a position farther away than the threshold for lateral distance on curved roads. In other words, when there is no inflection point, the determination processing unit 120 estimates that the reliability of road-dividing lines detected from images of distant areas is higher than when there is an inflection point.

[0063] Figure 13 shows the error between the position of road-dividing lines on non-curved roads and the measured position of road-dividing lines. The longer the vertical distance, the larger the error, but even with longer vertical distances, the error remains small. This confirms that the reliability of the road-dividing line detection results is high for all areas, or for areas with a vertical distance longer than that of curved roads.

[0064] [flowchart] 14 is a flowchart showing another example of the flow of the processing executed by the driving assistance device 100. The processing from step S100 to step S106, step S108, and step S110 is the same as the processing of the same step numbers in FIG.

[0065] After the process of step S106, the determination processing unit 120 determines whether the image is a target image for which reliability is to be calculated (step S108). The determination processing unit 120 makes this determination based on the change in the angle θ. Specifically, as described above, if (1) or both (1) and (2) are satisfied, the image is determined to be a target image, and the process proceeds to step S108. If not satisfied, the image is determined not to be a target image, and the process of one routine of this flowchart ends.

[0066] The above (1) indicates that an inflection point exists. The inflection point may be an inflection point in the transition of the change in the angle θ, or an inflection point in the transition of the result of quadratic differentiation. The above (2) indicates that the difference between the results of quadratic differentiation of the angle θ before and after the inflection point is greater than or equal to a threshold.

[0067] As described above, the determination processing unit 120 can easily determine whether or not a target is one for which reliability is to be calculated by utilizing a change in the angle θ.

[0068] Here, for example, in driving assistance control or autonomous driving control, the results of self-location estimation using map information or GNSS satellites are used to estimate the position of vehicle M on a road and control vehicle M. In this case, to correct an error in the position of vehicle M, information on road dividing lines obtained from an image captured by camera 10 and the shape characteristics of the road dividing lines in the map information are used to correct the error.

[0069] The information on road-dividing lines in the image may not accurately capture the shape of the road-dividing lines because distant features cannot be detected accurately due to obstructions such as side walls or the performance of the image sensor. This can lead to errors in matching the road-dividing line information obtained from the image captured by camera 10 with the shape features of the road-dividing lines in the map information.

[0070] In this embodiment, as described above, the driving assistance device 100 projects a sequence of points onto a two-dimensional space at a predetermined distance interval based on the first element, which is the road dividing line obtained from the image, and identifies an inflection point using the sequence of points.The driving assistance device 100 can accurately estimate the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element, which are the same vertical distance away from the inflection point, is below a threshold value.

[0071] [Utilizing reliability] The control unit (vehicle control unit) 150 may control the vehicle by referring to the reliability. The reliability is used in driving assistance, matching with map information, and autonomous driving. The driving assistance device 100, for example, suppresses driving assistance in areas with low reliability or suppresses the generation of a driving assistance plan in areas with low reliability. For example, it suppresses lane keeping control or lane change control in areas with low reliability. For example, it postpones the generation of a driving assistance plan in areas with low reliability. The driving assistance device 100 suppresses the matching process between road lane markings in areas with low reliability and road lane markings in map information or changes the content of the matching process. For example, in areas with low reliability, the driving assistance device 100 prioritizes the use of the positions of road lane markings in map information or performs matching processing assuming low reliability (by lowering the influence of information on road lane markings obtained from images in identifying the positions of road lane markings and identifying the vehicle's own location compared to when reliability is high). In this case, for example, the control unit 150 controls the traveling position of the vehicle M based on the results of the matching process. In this way, the control unit 150 may change the content of the control of the vehicle M by referring to the reliability.

[0072] The control unit 150 may change the level of control of the vehicle M by referring to the reliability. The control level refers to the degree of autonomous driving control or the degree of driving assistance control. For example, when the reliability is high, the control unit 150 increases the degree of autonomous driving control or the degree of driving assistance control compared to when the reliability is low. The degree of autonomous driving control or the degree of driving assistance control refers to the degree to which the control unit 150 controls the vehicle M by itself. A high degree of autonomous driving control or the degree of driving assistance control means that the control unit 150 controls the vehicle M by itself to a high degree, and a low degree of autonomous driving control or the degree of driving assistance control means that the control unit 150 controls the vehicle M by itself to a low degree and the driver controls the vehicle M by operating it to a high degree. For example, if the vehicle M is capable of autonomous driving, the autonomous driving level may be lowered in a region where the reliability is low. For example, in a region where the reliability is low, the driver may be required to perform obligations such as forward monitoring or hands-on driving, and these obligations may be lifted in other regions. Furthermore, the generation of an action plan for autonomous driving in an area where the reliability is low may be suppressed. The same applies to driving assistance control.

[0073] According to the embodiment described above, the judgment processing unit 120 can more accurately estimate the reliability of road dividing lines detected from an image based on whether the horizontal distance between a first element and a second element that are the same vertical distance away from the inflection point is less than a threshold value.

[0074] The above-described embodiment can be expressed as follows. a storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, detecting a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the surroundings of the vehicle, and projecting a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance; Identifying an inflection point using the sequence of points; estimating the reliability of the road dividing line detected from the image based on whether the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is equal to or less than a threshold value; The control device is configured as follows.

[0075] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0076] 1 Vehicle System 10 Camera 80 Controls 100 Driving assistance device 110 Recognition part 120 Judgment processing unit 122 First conversion processing unit 124 Second conversion processing section 126 Angle processing section 128 Distance processing section 150 control section

Claims

1. a first processing unit that detects a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the periphery of the vehicle, and projects a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to an actual distance; Identifying an inflection point using the sequence of points; a second processing unit that estimates an area where the horizontal distance between the first element and the second element that are the same vertical distance away from the inflection point is less than a threshold as an area where the reliability of the road-dividing line detected from the image is low, The second processing unit is deriving each angle between adjacent points in the sequence, and determining the inflection point where the transition of the angle for each distance crosses zero, or the inflection point where the transition of the result of quadratically differentiating the transition of the angle with respect to the distance crosses zero; Control device.

2. the second processing unit estimates the reliability of the road-dividing line detected from the image when a difference between the two largest angles before and after the inflection point is equal to or greater than a threshold. The control device according to claim 1 .

3. the second processing unit estimates that when the inflection point is not present, the reliability of the road-dividing line detected from the image in a distant area is higher than when the inflection point is present. The control device according to claim 1 or 2.

4. a vehicle control unit that controls the vehicle by referring to the reliability; The control device according to claim 1 or 2.

5. a vehicle control unit that changes a level of control of the vehicle by referring to the reliability; The control device according to claim 1 or 2.

6. The computer a process of detecting a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the periphery of the vehicle, and projecting a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance; A process of identifying an inflection point using the sequence of points; and estimating an area where the horizontal distance between the first element and the second element that are the same in vertical direction and farther from the inflection point is less than a threshold as an area where the reliability of the road-dividing line detected from the image is low; deriving each angle between adjacent points in the sequence, and determining the inflection point where the transition of the angle for each distance crosses zero, or the inflection point where the transition of the result of quadratically differentiating the transition of the angle with respect to the distance crosses zero; Control method.

7. On the computer, a process of detecting a first element corresponding to a road dividing line and a second element that is an element outside the road dividing line from an image of the periphery of the vehicle, and projecting a sequence of points corresponding to the first element onto a two-dimensional space at a predetermined distance interval according to the actual distance; A process of identifying an inflection point using the sequence of points; and estimating an area where the horizontal distance between the first element and the second element, which are at the same vertical distance from the inflection point, is less than a threshold value as an area where the reliability of the road-dividing line detected from the image is low, A program for executing a process of deriving each angle between adjacent sequences of points and finding the inflection point where the change in angle over distance crosses zero, or the inflection point where the change in angle obtained by quadratically differentiating the change in angle with respect to distance crosses zero.

Images