Vehicle control device, vehicle control method, and storage medium

Abstract

A vehicle control device, a vehicle control method, and a storage medium that can more accurately recognize the position of an object included in an image to perform driving control of the vehicle and can further improve the safety of traffic. The vehicle control device includes: an imaging section that captures the surroundings of the vehicle; a recognition section that recognizes the situation of the surroundings of the vehicle; a driving control section that controls one or both of the speed and the steering of the vehicle based on the recognition result of the recognition section; and a control section that controls the driving control section based on the captured content of the imaging section, determines the lane to which an object belongs based on the positional relationship between the left-right dividing line and the left and right ends of the object present on a two-dimensional image captured by the imaging section, and determines the lane to which the object belongs based on the positional relationship between the end of the object and the recognized dividing line in the case where one of the left-right dividing lines is not recognized, the left-right dividing line dividing the lane in which the vehicle is traveling, and the object being present in the surroundings of the vehicle.

Description

[0001] This application claims priority based on Japanese Patent Application No. 2021-167279, filed on October 12, 2021, the contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to vehicle control devices, vehicle control methods, and storage media. Background Technology

[0003] In recent years, research has been progressing on automatically controlling the movement of vehicles transporting passengers based on the recognition results of surrounding conditions, with the aim of providing sustainable transportation systems that improve traffic safety and convenience. Relatedly, in the field of vehicle surrounding recognition, techniques are known that involve extracting feature points from images of the road surface taken by multiple cameras, transforming the coordinates of the extracted feature points to a shared overhead coordinate system of the multiple cameras, and determining the type of dividing line based on the state transition of the feature points extracted from images taken by the cameras deemed usable (e.g., International Publication No. 2018 / 109999). Summary of the Invention

[0004] However, sometimes when a portion of the dividing line cannot be identified, the position of objects around the vehicle cannot be correctly identified, thus failing to improve traffic safety in driving control based on the identification results.

[0005] The present invention was made in consideration of such circumstances, and one of its objectives is to provide a vehicle control device, vehicle control method and storage medium that can more accurately identify the position of objects contained in an image to perform driving control of the vehicle and further improve traffic safety.

[0006] The vehicle control device, vehicle control method, and storage medium of the present invention adopt the following structure.

[0007] (1): A vehicle control device according to one aspect of the present invention includes: a camera unit that captures images of the vehicle's surroundings; an identification unit that identifies the surrounding conditions of the vehicle; a driving control unit that controls one or both of the vehicle's speed and steering based on the identification result of the identification unit; and a control unit that controls the driving control unit based on the content captured by the camera unit. The control unit determines the lane to which the object belongs based on the positional relationship between a left-right dividing line and the left and right ends of an object in a two-dimensional image captured by the camera unit. If neither of the left-right dividing lines is identified, the control unit determines the lane to which the object belongs based on the positional relationship between the end of the object and the identified dividing line. The left-right dividing line is the dividing line that divides the lane in which the vehicle travels, and the object is an object existing in the surroundings of the vehicle.

[0008] (2): In the above (1) scheme, if the control unit does not identify one of the left and right dividing lines in the two-dimensional image and the ends of the object are respectively present on the left and right sides relative to the identified dividing line, it determines that the object belongs to at least the current lane.

[0009] (3): In the above (1) scheme, if the control unit does not identify one of the left and right dividing lines in the two-dimensional image, and the object has left and right ends on either the left or right side relative to the identified dividing line, it determines that the object does not belong to the lane on the side where the ends do not exist relative to the dividing line.

[0010] (4): In the above (1) scheme, when the control unit does not identify one of the left and right dividing lines but identifies a part of the dividing line of the other, it determines the lane to which the object belongs based on the positional relationship between any end of the left or right end of the object and the identified dividing line that exists in the lateral direction of the end on the two-dimensional image.

[0011] (5): In the above (1) scheme, when both the left and right dividing lines are partially identified, the control unit determines the lane to which the object belongs based on the positional relationship between the left and right ends of the object and the identified dividing lines that exist horizontally on the ends of the two-dimensional image.

[0012] (6): In the above (1) scheme, the control unit corrects the position of the object identified by the identification unit based on information related to the lane to which the object belongs, and controls the driving control unit based on the corrected position of the object.

[0013] (7): A vehicle control method according to one aspect of the present invention causes a computer to perform the following processing: using a camera to capture images of the vehicle's surroundings; identifying the surrounding conditions of the vehicle; performing driving control based on the identification results to control one or both of the vehicle's speed and steering; controlling the driving control based on the content captured by the camera; determining the lane to which the object belongs based on the positional relationship between the left and right dividing lines and the left and right ends of the object in the two-dimensional image captured by the camera; and determining the lane to which the object belongs based on the positional relationship between the end of the object and the identified dividing line if neither of the left and right dividing lines is identified, wherein the left and right dividing lines are dividing lines that divide the lane in which the vehicle travels, and the object is an object existing in the surroundings of the vehicle.

[0014] (8): In one aspect of the present invention, the storage medium is a non-temporary storage medium that stores a program that can be read by a computer, wherein the program causes the computer to perform the following processing: using a camera to capture images of the vehicle's surroundings; identifying the surrounding conditions of the vehicle; performing driving control based on the identification results to control one or both of the vehicle's speed and steering; controlling the driving control based on the content captured by the camera; determining the lane to which the object belongs based on the positional relationship between the left and right dividing lines and the left and right ends of the object in the two-dimensional image captured by the camera; and determining the lane to which the object belongs based on the positional relationship between the end of the object and the identified dividing line if neither of the left and right dividing lines is identified, wherein the left and right dividing lines are dividing lines that divide the lane in which the vehicle is traveling, and the object is an object existing in the surroundings of the vehicle.

[0015] According to the above schemes (1) to (8), the position of the object contained in the image can be identified more accurately to control the driving of the vehicle, and traffic safety can be improved. Attached Figure Description

[0016] Figure 1 This is a diagram illustrating an example of the structure of a vehicle system equipped with a vehicle control device according to an embodiment.

[0017] Figure 2 This is a diagram illustrating an example of control performed by the driving control device.

[0018] Figure 3 This is a flowchart illustrating an example of the process performed by the driving control unit.

[0019] Figure 4 This is an example diagram showing the surrounding conditions of this vehicle.

[0020] Figure 5It is a diagram used to illustrate the area enclosed by the outlines of other vehicles contained in a two-dimensional image.

[0021] Figure 6 This is a diagram used to illustrate the extraction of dividing lines.

[0022] Figure 7 This is a diagram used to illustrate noise removal from the dividing lines.

[0023] Figure 8 This is a diagram used to illustrate the slope of the extracted dividing line.

[0024] Figure 9 This is a diagram used to illustrate how to obtain the left and right ends of other vehicles.

[0025] Figure 10 It is a diagram used to illustrate the ends of other vehicles by using segmented regions.

[0026] Figure 11 It is a diagram used to illustrate how to determine the lane to which other vehicles belong.

[0027] Figure 12 This is a flowchart illustrating an example of the process for determining and processing the lane to which a lane belongs.

[0028] Figure 13 It is a diagram used to illustrate multiple decision conditions.

[0029] Figure 14 This is a diagram used to illustrate the processing content of steps S212 to S214.

[0030] Figure 15 This diagram illustrates the process of determining if a lane is not in a designated lane.

[0031] Figure 16 This is an example of a diagram representing information that has been marked.

[0032] Figure 17 This is a diagram used to illustrate the first decision mode.

[0033] Figure 18 This is a diagram used to illustrate the second decision mode.

[0034] Figure 19 This is a diagram used to illustrate the third decision mode.

[0035] Figure 20 This is a diagram used to illustrate the fourth decision mode.

[0036] Figure 21 This is a diagram used to illustrate a second scenario where a portion of the dividing line between the two sides could not be identified.

[0037] Figure 22 This is a diagram used to illustrate the function of the target acquisition unit.

[0038] Figure 23 This is a diagram used to illustrate the setting of the control transfer ratio during the switching of the target vehicle. Detailed Implementation

[0039] Hereinafter, embodiments of the vehicle control device, vehicle control method, and storage medium of the present invention will be described with reference to the accompanying drawings. The vehicle control device of the embodiments is provided to be mounted on a vehicle. The vehicle is, for example, a four-wheeled vehicle, whose drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using electricity generated by a generator connected to the internal combustion engine, or electricity discharged from a secondary battery or fuel cell.

[0040] [Overall Structure]

[0041] Figure 1 This is a diagram illustrating an example of the structure of a vehicle system 1 equipped with a vehicle control device according to an embodiment. Figure 1 The vehicle system 1 shown includes, for example, a first camera 10, a radar device 12, a second camera 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a driving control device 80, a driving force output device 92, a braking device 94, a steering device 96, and a driving control device 100. These devices and equipment are interconnected via multiple communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, and wireless communication networks. Figure 1 The structure shown is only one example; a part of the structure may be omitted, or other structures may be added. The second camera 20 is an example of a "camera unit." The camera unit may also include the first camera 10. The second camera 20 and the driving control device 100 are an example of a "vehicle control device."

[0042] The first camera 10 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). One or more first cameras 10 are mounted at any location on the vehicle equipped with the vehicle system 1 (hereinafter referred to as the vehicle M). For example, when taking pictures of the front of the vehicle M, the first camera 10 is mounted on the upper part of the windshield, the back of the interior rearview mirror, etc. When taking pictures of the rear of the vehicle M, the first camera 10 is mounted on the upper part of the rear windshield, the tailgate, etc. When taking pictures of the sides and rear sides of the vehicle M, the first camera 10 is mounted on the rearview mirror on the door, etc. The first camera 10, for example, periodically and repeatedly takes pictures of the perimeter of the vehicle M. The first camera 10 can also be a stereo camera.

[0043] The first camera 10 also includes a fisheye camera capable of capturing wide-angle (e.g., 360-degree) images of the periphery of the vehicle M. The fisheye camera is mounted, for example, on the upper part of the vehicle M, and captures wide-angle images of the periphery of the vehicle M in the horizontal direction. The fisheye camera can also be implemented by combining multiple cameras (multiple cameras capturing 120-degree or 60-degree ranges in the horizontal direction).

[0044] Radar device 12 radiates millimeter-wave or other radio waves to the periphery of the vehicle M and detects the radio waves (reflected waves) reflected by surrounding objects to at least detect the position (distance and orientation) of the objects. One or more radar devices 12 may be installed at any location on the vehicle M. Radar device 12 may also detect the position and speed of surrounding objects using FM-CW (Frequency Modulated Continuous Wave) method.

[0045] The second camera 20 is, for example, a digital camera utilizing a solid-state imaging element such as a CCD or CMOS sensor. One or more second cameras 20 can be mounted at any location on the vehicle M. The second camera 20 can be positioned in the same location as the first camera 10, or in a portion of the location of the first camera 10 (e.g., a position for taking pictures of the front of the vehicle M). The second camera 20 can, for example, periodically and repeatedly take pictures of the perimeter of the vehicle M. The second camera 20 can also be a stereo camera.

[0046] The HMI 30 provides various information to the user of the vehicle M and accepts input operations performed by the user. Users include, for example, the driver of the vehicle M, passengers, and other occupants. Hereinafter, unless otherwise specified, they will be referred to as "occupants." The HMI 30, for example, includes a display unit and a speaker as an output unit for providing various information. The display unit displays images under the control of the HMI control unit 170 (described later), and the speaker outputs sound under the control of the HMI control unit 170. The HMI 30, as an input unit for receiving input from occupants, includes a touch panel, switches, buttons, a microphone, etc. Information received from the input unit is output to the HMI control unit 170.

[0047] The vehicle sensor 40 includes a vehicle speed sensor for detecting the speed of the vehicle M, an acceleration sensor (3-axis acceleration sensor) for detecting acceleration, a yaw rate sensor for detecting yaw rate (e.g., rotational angular velocity about a vertical axis passing through the center of gravity of the vehicle M), and an orientation sensor for detecting the orientation of the vehicle M. The vehicle sensor 40 may also include a position sensor for detecting the position of the vehicle M. The position sensor may be a sensor that obtains position information (longitude and latitude information) from a GPS (Global Positioning System) device. Alternatively, the position sensor may be a sensor that obtains position information using a GNSS (Global Navigation Satellite System) receiver. The GNSS receiver determines the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may also be determined or supplemented by an INS (Inertial Navigation System) utilizing the output of other sensors. The results detected by the vehicle sensor 40 are output to the driving control device 100.

[0048] The driving control unit 80 includes, for example, various control components such as a steering wheel, accelerator pedal, brake pedal, and gear shift lever for the driver to perform steering operations. Each control component of the driving control unit 80 is equipped with an operation detection unit that detects the amount of operation performed by the driver. The operation detection unit detects the amount of pressure applied to the accelerator and brake pedals, the position of the gear shift lever, the steering angle of the steering wheel, and the steering torque. Furthermore, the operation detection unit outputs a detection signal indicating the detection result to one or both of the following: the driving control unit 100, the driving force output device 92, the braking device 94, and the steering device 96.

[0049] The driving force output device 92 outputs the driving force (torque) used for driving the vehicle M to the drive wheels. The driving force output device 92 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as a power ECU (Electronic Control Unit) that controls them. The power ECU controls the above structure according to information input from the driving control device 100 or from the driving operation device 80.

[0050] The braking device 94 includes, for example, a brake caliper, a hydraulic cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the hydraulic cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the driving control device 100 or from the driving operation device 80, so that braking torque corresponding to the braking operation is output to each wheel. The braking device 94 may have a backup mechanism for transmitting hydraulic pressure generated by the operation of the brake pedal included in the driving operation device 80 via the master hydraulic cylinder to the hydraulic cylinder. The braking device 94 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that controls the actuator according to information input from the driving control device 100 and transmits hydraulic pressure from the master hydraulic cylinder to the hydraulic cylinder.

[0051] The steering system 96 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies force to a rack and pinion mechanism to change the direction of the steering wheels. The steering ECU drives the electric motor to change the direction of the steering wheels according to information input from the driving control unit 100 or from the driving operation unit 80.

[0052] [Structure of the driving control device 100]

[0053] The driving control device 100 includes, for example, an identification unit 120, a control unit 140, a driving control unit 160, an HMI control unit 170, and a storage unit 180. The identification unit 120, control unit 140, driving control unit 160, and HMI control unit 170 are each implemented by executing programs (software) using hardware processors such as CPUs (Central Processing Units). Some or all of these components can also be implemented using hardware (including circuitry) such as LSIs (Large Scale Integration), ASICs (Application Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and GPUs (Graphics Processing Units), or through the coordinated use of software and hardware. The aforementioned program can be pre-stored in a storage device such as an HDD or flash memory (a storage device with a non-temporary storage medium) of the driving control device 100, or it can be stored in a removable storage medium such as a DVD, CD-ROM, or memory card, and installed in the storage device of the driving control device 100 by assembling the storage medium (non-temporary storage medium) into the drive unit, card slot, etc.

[0054] The storage unit 180 may also be implemented using the various storage devices described above, or EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory), etc. The storage unit 180 stores, for example, information, programs, and other various types of information required for executing various controls in the implementation method.

[0055] [Identification Department]

[0056] The identification unit 120 includes, for example, a perimeter identification unit 122. In the perimeter identification unit 122, for example, sensor fusion processing is performed on the detection results detected by one or both of the first camera 10 and the radar device 12 to identify the surrounding conditions of the vehicle M. In this case, the perimeter identification unit 122 performs coordinate transformation from the image obtained from the first camera 10 to a top-view coordinate system, and performs image recognition processing based on the transformed coordinate system using well-known methods (binarization, contour extraction, image emphasis processing, feature extraction, pattern matching, etc.) to identify the surrounding conditions of the vehicle M. The perimeter identification unit 122 can also identify the position, type, speed, etc., of objects surrounding the vehicle M. Objects include, for example, other vehicles (e.g., vehicles located within a specified distance of the vehicle M). Objects may also include pedestrians, bicycles, road structures, etc. Road structures include, for example, road signs, traffic signals, curbs, median strips, guardrails, fences, walls, intersections, etc. Objects may also include obstacles that hinder the movement of the vehicle M.

[0057] The peripheral recognition unit 122 recognizes the position (relative position), speed, acceleration, and other states of the vehicle M and objects existing around the vehicle M. The position of an object is recognized, for example, as the position of an absolute coordinate system (hereinafter referred to as the vehicle coordinate system) with a representative point of the vehicle M (center of gravity, drive shaft center, etc.) as the origin, and is used for control. The position of an object can be represented by representative points such as the object's center of gravity, corners, or the front end in the direction of travel, or by the area it represents. Speed ​​includes, for example, the speed of the vehicle M and other vehicles relative to the direction of travel (longitudinal) of the lane (hereinafter referred to as longitudinal speed), and the lateral speed of the vehicle M and other vehicles relative to the lane (hereinafter referred to as lateral speed). The "state" of an object can also include, for example, the object's acceleration, jerk, or "action state" (e.g., whether a lane change is in progress or about to be performed) if the object is a moving body such as another vehicle.

[0058] The peripheral recognition unit 122 identifies road markings (hereinafter referred to as markings) that exist around the vehicle M. For example, the peripheral recognition unit 122 identifies the left and right markings that divide the lane in which the vehicle M travels (hereinafter referred to as the lane). Based on the left and right markings, the peripheral recognition unit 122 distinguishes and identifies the adjacent lane on the left side of the lane (hereinafter referred to as the left lane) and the adjacent lane on the right side of the lane (hereinafter referred to as the right lane).

[0059] [Control Department]

[0060] The control unit 140 controls the entire driving control device 100. The control unit 140 includes, for example, a contour extraction unit 142, a dividing line extraction unit 144, an end point acquisition unit 146, a lane determination unit 148, and a target capture unit 150.

[0061] The contour extraction unit 142 extracts edge points from a two-dimensional image captured by the second camera 20 using existing image analysis processing, and extracts the contours of objects in the two-dimensional image from the extracted edge point sequence. A two-dimensional image refers to an image captured by the second camera 20 directly represented by a two-dimensional coordinate system in the vertical (X-axis direction) and horizontal (Y-axis direction). For example, the contour extraction unit 142 extracts contours by connecting edge points within a specified distance. The contour extraction unit 142 can also obtain color information for each edge point through image analysis processing of the two-dimensional image and extract contours by connecting edge points with similar colors. The contour extraction unit 142 can also extract contours of only other vehicles (an example of an object) based on the shape, size, and position of the contour. When multiple objects exist in the two-dimensional image, the contour extraction unit 142 extracts each object separately.

[0062] The lane divider extraction unit 144 extracts the lane dividers contained in the two-dimensional image captured by the second camera 20. The lane divider extraction unit 144 can also distinguish and extract the left and right lane dividers of the lane. The lane divider extraction unit 144 obtains the position information (coordinate information) of the extracted lane dividers on the two-dimensional image.

[0063] The end-capturing unit 146 acquires the position of the end of the object contained in the two-dimensional image captured by the second camera 20 (position information on the two-dimensional image). The end of the object refers to, for example, the left and right ends on the two-dimensional image. Here, if the object is another vehicle, the end of the other vehicle refers to, for example, the left and right ground contact points of the other vehicle in the two-dimensional image (the parts where the other vehicle connects with the road). From now on, we will continue to describe the object using another vehicle as an example.

[0064] The lane determination unit 148 determines the lane to which other vehicles belong (the lane to which other vehicles are present is determined) based on the position information of the dividing lines extracted by the dividing line extraction unit 144 and the position information of the ends of other vehicles obtained by the end acquisition unit 146. The lane determination unit 148 may also determine the lane to which other vehicles do not belong (the lane to which other vehicles are not present is determined) instead of the lane to which other vehicles belong.

[0065] The target acquisition unit 150 captures the target vehicle for driving control by the driving control unit 160 based on the recognition result identified by the recognition unit 120 and the lane determination result of the lane determination unit 148 for other vehicles (relative lateral position information of other vehicles as observed from the perspective of the vehicle M). The target vehicle refers to, for example, the vehicle in front of the vehicle M following it at a predetermined inter-vehicle distance using driving control such as ACC (Adaptive Cruise Control) performed by the driving control unit 160. The target acquisition unit 150 can also extract other vehicles that interfere with the driving of the vehicle M from among other vehicles around the vehicle M. Other vehicles interfering with the driving of the vehicle M refer to, for example, other vehicles that require changes in driving control (speed control, steering control) of the vehicle M due to lane changes between the vehicle M and the vehicle M while the vehicle M is following the vehicle M, or other vehicles that may come into contact with the vehicle M and require avoidance driving control. For details regarding the functions of the outline extraction unit 142, the dividing line extraction unit 144, the end acquisition unit 146, the lane determination unit 148, and the target capture unit 150, please refer to the following description.

[0066] The driving control unit 160 controls one or both of the vehicle M's speed and steering based on the identification result of the identification unit 120 and the information from the control unit 140. For example, when performing a prescribed driving control, the driving control unit 160 generates a predetermined track (target track) for the vehicle M to travel in accordance with the content of the driving control, based on the identification result of the identification unit 120 and the information from the control unit 140. The predetermined track may include a speed element, for example. Furthermore, the driving control unit 160 controls the speed and steering of the vehicle M so that the vehicle M travels along the generated predetermined track. The driving control unit 160 includes, for example, a speed control unit 162 and a steering control unit 164. For example, when the speed control unit 162 receives the execution of ACC through the operation of the HMI 30 by the occupant, it controls the driving drive output device 92 or the braking device 94 to perform speed control such as acceleration or deceleration, so as to maintain a prescribed distance from the vehicle traveling in front in the same lane while traveling along the predetermined track. The speed control unit 162 controls the driving force output device 92 or the braking device 94, for example, based on the approach of other vehicles to the vehicle M, to perform speed control such as acceleration or deceleration, so as to prevent the vehicle M from contacting other vehicles.

[0067] For example, when driving control such as ACC or LKAS (LaneKeeping Assistance System) is performed by the HMI30 operated by the occupant, the steering control unit 164 controls the steering device 96 based on the lane recognition result and the position of the vehicle M in order to maintain driving in the lane based on a predetermined track. For example, when performing ALC (Auto Lane Changing) by the HMI30 operated by the occupant, the steering control unit 164 controls the steering device 96 to change lanes to the adjacent lane of the destination along a predetermined track based on the recognition result recognized by the recognition unit 120.

[0068] The processing of the speed control unit 162 and the steering control unit 164 is achieved, for example, through a combination of feedforward control and feedback control. As an example, the steering control unit 164 combines feedforward control corresponding to the curvature of the road in front of the vehicle M with feedback control based on deviation from a predetermined track (target track).

[0069] HMI control unit 170 provides prescribed information to the occupant via HMI 30. This prescribed information includes, for example, information related to the status of the vehicle M and information related to driving control. Information related to the status of the vehicle M includes, for example, the vehicle M's speed, engine speed, and gear position. Information related to driving control includes, for example, information controlled by control unit 140, whether driving controls such as ACC and ALC are being performed, information on manual driving, and information related to the status of the driving control being executed. The prescribed information may also include information unrelated to the driving control of the vehicle M, such as television programs or entries stored on storage media such as DVDs (e.g., movies).

[0070] For example, the HMI control unit 170 may also generate an image including the information specified above, and display the generated image on the display unit of the HMI 30. It may also generate a sound representing the specified information and output the generated sound from the speaker of the HMI 30. The HMI control unit 170 may also output the information received by the HMI 30 to the control unit 140, the driving control unit 160, etc.

[0071] [Control performed by the vehicle control unit]

[0072] The following describes the details of the control performed by the driving control device 100. Figure 2 This diagram illustrates an example of control performed by the driving control unit 100. Before correcting the identification results of the vehicle M's surroundings obtained through sensor fusion processing using information output from the first camera 10 and the radar device 12, the driving control unit 100 uses a two-dimensional image of the vehicle M's surroundings (especially in the direction of travel) captured by the second camera 20 to determine the lanes to which other vehicles belong, obtain the relative positional relationship between the dividing lines and other vehicles, etc.

[0073] Furthermore, the driving control device 100, based on the determination of the lane to which other vehicles belong, the relative positional relationship between the lane markings and other vehicles, and the object recognition results obtained through sensor fusion processing, can, for example, capture other vehicles (hereinafter referred to as target vehicles) that are being followed by the vehicle M during the execution of driving controls such as ACC. Based on information related to the captured target vehicles (e.g., the target vehicle's position, relative distance, relative speed, target vehicle spacing, and direction of travel), it generates the speed of the vehicle M and causes the vehicle M to travel at the generated speed. Thus, in this embodiment, by using a two-dimensional image captured by the second camera 20, the determination of the lane and the relative positional relationship are directly obtained while maintaining the coordinate system of the two-dimensional image without performing three-dimensional coordinate transformations for setting a top-down view. This suppresses errors and contradictions in the relative position between other vehicles and the vehicle's lane, and allows for the capture of target vehicles for driving control based on more accurate relative positions. The driving control device 100 can also perform steering control instead of speed control (or, further) on top of that.

[0074] [Processing Flow]

[0075] Next, the process flow performed by the driving control device 100 of the embodiment will be described. Figure 3 This is a flowchart illustrating an example of the process executed by the driving control device 100. The following description will primarily focus on the process executed by the driving control device 100 when the vehicle M performs ACC driving control. Figure 3 The process shown can be repeated during the execution of the above driving controls.

[0076] exist Figure 3 In this example, firstly, the identification unit 120 identifies the surrounding conditions, including the area in front of the vehicle M, by using sensor fusion processing of the outputs from the first camera 10 and the radar device 12 (step S100). The identification unit 120 then determines whether there are other vehicles in the vicinity of the vehicle M (step S120). Other vehicles are examples of objects, more specifically, other vehicles traveling in front of the vehicle M. The area in front of the vehicle M includes not only the area in its own lane, but also the areas in the left and right lanes.

[0077] Figure 4 This is a diagram illustrating an example of the surrounding conditions of vehicle M. In Figure 4 The example shows a three-lane road consisting of lane L1, the left lane L2 of lane L1, and the right lane L3. Figure 4 Assume that vehicle M (not shown) is traveling in lane L1 (hereinafter referred to as lane L1). Figure 4In the surrounding conditions shown, the identification unit 120 identifies other vehicles m1 that are in front of the vehicle M and are traveling in lane L1, and other vehicles m2 and m3 that are traveling in the left lane L2 as objects existing in front of the vehicle M.

[0078] The control unit 140 repeatedly (loops) executes steps S142 to S154 (step S140) for each other vehicle m1 to m3 identified by the recognition unit 120. The contour extraction unit 142 extracts the contours of the other vehicles contained in the two-dimensional image captured by the second camera 20 (step S142). Next, the contour extraction unit 142 sets the area that surrounds the contours of the other vehicles (step S144).

[0079] Figure 5 This is a diagram used to illustrate the area enclosed by the outlines of other vehicles contained in a two-dimensional image. Figure 5 In the example, in relation to Figure 4 The image IM10, captured by the second camera 20, is shown in relation to the surrounding conditions of the vehicle M. Image IM10 is a two-dimensional image directly represented by the image coordinate system (without undergoing three-dimensional coordinate transformation). Figure 5 The image IM10 shown is configured such that the contours of other vehicles m1 to m3 are extracted by the contour extraction unit 142. Figure 5 In the example, the contour extraction unit 142 generates three regions (hereinafter referred to as bounding boxes) BB1 to BB3 that enclose the contours of the other vehicles m1 to m3 respectively (step S144). The bounding boxes BB1 to BB3 are, for example, rectangular regions formed along the X-axis (vertical) and Y-axis (horizontal) of the image IM10. The contour extraction unit 142 extracts the coordinates (XY coordinates) of the four corners of each bounding box BB1 to BB3 in the two-dimensional coordinates of the image IM10.

[0080] Next, the dividing line extraction unit 144 extracts the point sequence of dividing lines that divide this lane L1 from the two-dimensional image captured by the second camera 20 (step S146). Figure 6 This is a diagram used to illustrate the extraction of dividing lines. Figure 6In the example, a portion of lanes L1 and L2 and other vehicles m2 traveling on lane L2 in the image IM10 captured by the second camera 20 is shown in magnification. The dividing line extraction unit 144, for example, scans the lower end of the bounding box BB2 of the other vehicle m2 in the horizontal direction (left-right direction, Y-axis direction) of the image IM10 and obtains the position (XY coordinate position) on the dividing lines LL and LR when they contact the left and right dividing lines LL and LR that divide lane L1. Furthermore, the dividing line extraction unit 144, using the lower end of the bounding box BB2 as a reference, scans in the vertical direction (X-axis direction, longitudinal direction) by a predetermined pixel (e.g., 1 pixel) each time, extracting the area of ​​the dividing lines LL and LR in the two-dimensional image within a predetermined range (hereinafter referred to as the specific region SP) that is, for example, 2n+1 pixels in the vertical direction. n is, for example, about 10 to 30, preferably 20, but can be appropriately changed according to the resolution of the two-dimensional image, the surrounding environment of the vehicle M, etc. The horizontal width (range in the left-right direction) of the specific region SP is comparable to the horizontal width of the image IM10. The dividing line extraction unit 144 extracts dividing lines LL and LR by connecting the coordinate point sequence obtained from the specific region SP.

[0081] return Figure 3 Next, the dividing line extraction unit 144 removes noise from the dividing lines LL and LR extracted in the specific region SP (step S148). Figure 7 This is a diagram used to illustrate noise removal of dividing lines. For example, the dividing line extraction unit 144 generates a rectangular diagram by counting the number of pixels projected onto the Y-axis within a specific region SP of the image IM10, where the pixels to be extracted as dividing lines are located. The rectangular diagram may, for example, be a diagram obtained by counting the total number of pixels along a predetermined width in the Y-axis direction. Furthermore, the dividing line extraction unit 144 extracts from the generated rectangular diagram intervals where the continuous interval in the Y-axis direction (horizontal direction) is a predetermined length or more, or where the number of pixels is a threshold or more, as intervals containing dividing lines. Therefore, it is possible to, for example, prevent noise removal such as… Figure 7 As shown, when noise such as trash in the image, such as NS1 and NS2, is identified as dividing lines, the dividing lines can be extracted more accurately.

[0082] return Figure 3 Next, the dividing line extraction unit 144 extracts the inclination of the dividing lines (step S150). The inclination of the dividing lines refers to, for example, the angle between the lower end of the two-dimensional image captured by the second camera 20 and the dividing lines contained in the two-dimensional image. For example, the dividing line extraction unit 144 derives the first principal component through principal component analysis processing of the dividing lines LL and LR in a specific region SP. Figure 8 This is a diagram used to illustrate the slope of the extracted dividing lines. In Figure 8In the example, the dividing line LL extracted from a specific area SP for other vehicles m2 is shown in a magnified manner. Figure 8 The example shows a group of pixels (a collection of pixel data) extracted as the dividing line LL.

[0083] The dividing line extraction unit 144 performs conventional principal component analysis on the pixel group extracted as the dividing line LL, and extracts, for example, the direction with the largest variance of the principal component score as the axis C1 of the first principal component. For example, the dividing line extraction unit 144 targets the pixel group extracted as the dividing line LL. Figure 8 The covariance matrix is ​​calculated by taking the coordinates of pixels corresponding to the dividing line LL within a specific region SP as shown in the figure, and the eigenvectors of the calculated covariance matrix are derived. The direction with the largest variance (eigenvalue) is extracted from the derived eigenvectors and used as the axis C1 of the first principal component.

[0084] The dividing line extraction unit 144 can also extract an axis orthogonal to the axis C1 of the first principal component as the axis C2 of the second principal component. Furthermore, the dividing line extraction unit 144 extracts the axis C1 of the first principal component relative to the Y-axis as the inclination of the dividing line LL. The inclination is represented, for example, by the angle θ1 formed by the Y-axis and the axis C1 of the first principal component. Similarly, the dividing line extraction unit 144 extracts the angle θ2 (not shown) formed by the Y-axis (more specifically, the -Y-axis) and the axis C1# (not shown) of the first principal component of the dividing line LR, which divides the lane L1, as the inclination of the dividing line LR using principal component analysis. As described above, the dividing line extraction unit 144 obtains the angle when viewing the dividing line of a specific region SP (an example of a defined region) from the left-right direction (Y-axis direction, lateral direction) of the two-dimensional image as the inclination of the dividing lines LL and LR, with the position of the dividing line where the lower end of the bounding box BB2 meets when extended laterally along the two-dimensional image as a reference.

[0085] return Figure 3 Next, the end-capturing unit 146 acquires the positions of the left and right ends of other vehicles present in the two-dimensional image (step S152). Next, the lane determination unit 148 determines the lane in which the other vehicle is traveling (the lane to which the other vehicle belongs) based on the positional relationship between the left and right ends acquired by the end-capturing unit 146 and the dividing line (step S154). Next, the target capture unit 150 captures other vehicles (target vehicles) that are the objects of the driving control performed by this vehicle M (step S160). Details regarding the processing of steps S152 to S154 and S160 described above will be described later.

[0086] Next, the driving control unit 160 performs driving control of the vehicle M (one or both of speed control and steering control) based on the surrounding conditions and the behavior of the detected target vehicle (step S170). In the process of step S120, if there are no other vehicles around the vehicle M, the driving control unit 160 performs driving control of the vehicle M based on the surrounding conditions (step S172). Thus, the processing of this flowchart ends.

[0087] [Obtaining the left and right ends of other vehicles (step S152)]

[0088] Next, the processing of step S152 will be explained in detail. Figure 9 This diagram illustrates how to obtain the left and right ends of other vehicles. Figure 9 In the example, it is shown that the left and right ends of other vehicles m2 are obtained, but the same process is performed on other vehicles (other vehicles m1, m3) contained in the two-dimensional image.

[0089] The end-capturing unit 146 scans along an inclined direction, for example, from the lower left and right ends of the image IM10 captured by the second camera 20, at a predetermined angle centered on the inner side of the image IM10, and captures the position where it meets the outline of another vehicle m2 as the end of the other vehicle m2. Alternatively, the end-capturing unit 146 may scan from the lower left and right ends of the bounding box BB2 surrounding the outline of the other vehicle m2, rather than from the lower left and right ends of the entire image IM10, to the position where there is a pixel corresponding to the other vehicle m2 (e.g., the outline pixel of the other vehicle m2), along the aforementioned inclined direction, and capture the position where it meets the outline of the other vehicle m2 as the end of the other vehicle. By scanning within the bounding box BB2, the processing load can be reduced compared to scanning the entire image IM10.

[0090] Here, as a predetermined angle for scanning (scanning angle), for example, the inclination θ1, θ2 of the dividing lines LR, LL is used. The inclination θ1, θ2 of the dividing lines LR, LL is the angle formed by the lower end of the image IM10 captured by the second camera 20 and the dividing lines contained in the image IM. For example, the end-capturing unit 146... Figure 9As shown, when scanning from the lower right end of bounding box BB2, the scan is performed at an angle θ1, centered on the inner side of the lower end of bounding box BB2 and moving upwards to form the dividing line LL. When scanning from the lower left end of bounding box BB2, the scan is performed at an angle θ2, centered on the inner side of the lower end of bounding box BB2 and moving upwards to form the dividing line LR. The end acquisition unit 146 scans while being offset at the same angle in the direction separating from the lower left and right ends, and acquires the positions that connect with the images of other vehicles m2 as the left end Le and the right end Re. The scanning can be performed, for example, as follows: Figure 9 As shown, scanning can be performed in the same direction at each end, or it can be performed by alternating between the same and reverse directions while shifting away from the lower end, or it can be performed in a zigzag pattern based on the same angle. For example, the left end Le is the leftmost point when viewing other vehicles m2 from above, and the right end Re is the rightmost point when viewing other vehicles m2 from above.

[0091] Typically, other vehicles travel along the extension direction of the lane dividers. Therefore, the tilt of other vehicles as observed from vehicle M is highly likely to depend on the tilt of the lane dividers. Thus, by scanning at an angle based on the tilt of the lane dividers (i.e., the lane divider of this vehicle) of other vehicles traveling in adjacent lanes, it is possible to more accurately obtain the left and right ends (grounding points) of other vehicles along the lane in a two-dimensional image relative to the tilt. In particular, for other vehicles changing lanes from adjacent lanes to this lane, since their bodies are facing this lane, it is easier to capture the grounding point of their front wheels.

[0092] Alternatively, the end-capping unit 146, while scanning at an angle defined as a predetermined angle, generates a segmented region that further divides the rectangular region of the bounding box BB2, and acquires the left and right ends by referring to the generated segmented regions in a predetermined order. The segmented region is a rectangular region containing one or more pixels.

[0093] Figure 10 This is a diagram used to illustrate the ends of other vehicles by using segmented regions. Figure 10 In the example, a scan is shown from the lower right end of the bounding box BB2 corresponding to other vehicles m2, but the same process is performed for scanning from the lower left end. The end-capturing unit 146 sets a segmented region SA within the bounding box BB2, and scans the image within the set region in a jagged pattern from the lower right end, based on the tilt direction of the aforementioned predetermined angle θ1. Figure 10In the example, the numbers within the segmented region SA indicate the scanning order set based on the tilt direction C1 of the dividing line LL. Furthermore, the end acquisition unit 146 determines whether each segmented region contains an image of another vehicle M, and acquires the coordinates (e.g., the coordinates of the center of the region) of the segmented region that initially contains an image of another vehicle as the coordinates of the right end of the other vehicle M. Figure 10 In the example, it is shown that the end of another vehicle M was detected in the 14th segment region from the start of the scan.

[0094] When determining whether an image of another vehicle M is included in each segmented region, the end-capturing unit 146 may not make the determination by keeping the original image unchanged, but instead use an image containing only the necessary information. For example, during the determination, image information with a reduced bit count, such as 8-bit, is used instead of full-color. This reduces the amount of data used in the determination process, thereby reducing the processing load.

[0095] When the angles θ1 and θ2 to be scanned (the angles of the dividing line relative to the left and right directions (Y-axis direction, horizontal direction) of the image IM10) exceed a threshold angle θth, the end-capping unit 146 sets a fixed angle. In this case, the fixed angle is, for example, approximately 40 to 50 degrees, and more preferably 45 degrees. Thus, by pre-setting an upper limit for the scanning angle, even when other vehicles are traveling on a curve, the left and right ends (grounding portions) can be accurately acquired based on the posture of other vehicles.

[0096] [Determination of the lane belonging to other vehicles (step S154)]

[0097] Next, the processing of step S154 will be explained in detail. The lane determination unit 148, for example, compares the relative positions of the dividing lines LL and LR of the current lane L1 as viewed from the dividing lines LL and LR in the two-dimensional image captured by the second camera 20, and the left and right contact points of other vehicles, and determines which lane (or not which lane) the other vehicle belongs to. The lane determination unit 148 can also determine whether other vehicles are entering the current lane. "Other vehicles are entering the current lane" means, for example, that at least one of the left and right ends (contact points) of the other vehicle is present in the current lane.

[0098] Figure 11 This is a diagram used to illustrate how to determine the lane belonging to other vehicles. Figure 11 The example illustrates the scenario of determining the lane to which other vehicle m2 belongs, but it does not address the issue of other surrounding vehicles ( Figure 4 The other vehicles shown (m1, m3) were also subjected to the same treatment. Figure 11In the example, it is assumed that the left and right grounding points (left grounding point Le, right grounding point Re) for other vehicles m2 are obtained by the end acquisition unit 146.

[0099] The lane determination unit 148 obtains the relative positions LD1 and RD1 of the right ground contact point Re when viewed from the left and right dividing lines LL and LR that divide this lane L1 on a two-dimensional image. The lane determination unit 148 also obtains the relative positions LD2 and RD2 of the left ground contact point Le when viewed from the dividing lines LL and LR. Regarding the positions that serve as references for the dividing lines LL and LR, for example, the position of the dividing line that has the shortest distance from the left and right ground contact points Le and Re to the area of ​​the dividing lines LL and LR included in a specific area SP (an area offset by n pixels in the vertical direction based on the same height as the end) based on the positions of the ground contact points Le and Re is obtained.

[0100] The lane determination unit 148 can also use an imaginary dividing line to obtain the reference position of the dividing line with the shortest distance from the above-mentioned grounding point when it cannot identify a part of the dividing line due to the presence of other vehicles m2 on the dividing line. This imaginary dividing line is obtained by linearly or non-linearly connecting the dividing lines that can be identified in front of and behind (in front and far) the part of the dividing line that cannot be identified in the extension direction of lane L1, according to the road shape or the shape of the dividing line of the other party that can be identified.

[0101] The lane determination unit 148 determines whether the relative position of the contact points Le and Re when viewed from the dividing lines LL and LR, respectively, is positive or negative. Relative position refers to, for example, relative to the two-dimensional image captured by the second camera 20, setting the right direction as positive and the left direction as negative from a reference point RP. The lane determination unit 148 can also reverse the positive and negative values ​​relative to the left and right directions. Alternatively, the lane determination unit 148 can also determine the positive or negative value based on the relative position of the dividing lines LL and LR when viewed from the contact points Le and Re. Figure 11 In the example, when the right direction is set to positive and the left direction to negative, the relative position of the right contact point Re when viewed from the dividing line LL is positive (+), and the relative position of the left contact point Le is negative (-). When viewed from the dividing line LR, the relative positions of both the left and right contact points Le and Re are negative (-). The lane determination unit 148 can also determine the relative position of the contact points when viewed from the dividing line if the size (e.g., length) of the relative positions LD1, RD1, LD2, and RD2 is greater than a predetermined length. Therefore, it is possible to suppress frequent changes in the positive and negative signs when other vehicles are traveling near the dividing line.

[0102] Next, the lane determination unit 148 determines the lane to which other vehicle m2 belongs based on the positional relationship (relative position) between the left and right contact points Le, Re and the left and right dividing lines LL, LR. Figure 12 This is a flowchart illustrating an example of the process for determining which lane a lane belongs to. Figure 12 In the example, firstly, the lane determination unit 148 determines whether the relative position of the right contact point Re when viewed from the left dividing line LL is negative (step S200). If the relative position of the right contact point Re when viewed from the left dividing line LL is negative, it determines that another vehicle m2 exists in the left lane (step S202). Alternatively, if the relative position of the right contact point Re when viewed from the left dividing line LL is not negative, the lane determination unit 148 determines whether the relative position of the left contact point Le when viewed from the right dividing line LR is positive (step S204). If the relative position of the left contact point Le when viewed from the right dividing line LR is positive, the lane determination unit 148 determines that another vehicle m2 exists in the right lane (step S206).

[0103] If it is determined that the relative position of the left ground point Le when viewed from the right dividing line LR is not upright, the lane determination unit 148 uses the left and right ground points Le and Re and the left and right dividing lines LL and LR to perform condition determination processing for the lane to which other vehicles belong (step S208). In the condition determination processing of step S208, the lane determination unit 148 determines whether multiple preset determination conditions are met. If the conditions are met, it determines that other vehicle m2 belongs to the lane corresponding to each condition. Figure 13 This is a diagram used to illustrate multiple decision conditions. In Figure 13 In the example, a correspondence was established between condition categories 1-8, condition content, and the lanes to which the conditions are met. The types and content of condition categories are not limited to this.

[0104] exist Figure 13In the example, when the relative position of the left contact point Le when viewed from the left dividing line LL is positive and the relative position of the left contact point Le when viewed from the right dividing line LR is negative (condition category 1 is met), the lane determination unit 148 determines that other vehicle m2 belongs to this lane. When the relative position of the right contact point Re when viewed from the left dividing line LL is positive and the relative position of the right contact point Re when viewed from the right dividing line LR is negative (condition category 2 is met), the lane determination unit 148 determines that other vehicle m2 belongs to this lane. When the relative position of the left contact point Le when viewed from the left dividing line LL is negative (condition category 3 is met), the lane determination unit 148 determines that other vehicle m2 belongs to the left lane. When the relative position of the right contact point Re when viewed from the right dividing line LR is positive (condition category 4 is met), the lane determination unit 148 determines that other vehicle m2 belongs to the right lane.

[0105] When the relative position of the left contact point Le when viewed from the left dividing line LL is negative and the relative position of the right contact point Re when viewed from the left dividing line LL is positive (condition category 5 is met), the lane determination unit 148 determines that other vehicle m2 belongs to both the left lane and its own lane. When the relative position of the left contact point Le when viewed from the right dividing line LR is negative and the relative position of the right contact point Re when viewed from the right dividing line LR is positive (condition category 6 is met), the lane determination unit 148 determines that other vehicle m2 belongs to both the right lane and its own lane. Furthermore, when the relative position of the left contact point Le when viewed from the left dividing line LL is positive and the relative position of the right contact point Re when viewed from the right dividing line LR is negative (condition category 7 is met), the lane determination unit 148 determines that other vehicle m2 belongs to its own lane. If the relative position of the right ground point Re when viewed from the left dividing line LL is positive and the relative position of the left ground point Le when viewed from the right dividing line LR is negative (the condition of condition category 8 is met), the lane determination unit 148 determines that other vehicle m2 belongs to this lane.

[0106] After the processing in step S208 is completed, the lane determination unit 148 merges the determination results obtained based on each condition (merge: OR processing, logical "OR" operation) (step S210), and uses the merged result to determine whether the other vehicle m2 belongs to both the right lane and the left lane (step S212). If it is determined that it belongs to both lanes, the lane determination unit 148 determines that the other vehicle m2 belongs to the left lane, its own lane, and the right lane (step S214). Thus, this flowchart ends.

[0107] Figure 14 This is a diagram used to illustrate the processing steps S212 to S214. Figure 14 In the example, the conditions in condition category 3 and condition category 4 are met, and a presence determination flag "1" is set for the lane to which the vehicle belongs. For example, if another vehicle sets a presence determination flag for both left and right lanes based on the conditions in condition category 3 and condition category 4, the lane determination unit 148 sets a presence determination flag of "1" for this lane as if the other vehicle also exists in this lane. Thus, the lane determination unit 148 can identify that the other vehicle belongs to (exists in) this lane and the lanes to its left and right. Regarding the flag, instead of the "1" flag, text, markings, or other symbols such as "Yes" or "○" can also be used.

[0108] The lane determination unit 148 determines the lane to which other vehicle m2 belongs (there is a defined lane), but it can also determine a lane that it does not belong to (not in a defined lane) instead (or on the basis of this). In this case, the lane determination unit 148 makes the determination based on predetermined determination conditions for determining that the vehicle is not in a defined lane. Figure 15 This diagram illustrates the process of determining if a lane is not in a designated lane. Figure 15 In the example, a correspondence was established between the condition categories A to D used to determine if a lane is not in a specific lane, the condition content, and the condition being outside the lane. The types and content of the condition categories are not limited to this.

[0109] exist Figure 15 In the example, when the relative position of the right ground point Re when viewed from the left dividing line LL is negative (condition A is met), the lane determination unit 148 determines that the other vehicle m2 is not in (does not belong to) the current lane or the right lane. When the relative position of the left ground point Le when viewed from the right dividing line LR is positive (condition B is met), the lane determination unit 148 determines that the other vehicle m2 is not in the current lane or the left lane. When the relative position of the left ground point Le when viewed from the left dividing line LL is positive (condition C is met), the lane determination unit 148 determines that the other vehicle m2 is not in the left lane. When the relative position of the right ground point Re when viewed from the right dividing line LR is negative (condition D is met), the lane determination unit 148 determines that the other vehicle m2 is not in the right lane. The lane determination unit 148 combines the determination results obtained based on conditions A to D to determine which lane the other vehicle is not in.

[0110] The lane determination unit 148 can also generate information with a flag set for each other vehicle relative to its own lane, left lane, and right lane, based on the determination result of the lane to which the other vehicle m2 belongs. The flag indicates whether the vehicle is in a defined lane or not in a defined lane (not in a lane). Figure 16 This is an example of a diagram representing information that has been marked. Figure 16 The information shown is an example of the relative lateral position information of other vehicles when viewed from vehicle M, and is an example of the relative position information between other vehicles and vehicle M.

[0111] exist Figure 16 In the example, the positional relationship between other vehicles m2 and the lane dividing lines LL and LR on a two-dimensional image (planar image) is shown. Figure 11 The example of a flag set in the context of the relationship shown. Figure 16 In the example, the left lane, the current lane, and the right lane are marked with a sign "1" for lanes where the presence of another vehicle m2 is confirmed, and lanes where it is confirmed that no other vehicle m2 is present. According to... Figure 16 For example, through the processing of the lane determination unit 148, it can be determined that other vehicle m2 exists in the left lane and the current lane, but not in the right lane. For example, even if it is only known that the vehicle does not exist in the current lane, it can be used in subsequent driving controls such as follow control, including target vehicle selection and speed control. Figure 16 In the example, different flags can be set in the presence of a definite flag and in the absence of a definite flag.

[0112] The lane determination unit 148 can determine, for example, whether or not it can identify one or a portion of the left and right dividing lines LL and LR that divide its lane from the two-dimensional image captured by the second camera 20, the lane belonging to other vehicles (there is a defined lane) and the lane not belonging to other vehicles (not in a defined lane). The inability to identify the dividing lines means, for example, that the edges of the dividing lines cannot be extracted from the two-dimensional image. Several lane determination modes for situations where the dividing lines cannot be identified will be explained below.

[0113] <First Judgment Mode>

[0114] Figure 17 This diagram illustrates the first determination mode. The first determination mode determines the lane (the lane to which the vehicle belongs) by determining the presence of another vehicle m2 when the right dividing line LR (LL or LR) of the lane dividing L1 cannot be detected. In the first determination mode, the lane determination unit 148 determines the lane based on the relative positions of the right ground contact point Re and the left ground contact point Le when viewed from the left dividing line LL. Figure 17In the example, the relative position of the right grounding point Re when viewed from the left dividing line LL is positive, and the relative position of the left grounding point Le when viewed from the left dividing line LL is negative. Therefore, it is known that grounding points Re and Le exist across the dividing line LL (grounding points (ends) exist on the left and right sides of the left dividing line LL respectively). Therefore, in the first determination mode, the lane determination unit 148 cannot identify whether other vehicle m2 belongs to the right lane, but determines that it belongs to at least the current lane L1 (more specifically, to both the current lane L1 and the left lane).

[0115] <Second Judgment Mode>

[0116] Figure 18 This diagram illustrates the second determination mode. The second determination mode determines that a vehicle m2 is not in a defined lane if the left dividing line LL of the dividing lines LL and LR that divide lane L1 cannot be identified. In the second determination mode, the lane determination unit 148 determines that the vehicle is not in a defined lane based on the relative positions of the right contact point Re and the left contact point Le when viewed from the right dividing line LR. Figure 18 In the example, the relative positions of the right ground contact point Re and the left ground contact point Le when viewed from the right dividing line LR are both negative. Therefore, in the second determination mode, regarding other vehicle m2, the lane determination unit 148 determines that other vehicle m2 does not belong to the right lane (the right lane is not a determined lane).

[0117] <Third Judgment Mode>

[0118] Figure 19 This diagram illustrates the third determination mode. The third determination mode determines the presence of other vehicles m2 when it is impossible to identify all of one side of the dividing line and a portion of the other side, thus determining the lane and the party or both vehicles not in the determined lane. Figure 19 In the example shown, a scenario is presented where only a portion of the left dividing line LL is identified, in addition to the right dividing line LR not being identified. Regarding the left dividing line LL, the dividing line LL is identified within a specific area based on the horizontal direction (horizontal direction of the image) of the right ground point Re, but the dividing line LL within a specific area based on the horizontal direction of the left ground point Le is not identified. Therefore, the relative position between the dividing line LL and the left ground point Le cannot be obtained. In this case, the lane determination unit 148 determines the lane of the other vehicle m2 solely based on the relative position of the right ground point Re when viewed from the dividing line LL.

[0119] exist Figure 19In the example, when viewed from the dividing line LL, the relative position of the right landing point Re of the other vehicle m2 is negative. Therefore, the lane determination unit 148 can determine that the other vehicle m2 belongs to the left lane (the left lane is the lane where the presence of the other vehicle m2 is determined). The lane determination unit 148 can also determine that the other vehicle m2 does not belong to this lane L1 or the right lane (this lane L1 and the right lane are the lanes where the other vehicle m2 is not determined).

[0120] <Fourth Judgment Mode>

[0121] Figure 20 This diagram illustrates the fourth decision mode. The fourth decision mode determines the lane by determining the presence of other vehicles (m2) in the first scenario where a portion of the dividing lines LL and LR cannot be identified. Figure 20 In the example, the right dividing line LR of the dividing lines LL and LR can be identified in the Y-axis direction (left-right direction in a two-dimensional image) at the right ground contact point Re of the vehicle M, while only the left dividing line LL of the dividing lines LL and LR can be identified in the Y-axis direction at the left ground contact point Le. In this case, since the main body of the vehicle M exists at the left and right ground contact points Le and Re, the lane determination unit 148 can identify the right dividing line LL of the dividing lines LL and LR. Figure 20 In the first scenario shown, it is determined that other vehicle m2 belongs to lane L1 (lane L1 is the lane where the presence of other vehicle m2 is determined).

[0122] Figure 21 This diagram illustrates a second scenario where a portion of the lane divider lines on both sides cannot be identified. In this second scenario of the fourth determination mode, only the dividing line LL of the lane divider lines LL and LR can be identified along the Y-axis at the right contact point Re of the vehicle M, and only the dividing line LR of the lane divider lines LL and LR can be identified along the Y-axis at the left contact point Le. In this case, since the main body of the vehicle M exists at both the left and right contact points Le and Re, the lane determination unit 148 also... Figure 21 In the scenario shown, it can be determined that other vehicle m2 belongs to lane L1. As with the fourth determination mode mentioned above, even when no part of the lane is identified, it is possible to determine whether other vehicles are in the lane or not by using the determination of the relative position of the identifiable part.

[0123] Based on the first to fourth determination modes described above, even when one or a portion of the dividing line cannot be identified due to poor weather or other reasons (edge ​​extraction is not possible), it is still possible to determine whether other vehicles belong to one or both of the lanes they belong to (a defined lane exists) and the lanes they do not belong to (not in a defined lane). This lane assignment process further improves the accuracy of determining the lane a vehicle belongs to.

[0124] [Target acquisition (step S160)]

[0125] Next, the details of the target acquisition process will be explained. Figure 22 This diagram illustrates the function of the target acquisition unit 150. The target acquisition unit 150 includes, for example, an object filter 152, a controlled object target acquisition unit 154, an interference probability target extraction unit 156, and a verification unit 158.

[0126] The object filter 152, for example, extracts other vehicles (an example of object objects) existing in the three lanes consisting of the vehicle M, the left lane, and the right lane, from the objects surrounding the vehicle M, based on object information obtained through sensor fusion processing by the recognition unit 120. For example, the object filter 152 transforms the image captured by the first camera 10 to a top-view coordinate system (three-dimensional coordinate transformation) and extracts other vehicles existing in the three lanes based on the position and shape of objects in the image.

[0127] When the vehicle M performs driving control such as ACC, the target acquisition unit 154 captures other vehicles that the vehicle M is following, extracted by the object filter 152, as target vehicles (following target vehicles). Following vehicles refer to, for example, vehicles that are present or may be present in the future on a predetermined track where the vehicle M is traveling. Additionally, when the vehicle M changes lanes, the target acquisition unit 154 captures vehicles that are present in the lane of the lane change destination as new target vehicles.

[0128] When the target vehicle is switched due to a lane change, the target capture unit 154 sets the control transfer ratio during the switching period of the target vehicle. Figure 23 This is a diagram used to illustrate the setting of the control transfer ratio during the switching period of the target vehicle. Figure 23 In the example, suppose vehicle M is traveling at speed VM in lane L1, the center lane of a road consisting of three lanes L1 to L3 that allow travel in the same direction, and another vehicle m1 is traveling in front of vehicle M at speed Vm1, while another vehicle m2 is traveling in lane L2, to the left of lane L1, in front of vehicle M at speed Vm2. Figure 23In the example, let's assume that vehicle M is following other vehicles m1 using driving controls such as ACC.

[0129] In this situation, when the vehicle M changes lanes from lane L1 to lane L2 (e.g., ALC), the target acquisition unit 154 switches the following target vehicle (target vehicle) from other vehicle m1 to other vehicle m2. For example, at each position on the predetermined track K2 before the vehicle M changes lanes from lane L1 to lane L2, the target acquisition unit 154 sets the control transfer ratio based on the relative position and speed of the other vehicles m1 and m2 being followed. Thus, for example, by setting the control for the behavior of other vehicle m1 to 70% and the control for the behavior of other vehicle m2 to 30%, the speed, steering, and other control parameters are adjusted (corrected), thereby enabling smooth driving control during lane changes.

[0130] Regarding the setting of the control transfer ratio mentioned above, for example, in order to switch the target vehicle of vehicle M from other vehicle m1 to other vehicle m2 when other vehicle m2 enters between vehicle M and other vehicles by changing lanes while vehicle M is following other vehicle m1, the control transfer ratio mentioned above is set during the switching period, and driving control based on the set ratio is executed.

[0131] The interference possibility target extraction unit 156 defines an area containing the current lane and the lanes to its left and right, within a predetermined distance from the reference position of the vehicle M (e.g., the front end or center of gravity) towards the direction of travel, as an interference possibility zone, and extracts other vehicles existing within this area as target vehicles with interference possibility. The predetermined distance can be a fixed distance or can be set to be variable based on road shape, road type, vehicle speed, etc. The interference possibility target extraction unit 156 can extract a predetermined number of other vehicles from each of the left lane, the current lane, and the right lane, or extract a predetermined number of other vehicles from the lane closest to the current vehicle if the total number of other vehicles in the three lanes exceeds a threshold. Furthermore, when the driving control unit 160 is performing ACC (Adaptive Cruise Control), the interference possibility target extraction unit 156 extracts vehicles (other vehicles) that have entered (successfully cut in) the current lane from an adjacent lane between the preceding vehicle and the current vehicle M as target vehicles that may interfere with the current vehicle M. By extracting potential target vehicles, such as adjusting (correcting) speed, steering, and other control parameters in the driving control unit 160 to avoid contact with the target vehicle, more appropriate driving control can be performed.

[0132] The verification unit 158 ​​verifies whether it is correct to use the target vehicle as the object of driving control (speed control, driving control) of the vehicle M based on the determination results of the target vehicle captured by the target capture unit 154, the target vehicle extracted by the interference possibility target extraction unit 156, and the lane belonging of other vehicles determined by the lane belonging determination unit 148. The correspondence between each target vehicle (e.g., other vehicles m1 to m3) and each other vehicle in the lane belonging to which the lane belongs determined by the lane belonging unit 148 (e.g., other vehicles m1 to m3) is established, for example, by their relative position, shape, size, etc. from the vehicle M.

[0133] For example, if a target vehicle (other vehicle) identified as existing in lane L1 and extracted as a control target or interference possibility target is determined by the lane determination unit 148 to not belong to lane L1 (or exist in another lane), the verification unit 158 ​​determines that it is not the correct target for driving control and corrects the identification result identified by the identification unit 120. Specifically, the verification unit 158 ​​excludes other vehicles that do not belong to lane L1 (or exist in another lane) from the target vehicles (vehicles that become the target of driving control) extracted as target vehicles based on the identification result identified by the identification unit 120. In this way, compared with the identification unit 120, which identifies objects while using three-dimensional image transformation, the position error of the object in a two-dimensional image is smaller. Therefore, by using the lane determination result of other vehicles based on a two-dimensional image (relative lateral position information of other vehicles), unwanted target vehicles in driving control can be excluded, and excessive speed control caused by misidentification can be suppressed. The verification unit 158 ​​can also replace the exclusion of unwanted target vehicles in driving control by correcting the position of the excluded target vehicle so that it is consistent with the lane determined by the lane determination unit 148.

[0134] When a target vehicle (other vehicle) is identified as existing in lane L1 and determined by the lane determination unit 148 to belong to (exist in) lane L1, the verification unit 158 ​​sets it as the correct driving control object and outputs information (target information) related to other vehicles that are the target objects.

[0135] The speed control unit 162 performs speed control of the vehicle M based on the target information output by the verification unit 158. For example, the speed control unit 162 calculates a following target control quantity for following the target at an appropriate inter-vehicle distance based on the target vehicle's state quantities (relative distance, relative speed, target vehicle spacing, etc.) contained in the target information. The speed control unit 162 adjusts the control quantity based on the interference state with potential interference targets contained in the target information to prevent the vehicle M from exceeding a predetermined acceleration limit (limit: the limit value of the vehicle M's forward and backward acceleration obtained from the vehicle sensor 40) due to acceleration or deceleration. The speed control unit 162 adjusts the following control quantity based on information such as the behavior (position, speed) of potential interference targets regarding other vehicles changing lanes into lane L1 or other vehicles traveling with part of their vehicle entering the lane (an example of potential interference targets). The speed control unit 162 can also adjust the control quantity based on the behavior and other states of potential interference targets if there are potential interference targets that cannot be captured as control targets. Furthermore, the speed control unit 162 generates a speed profile for driving the vehicle M based on the following control quantity obtained through adjustments, etc., and the current speed control quantity of the vehicle M, and performs speed control of the vehicle M based on the generated speed profile.

[0136] The steering control unit 164 also performs driving controls such as ALC and LKAS based on the aforementioned target information to avoid contact with other vehicles.

[0137] Therefore, it is possible to appropriately correct for positional deviations of target vehicles caused by recognition errors in objects identified based on, for example, 3D image transformation relative to camera images or the outputs of multiple sensors. This improves the recognition accuracy of following vehicles and vehicles cutting in line. When the lane determination unit 148 determines that a vehicle does not exist in its lane, it can be excluded from following vehicles and vehicles cutting in line to suppress excessive deceleration. This enables vehicle control with less occupant discomfort.

[0138] The HMI control unit 170 can also generate images that distinguish other vehicles identified as the target vehicle from other vehicles excluded from the target vehicle, and display the generated images on the display unit of the HMI 30. This allows for more accurate notification to the occupant of the vehicle to be controlled during driving.

[0139] As described above, according to this embodiment, the vehicle control device includes: a second camera (an example of an imaging unit) 20 that captures images of the surroundings of the vehicle M; an identification unit 120 that identifies the surrounding conditions of the vehicle M; a driving control unit 160 that controls one or both of the vehicle M's speed and steering based on the identification result of the identification unit 120; and a control unit 140 that controls the driving control unit 160 based on the content captured by the second camera 20. The control unit 140 determines the lane to which an object belongs based on the positional relationship between the left and right dividing lines that divide the vehicle M's driving lane on the two-dimensional image captured by the second camera 20 and the left and right ends of objects existing around the vehicle M. If neither of the left or right dividing lines is identified, the control unit determines the lane to which the object belongs based on the positional relationship of the object's end relative to the identified dividing lines. This allows for more accurate identification of the position of objects contained in the image and more precise driving control of the vehicle. Therefore, traffic safety can be further improved.

[0140] Specifically, according to the embodiment, the lane to which the object belongs is determined based on the positional relationship between the dividing line on the two-dimensional image captured by the second camera 20 and the end of the object. This suppresses positional shifts (lane shifts) of the object that occur when the position is estimated based on the image (overhead view) after three-dimensional transformation by the recognition unit 120 and the outputs of multiple sensors. According to the embodiment, when the object is another vehicle, the end is designated as the ground contact point, thereby improving the accuracy of determining the lane to which other vehicles belong. According to the embodiment, the ground contact points as the left and right ends are obtained from the rectangular information of the bounding box by referring to the segmented image, thereby limiting the search range on the image and reducing processing costs. According to the embodiment, the lane to which the object belongs can be identified with high accuracy even when other vehicles cross the lane.

[0141] According to the implementation method, even when one lane marking cannot be detected, the lane to which other vehicles belong can be determined more accurately based on the positional relationship between the left and right ends relative to the detectable lane marking. Therefore, even when one lane marking is not detected due to bad weather, obstacles, etc., the lane to which the object belongs can be determined, and the position of the object can be identified more accurately based on the determined lane.

[0142] The implementation methods described above can be performed as follows.

[0143] A vehicle control device comprising:

[0144] Storage device, which stores a program; and

[0145] Hardware processor,

[0146] The hardware processor executes the program stored in the storage device to perform the following processing:

[0147] Use the camera unit to film the area around this vehicle;

[0148] Identify the surrounding conditions of the vehicle;

[0149] Perform driving control based on the recognition results to control the speed and steering of the vehicle by one or both of these aspects;

[0150] The driving control is based on the content captured by the camera.

[0151] The lane to which the object belongs is determined based on the positional relationship between the left and right dividing lines on the two-dimensional image captured by the camera and the left and right ends of the object. If neither of the left and right dividing lines is identified, the lane to which the object belongs is determined based on the positional relationship between the ends of the object and the identified dividing lines. The left and right dividing lines divide the lane in which the vehicle is traveling. The object exists around the vehicle.

[0152] The above description illustrates specific embodiments of the present invention, but the present invention is not limited to such embodiments in any way, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A vehicle control device, wherein, The vehicle control device includes: The camera department, which films the area around the vehicle; The identification unit identifies the surrounding conditions of the vehicle. The driving control unit controls one or both of the vehicle's speed and steering based on the recognition result of the recognition unit; and The control unit controls the driving control unit based on the content captured by the camera unit. The control unit determines the lane to which the object belongs based on the positional relationship between the left and right dividing lines on the two-dimensional image captured by the camera unit and the left and right ends of the object. If neither of the left or right dividing lines is detected, the control unit determines the lane to which the object belongs based on the positional relationship between the ends of the object and the detected dividing lines. Here, the left and right dividing lines are the lines that divide the lane in which the vehicle is traveling, and the object is an object existing around the vehicle. The control unit corrects the position of the object identified by the recognition unit based on information related to the lane to which the object belongs, and controls the driving control unit based on the corrected position of the object.

2. The vehicle control device according to claim 1, wherein, If the control unit does not identify one of the left and right dividing lines in the two-dimensional image, and the object has its ends on the left and right sides relative to the identified dividing lines, it determines that the object belongs to at least the current lane.

3. The vehicle control device according to claim 1, wherein, If the control unit does not identify one of the left and right dividing lines in the two-dimensional image, and the object has its left or right ends relative to the identified dividing line on either the left or right, it determines that the object does not belong to the lane on the side where the ends do not exist relative to the dividing line.

4. The vehicle control device according to claim 1, wherein, When the control unit does not identify one of the left and right dividing lines but identifies a portion of the other dividing line, it determines the lane to which the object belongs based on the positional relationship between any end of the object's left or right end and the identified dividing line that exists laterally on the two-dimensional image of the end.

5. The vehicle control device according to claim 1, wherein, When a portion of both the left and right dividing lines are identified, the control unit determines the lane to which the object belongs based on the positional relationship between the left and right ends of the object and the identified dividing lines that exist laterally on the two-dimensional image of the ends.

6. A vehicle control method, wherein, The vehicle control method causes the computer to perform the following processing: Use the camera unit to film the area around this vehicle; Identify the surrounding conditions of the vehicle; Perform driving control based on the recognition results to control the speed and steering of the vehicle by one or both of them; The driving control is based on the content captured by the camera. The lane to which the object belongs is determined based on the positional relationship between the left and right dividing lines on the two-dimensional image captured by the camera and the left and right ends of the object. If neither of the left and right dividing lines is identified, the lane to which the object belongs is determined based on the positional relationship between the ends of the object and the identified dividing lines. The left and right dividing lines are dividing lines that divide the lane in which the vehicle is traveling, and the object is an object that exists around the vehicle. The position of the identified object is corrected based on information related to the lane to which the object belongs, and the driving control is based on the corrected position of the object.

7. A storage medium storing a program, wherein, The program causes the computer to perform the following processing: Use the camera unit to film the area around this vehicle; Identify the surrounding conditions of the vehicle; Perform driving control based on the recognition results to control the speed and steering of the vehicle by one or both of them; The driving control is based on the content captured by the camera. The lane to which the object belongs is determined based on the positional relationship between the left and right dividing lines on the two-dimensional image captured by the camera and the left and right ends of the object. If neither of the left or right dividing lines is identified, the lane to which the object belongs is determined based on the positional relationship between the ends of the object and the identified dividing lines. The left and right dividing lines are dividing lines that divide the lane in which the vehicle is traveling, and the object is an object existing around the vehicle. The position of the identified object is corrected based on information related to the lane to which the object belongs, and the driving control is based on the corrected position of the object.

Images