Vehicle control device, vehicle control method, and program

Abstract

To appropriately recognize an azimuth direction of a vehicle.SOLUTION: A vehicle control device comprises: an image acquisition section which acquires an image showing an external space of a vehicle; an object detection section which detects a plurality of types of objects including a road structure and a mobile body shown in the image through image processing; a reference line setting section which selects one type of object from the plurality of types of objects and sets a reference line along an azimuth direction of the selected type of object; and a vehicle azimuth direction estimation section which estimates an angle between the reference line and a travel direction line of the vehicle as the azimuth direction of the vehicle with respect to a lane where the vehicle is traveling or going to travel.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a program. [Background technology]

[0002] There are known techniques for detecting relative information of another vehicle with respect to a host vehicle. For example, Patent Document 1 discloses a technique for detecting a relative position of another vehicle based on a wireless signal transmitted from the other vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-161430 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when only relative information such as the relative positions of other vehicles is used, there are cases where the vehicle's direction cannot be properly recognized, for example, when the vehicle changes lanes.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objects is to provide a vehicle control device, a vehicle control method, and a program that can appropriately recognize the direction of a vehicle. [Means for solving the problem]

[0006] The vehicle control device according to the present invention employs the following configuration. (1): A vehicle control device according to one embodiment of the present invention includes an image acquisition unit that acquires an image of an external space of a vehicle, a target detection unit that detects a plurality of types of targets, including road structures and moving objects, captured in the image by image processing, a reference line setting unit that selects one type of target from the plurality of types of targets and sets a reference line along the direction of the selected type of target, and a vehicle orientation estimation unit that estimates the angle between the reference line and the vehicle's travel orientation line as the orientation of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel.

[0007] (2): In the above aspect (1), among the orientations of the target, the orientation of the road structure is the extension direction of the road structure, and the orientation of the moving body is the traveling direction of the moving body.

[0008] (3) In the above aspect (1) or (2), the reference line setting unit selects one type of target object from the multiple types of detected targets based on a predetermined priority order.

[0009] (4): In the above aspect (3), the predetermined priority is given to the boundary line of the lane in which the vehicle is traveling or is scheduled to travel, the boundary line of the lane adjacent to the lane is given the second highest priority, and other vehicles around the vehicle are given the third highest priority.

[0010] (5): In any of the above aspects (1) to (4), a driving control unit is further provided that generates a target trajectory based on the direction of the vehicle estimated by the vehicle direction estimation unit, and controls the steering and acceleration / deceleration of the vehicle so that the vehicle travels along the generated target trajectory without relying on the operation of a driver of the vehicle.

[0011] (6): In any of the above aspects (1) to (5), a driving instruction unit is further provided that generates a target trajectory based on the direction of the vehicle estimated by the vehicle direction estimation unit, and provides at least one of steering instructions and acceleration / deceleration instructions to an occupant of the vehicle to drive along the generated target trajectory.

[0012] (7): A vehicle control method according to another aspect of the present invention involves a computer mounted on a vehicle acquiring an image of the space outside the vehicle, detecting a plurality of types of targets, including road structures and moving objects, captured in the image by image processing, selecting one type of target from the plurality of types of targets, setting a reference line along the direction of the selected type of target, and estimating the angle between the reference line and the vehicle's travel direction line as the direction of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel.

[0013] (8): A program according to another aspect of the present invention causes a computer mounted on a vehicle to acquire an image of the space outside the vehicle, detect multiple types of targets, including road structures and moving objects, captured in the image by image processing, select one type of target from the multiple types of targets, set a reference line along the direction of the selected type of target, and estimate the angle between the reference line and the vehicle's traveling direction line as the direction of the vehicle relative to the lane in which the vehicle is traveling or planned to travel. Effect of the Invention

[0014] According to (1) to (8), the vehicle direction can be appropriately recognized.

[0015] According to (5), the vehicle can be appropriately controlled based on the recognized vehicle direction. [Brief description of the drawings]

[0016] [Figure 1] 1 is a configuration diagram of a vehicle system 1 that uses a vehicle control device according to an embodiment. [Diagram 2] 2 is a functional configuration diagram of a first control unit 120 and a second control unit 160. FIG. [Diagram 3] FIG. 4 is a diagram illustrating an example of vehicle travel control according to a comparative example. [Figure 4] 11 is a diagram showing an example of a priority order in which a reference line setting unit 130A selects a target. FIG. [Diagram 5]1 is a diagram showing a first example of a scene in which a reference line RL is set and a direction of a host vehicle M is estimated. [Figure 6] 13 is a diagram showing a second example of a scene in which the reference line RL is set and the direction of the host vehicle M is estimated. FIG. [Figure 7] 13 is a diagram showing a third example of a scene in which the reference line RL is set and the direction of the host vehicle M is estimated. FIG. [Figure 8] 13 is a diagram showing a fourth example of a scene in which the reference line RL is set and the direction of the host vehicle M is estimated. FIG. [Figure 9] 1 is a flowchart showing an example of a flow of processing executed by cooperation between the camera 10, the object recognition device 16, and the automatic driving control device 100. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of a vehicle control device, a vehicle control method, and a program according to the present invention will be described with reference to the drawings.

[0018] [Overall configuration] 1 is a configuration diagram of a vehicle system 1 that uses a vehicle control device according to an embodiment. The vehicle on which the vehicle system 1 is mounted is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or discharged power from a secondary battery or a fuel cell.

[0019] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, a vehicle sensor 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driving operator 80, an automatic driving control device 100, a driving force output device 200, a brake device 210, and a steering device 220. These devices and equipment are connected to each other by multiple communication lines such as a CAN (Controller Area Network) communication line, a serial communication line, a wireless communication network, etc. Note that the configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configuration may be added.

[0020] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to an arbitrary location of a vehicle (hereinafter, the vehicle M) in which the vehicle system 1 is mounted. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically and repeatedly captures images of the surroundings of the vehicle M. The camera 10 may be a stereo camera.

[0021] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to an arbitrary location of the vehicle M. The radar device 12 may detect the position and speed of an object by an FM-CW (Frequency Modulated Continuous Wave) method.

[0022] The LIDAR 14 irradiates light (or electromagnetic waves with a wavelength close to that of light) around the vehicle M and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, a pulsed laser light. The LIDAR 14 is attached to an arbitrary location of the vehicle M.

[0023] The object recognition device 16 performs sensor fusion processing on some or all of the detection results from the camera 10, the radar device 12, and the LIDAR 14 to recognize the position, type, speed, and the like of an object. The object recognition device 16 outputs the recognition result to the automatic driving control device 100. The object recognition device 16 may output the detection results from the camera 10, the radar device 12, and the LIDAR 14 to the automatic driving control device 100 as they are. The object recognition device 16 may be omitted from the vehicle system 1. In this embodiment, the object recognition device 16 includes an image acquisition unit 16A and a target detection unit 16B. The image acquisition unit 16A acquires an image of the external space of the vehicle captured by the camera 10. The target detection unit 16B detects a plurality of types of targets, including road structures and moving objects, captured in the image by image processing.

[0024] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via a wireless base station.

[0025] The HMI 30 presents various information to an occupant of the vehicle M and accepts input operations by the occupant. The HMI 30 includes various display devices, a speaker, a buzzer, a touch panel, switches, keys, and the like.

[0026] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the host vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, a direction sensor that detects the direction of the host vehicle M, and the like.

[0027] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 holds first map information 54 in a storage device such as a HDD (Hard Disk Drive) or a flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on a signal received from a GNSS satellite. The position of the vehicle M may be identified or complemented by an INS (Inertial Navigation System) using the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, and the like. The navigation HMI 52 may be partially or entirely shared with the above-mentioned HMI 30. The route determination unit 53, for example, determines a route (hereinafter, a route on a map) from the position of the vehicle M identified by the GNSS receiver 51 (or an arbitrary position input) to a destination input by a driver using the navigation HMI 52, with reference to the first map information 54. The first map information 54 is, for example, information in which road shapes are expressed by links indicating roads and nodes connected by the links. The first map information 54 may include road curvature and POI (Point Of Interest) information. The route on the map is output to the MPU 60. The navigation device 50 may perform route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized by, for example, a function of a terminal device such as a smartphone or tablet terminal owned by a passenger. The navigation device 50 may transmit a current position and a destination to a navigation server via the communication device 20, and obtain a route equivalent to the route on the map from the navigation server.

[0028] The MPU 60 includes, for example, a recommended lane determination unit 61, and holds second map information 62 in a storage device such as an HDD or a flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of blocks (for example, by dividing it into 100 m intervals in the vehicle travel direction), and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines which lane from the left the vehicle should travel in. When a branch point is present on the route on the map, the recommended lane determination unit 61 determines a recommended lane so that the host vehicle M can travel along a reasonable route to proceed to the branch point.

[0029] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on boundaries of lanes. The second map information 62 may also include road information, traffic regulation information, address information (address and zip code), facility information, telephone number information, and the like. The second map information 62 may be updated as needed by the communication device 20 communicating with another device.

[0030] The driving operators 80 include, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, an irregular steering wheel, a joystick, and other operators. The driving operators 80 are fitted with sensors that detect the amount of operation or the presence or absence of operation, and the detection results are output to the automatic driving control device 100, or some or all of the driving force output device 200, the brake device 210, and the steering device 220.

[0031] The automatic driving control device 100 includes, for example, a first control unit 120 and a second control unit 160. The first control unit 120 and the second control unit 160 are each realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage device (storage device having a non-transient storage medium) such as an HDD or flash memory of the automatic driving control device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the automatic driving control device 100 by mounting the storage medium (non-transient storage medium) in a drive device. The combination of the object recognition device 16 and the automatic driving control device 100 is an example of a "vehicle control device", and the combination of the action plan generation unit 140 and the second control unit 160 is an example of a "driving control unit".

[0032] FIG. 2 is a functional configuration diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, a recognition unit 130 and an action plan generation unit 140. The first control unit 120 realizes, for example, a function based on AI (Artificial Intelligence) and a function based on a pre-given model in parallel. For example, the function of "recognizing an intersection" may be realized by executing in parallel recognition of an intersection by deep learning or the like and recognition based on pre-given conditions (such as traffic lights and road markings that can be pattern matched), and by scoring both and evaluating them comprehensively. This ensures the reliability of the autonomous driving.

[0033] The recognition unit 130 recognizes the position, speed, acceleration, and other states of objects around the vehicle M based on information input from the camera 10, the radar device 12, and the LIDAR 14 via the object recognition device 16. The position of the object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of the object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by a represented area. The "state" of the object may include the acceleration or jerk of the object, or the "behavior state" (for example, whether or not the object is changing lanes or is about to change lanes).

[0034] The recognition unit 130 also recognizes, for example, the lane (driving lane) in which the host vehicle M is traveling. For example, the recognition unit 130 recognizes the driving lane by comparing a pattern (for example, an arrangement of solid lines and dashed lines) of road dividing lines (hereinafter referred to as "boundary lines") obtained from the second map information 62 with a pattern of boundary lines around the host vehicle M recognized from an image captured by the camera 10. The recognition unit 130 may recognize the driving lane by recognizing road boundaries (road boundaries) including not only boundary lines but also boundary lines, road shoulders, curbs, median strips, guard rails, etc. In this recognition, the position of the host vehicle M obtained from the navigation device 50 and the processing results by the INS may be taken into consideration. The recognition unit 130 also recognizes stop lines, obstacles, red lights, toll booths, and other road phenomena.

[0035] The recognition unit 130 further includes a reference line setting unit 130A and a vehicle orientation estimation unit 130B, and estimates the orientation of the vehicle M with respect to the reference line set by the reference line setting unit 130A. The functions of the reference line setting unit 130A and the vehicle orientation estimation unit 130B will be described in detail later.

[0036] In principle, the action plan generating unit 140 drives the recommended lane determined by the recommended lane determining unit 61, and further generates a target trajectory along which the host vehicle M will automatically (without the driver's operation) drive in the future based on the vehicle's orientation estimated by the vehicle orientation estimating unit 130B so as to be able to respond to the surrounding conditions of the host vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is expressed as a sequence of points (trajectory points) to be reached by the host vehicle M. The trajectory points are points to be reached by the host vehicle M at each predetermined travel distance (for example, about several [m]) along the road, and separately, the target speed and target acceleration for each predetermined sampling time (for example, about a few tenths [sec]) are generated as part of the target trajectory. The trajectory points may also be positions to be reached by the host vehicle M at each sampling time for each predetermined sampling time. In this case, the information on the target speed and target acceleration is expressed as the interval between the trajectory points.

[0037] The behavior plan generating unit 140 may set an event of autonomous driving when generating the target trajectory. The autonomous driving events include a constant speed driving event, a low speed following driving event, a lane change event, a branching event, a merging event, a takeover event, etc. The behavior plan generating unit 140 generates a target trajectory according to the activated event.

[0038] The second control unit 160 controls the traveling driving force output device 200, the braking device 210, and the steering device 220 so that the host vehicle M passes the target trajectory generated by the action plan generating unit 140 at the scheduled time.

[0039] Returning to FIG. 2, the second control unit 160 includes, for example, an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires information on the target trajectory (trajectory points) generated by the action plan generation unit 140, and stores it in a memory (not shown). The speed control unit 164 controls the traveling driving force output device 200 or the brake device 210 based on a speed element associated with the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 executes a combination of feedforward control according to the curvature of the road ahead of the host vehicle M and feedback control based on the deviation from the target trajectory.

[0040] The driving force output device 200 outputs a driving force (torque) for the vehicle to travel to the driving wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, and an ECU (Electronic Control Unit) that controls these. The ECU controls the above components according to information input from the second control unit 160 or information input from the driving operation device 80.

[0041] The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor according to information input from the second control unit 160 or information input from the driving operation element 80, so that a brake torque corresponding to a braking operation is output to each wheel. The brake device 210 may include a backup mechanism that transmits hydraulic pressure generated by operation of a brake pedal included in the driving operation element 80 to the cylinder via a master cylinder. Note that the brake device 210 is not limited to the configuration described above, and may be an electronically controlled hydraulic brake device that controls an actuator according to information input from the second control unit 160 to transmit hydraulic pressure in the master cylinder to the cylinder.

[0042] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by applying a force to, for example, a rack and pinion mechanism. The steering ECU drives the electric motor in accordance with information input from the second control unit 160 or information input from the driving operator 80 to change the direction of the steered wheels.

[0043] [Comparative Example] Next, a comparative example will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of driving control of a vehicle m according to the comparative example. The vehicle m has an automatic driving function, but does not include at least the reference line setting unit 130A and the vehicle direction estimation unit 130B shown in FIG. 2. In FIG. 3(a), the vehicle m is traveling on the lane L1 and is planning to change lanes to the lane L2. Vehicles m1 and m2, which are surrounding vehicles of the vehicle m, are traveling straight on the lanes L2 and L3, respectively, so the vehicle m can change lanes without any problems by maintaining or increasing the speed when changing lanes to the lane L2.

[0044] On the other hand, in FIG. 3(b), vehicle m is also traveling in lane L1 and is planning to change lanes to lane L2. At this time, the relative positional relationship between vehicle m and vehicles m1 and m2 is similar to that in FIG. 3(a). However, in FIG. 3(b), vehicles m1 and m2 are traveling in directions that deviate from lanes L2 and L3, respectively, and changing lanes of vehicle m to lane L2 may cause problems, particularly with vehicle m2. That is, it may not be appropriate to control vehicle m based only on the relative positional relationship between vehicle m and vehicles m1 and m2.

[0045] Comparing FIG. 3(a) with FIG. 3(b), the relative positional relationship between the vehicle m and the vehicles m1 and m2 is similar, but the angle between the traveling direction of the vehicle m and the boundary line of the lane L1 on which the vehicle m is traveling is different. Specifically, in FIG. 3(a), the angle between the traveling direction of the vehicle m and the extension direction of the boundary line of the lane L1, i.e., the orientation of the vehicle m, is a positive value, whereas in FIG. 3(b), the orientation of the vehicle m is approximately zero. Therefore, in the scene of FIG. 3(b), since the orientation of the vehicle m is approximately zero, not the vehicle m but another vehicle is traveling in a direction that deviates from the lane, and a problem may occur in the execution of lane change by the vehicle m. In contrast, the vehicle M of this embodiment can more appropriately control the traveling of the vehicle M by setting a reference line and estimating the orientation of the vehicle M with respect to the reference line. Details will be described below.

[0046] [Setting the reference line and estimating the direction of vehicle M] The reference line setting unit 130A selects one type of target from the multiple types of targets recognized by the object recognition device 16 based on a predetermined priority order, and sets a virtual reference line RL (corresponding to the above-mentioned "extension direction of the boundary line") along the direction of the selected type of target. At this time, the target recognized by the object recognition device 16 is expressed by a camera coordinate system with the camera 10 as the origin, but the reference line setting unit 130A sets the reference line RL after, for example, converting the coordinates of the camera coordinate system into coordinates on an imaginary plane that represents the space around the vehicle M as a two-dimensional plane viewed from above.

[0047] The vehicle orientation estimation unit 130B estimates the angle between the set reference line RL and the travel orientation line of the host vehicle M (corresponding to the above-mentioned "travel direction of the vehicle m") as the orientation of the host vehicle M with respect to the lane in which the host vehicle M is traveling or is scheduled to travel. In this embodiment, the travel orientation line is the central axis of the host vehicle M, but alternatively, for example, it may be the instantaneous moving direction of the host vehicle M, i.e., the actual travel direction of the host vehicle M.

[0048] FIG. 4 is a diagram showing an example of the priority order in which the reference line setting unit 130A selects targets. As shown in FIG. 4, the boundary line, guardrail, and wall of the own lane are set in the first priority order, which indicates the highest priority order. Here, the boundary line, guardrail, and wall of the own lane may be of the lane in which the own vehicle M is traveling, or may be of the lane in which the own vehicle M is scheduled to travel. For example, immediately before the own vehicle M executes a lane change or during the execution of the lane change, the boundary line, guardrail, or wall of the lane to which the change is to be made may be selected as the target, instead of the boundary line, guardrail, or wall of the lane in which the own vehicle M is traveling. Note that, when two or more types of targets among the boundary line, guardrail, and wall of the own lane are recognized by the object recognition device 16, the reference line setting unit 130A may select one type of target by any method, and may select, for example, a target closer to the own vehicle M. Some or all of the boundary lines, guardrails, and walls are examples of "road structures".

[0049] The boundary line, guardrail, and wall of the adjacent lane are set at the second priority level, which indicates the second highest priority level. That is, when the boundary line, guardrail, and wall of the own lane are not included in the targets recognized by the object recognition device 16, the reference line setting unit 130A selects the boundary line, guardrail, or wall of the adjacent lane as the target. When two or more types of targets among the boundary line, guardrail, and wall of the adjacent lane are recognized by the object recognition device 16, the reference line setting unit 130A may select one type of target by any method, and may select, for example, a target closer to the own vehicle M.

[0050] The third highest priority is set to another vehicle in the adjacent lane. That is, when the boundary line, guard rail, and wall between the own lane and the adjacent lane are not included in the targets recognized by the object recognition device 16, the reference line setting unit 130A selects another vehicle in the adjacent lane as a target. Here, when there are multiple other vehicles in the adjacent lane, the reference line setting unit 130A selects the multiple other vehicles as targets. The other vehicle in the adjacent lane is an example of a "moving body."

[0051] In this way, the reference line setting unit 130A selects one type of target based on the priority order shown in FIG. 4, and sets a reference line RL along the azimuth of the selected type of target. Here, the "azimuth of the target" refers to the extension direction of a boundary line, a guardrail, and a wall, and refers to the moving direction of the other vehicle in the case of another vehicle in an adjacent lane. In this case, the moving direction may be the central axis of the other vehicle or the estimated moving direction of the other vehicle. Furthermore, when multiple targets of the same type are selected or multiple vehicles in an adjacent lane are selected, the reference line setting unit 130A sets a reference line RL along the average azimuth of the multiple targets.

[0052] [Scene example] Next, with reference to Figs. 5 to 8, an example of a scene in which the reference line RL is set and the orientation of the vehicle M is estimated will be described. Fig. 5 is a diagram showing a first example of a scene in which the reference line RL is set and the orientation of the vehicle M is estimated. In Fig. 5, the vehicle M is traveling in the lane L1 and is about to change lanes to the lane L2 because the number of lanes is decreasing. In this scene, the camera 10 acquires images of the boundary lines BL1 and BL2 of the lane L1, the boundary line BL3 of the lane L2, and the other vehicle M1, and the object recognition device 16 detects the boundary lines BL1, BL2, BL3, and the other vehicle M1 as targets by image processing. Next, the reference line setting unit 130A selects the boundary lines BL1 and BL2, which have the first priority, as targets based on the priority order shown in Fig. 4, and sets the reference line RL along the average orientation of the boundary lines BL1 and BL2. Next, the vehicle orientation estimation unit 130B estimates the angle θ between the set reference line RL and the traveling direction line of the host vehicle M as the orientation of the host vehicle M with respect to the lane L1. Then, the action plan generation unit 140 generates a target trajectory of the host vehicle M based on the orientation of the host vehicle estimated by the vehicle orientation estimation unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the host vehicle M so as to travel along the generated target trajectory without depending on the operation of the driver of the host vehicle M.

[0053] In the first example scene, the reference line setting unit 130A selects the boundary lines BL1 and BL2, which have the highest priority, as targets and sets the reference line RL along their average orientation, but the reference line setting unit 130A may select only one of the boundaries BL1 and BL2 as a target and set the reference line RL along the orientation. For example, of the boundary lines BL1 and BL2, the boundary line closer to the host vehicle M may be selected as a target. Furthermore, the reference line setting unit 130A may recognize the lane L2, which is the lane after the lane change, as the lane on which the host vehicle M is scheduled to travel, rather than the boundary line of the lane L1, which is the lane before the lane change, and select the boundary line BL2 as a target.

[0054] FIG. 6 is a diagram showing a second example of a scene in which the reference line RL is set and the direction of the vehicle M is estimated. In FIG. 6, the vehicle M changes lanes from the lane L1 to the lane L2 and enters the lane L2. In this scene, the camera 10 acquires images of the boundary lines BL1 and BL3 of the lane L2, the boundary line BL2 of the lane L1, and the other vehicle M1, and the object recognition device 16 detects the boundary lines BL1, BL3, BL2, and the other vehicle M1 as targets by image processing. Next, the reference line setting unit 130A selects the boundary lines BL1 and BL3, which have the first priority, as targets based on the priority order shown in FIG. 4, and sets the reference line RL along the average direction of the boundary lines BL1 and BL3. Next, the vehicle direction estimation unit 130B estimates the angle θ between the set reference line RL and the traveling direction line of the vehicle M as the direction of the vehicle M with respect to the lane L2. Then, the action plan generating unit 140 generates a target trajectory of the host vehicle M based on the orientation of the host vehicle estimated by the vehicle orientation estimating unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the host vehicle M so that the host vehicle M travels along the generated target trajectory without relying on the operation of the driver of the host vehicle M. As in the case of Fig. 5, the reference line setting unit 130A may select only one of the boundary line BL1 and the boundary line BL3 as a target and set a reference line RL along the orientation.

[0055] FIG. 7 is a diagram showing a third example of a scene in which the reference line RL is set and the direction of the vehicle M is estimated. In FIG. 7, the vehicle M is changing lanes from the lane L1 to the lane L2. In this scene, the camera 10 acquires an image of the boundary line BL3 of the lane L2 and the other vehicle M1. However, at this time, the boundary line BL3 is thin and intermittent, so the object recognition device 16 may not be able to detect the boundary line BL3 as a target by image processing. Even if the object recognition device 16 detects the boundary line BL3 as a target, the intermittent boundary line may have low accuracy as a reference line. Therefore, the reference line setting unit 130A selects the other vehicle with the third priority as a target based on the priority order shown in FIG. 4, and sets the reference line RL along the direction of the other vehicle M1. Next, the vehicle direction estimation unit 130B estimates the angle θ between the set reference line RL and the traveling direction line of the vehicle M as the direction of the vehicle M with respect to the lane L2. Then, the action plan generation unit 140 generates a target trajectory for the vehicle M based on the direction of the vehicle estimated by the vehicle direction estimation unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the vehicle M so that the vehicle M travels along the generated target trajectory without relying on the operation of the driver of the vehicle M.

[0056] FIG. 8 is a diagram showing a fourth example of a scene in which the reference line RL is set and the direction of the vehicle M is estimated. In FIG. 8, the vehicle M is traveling on the lane L1 and is about to merge into the lane L2. In this scene, the camera 10 acquires images of the boundary line BL1 of the lane L1, the boundary line BL2 of the lane L2, the boundary line BL3 of the lane L3, and the other vehicle M1, and the object recognition device 16 detects the boundary lines BL1, BL2, BL3, and the other vehicle M1 as targets by image processing. Next, the reference line setting unit 130A selects the boundary line BL1, which has the first priority, as a target based on the priority order shown in FIG. 4, and sets the reference line RL along the direction of the boundary line BL1. Next, the vehicle direction estimation unit 130B estimates the angle θ between the set reference line RL and the traveling direction line of the vehicle M as the direction of the vehicle M with respect to the lane L1. Then, the action plan generating unit 140 generates a target trajectory of the host vehicle M based on the orientation of the host vehicle estimated by the vehicle orientation estimating unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the host vehicle M so that the host vehicle M travels along the generated target trajectory without relying on the operation of the driver of the host vehicle M. Note that, similarly to the case of Fig. 5, the reference line setting unit 130A may recognize the lane L2, which is the lane after the merging, as the lane on which the host vehicle M is scheduled to travel, instead of the boundary line of the lane L1, which is the lane before the merging, and select the boundary line BL2 as the target.

[0057] [Operation flow] Next, a process flow executed by the cooperation of the camera 10, the object recognition device 16, and the automatic driving control device 100 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of a process flow executed by the cooperation of the camera 10, the object recognition device 16, and the automatic driving control device 100. The process of the flowchart is repeatedly executed at each predetermined control cycle while the host vehicle M is traveling.

[0058] First, the camera 10 acquires an image of the external space of the host vehicle M (S100). Next, the object recognition device 16 detects a plurality of types of targets, including road structures and moving objects, captured in the image acquired by the camera 10 by image processing (S101). Next, the reference line setting unit 130A of the automatic driving control device 100 selects one type of target from the plurality of types of detected targets based on the priority order shown in FIG. 4 (S102). Next, the reference line setting unit 130A of the automatic driving control device 100 sets a reference line along the direction of the selected type of target (S103). Next, the vehicle direction estimation unit 130B of the automatic driving control device 100 estimates the angle between the set reference line and the traveling direction line of the host vehicle M as the direction of the host vehicle M with respect to the lane in which the host vehicle M is traveling or is scheduled to travel (S104). Next, the driving control unit of the automatic driving control device 100 generates a target trajectory based on the direction of the vehicle estimated by the vehicle direction estimation unit, and controls the steering and acceleration / deceleration of the host vehicle M without relying on the operation of the driver of the host vehicle M so that the host vehicle M travels along the generated target trajectory (S105). This ends the processing of this flowchart.

[0059] According to the process of this embodiment described above, targets included in an image captured by the camera 10 are detected, one type of target is selected from the detected targets based on a predetermined priority order, a reference line is set along the direction of the selected target, and the angle between the set reference line and the traveling direction line of the host vehicle M is estimated as the direction of the host vehicle M. This makes it possible to appropriately recognize the direction of the vehicle.

[0060] In the above embodiment, an example in which the vehicle control device of the present invention is applied to automatic driving has been described. However, the vehicle control device of the present invention is not limited to this configuration and can also be applied to manual driving. In that case, the vehicle control device of the present invention may further include a driving instruction unit that generates a target trajectory based on the direction of the host vehicle M estimated by the vehicle direction estimation unit 130B, instead of the driving control unit, and gives at least one of a steering instruction and an acceleration / deceleration instruction so that the occupant of the host vehicle M drives along the generated target trajectory. The driving instruction unit can be realized, for example, as a part of the function of the navigation device 50.

[0061] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Acquire an image of the external space of the vehicle; Detecting a plurality of types of targets including road structures and moving objects captured in the image by image processing; Selecting one type of target from the plurality of types of targets, and setting a reference line along the azimuth of the selected type of target; estimating an angle between the reference line and a line of travel of the vehicle as an orientation of the vehicle relative to a lane in which the vehicle is traveling or will travel; The vehicle control device is configured as follows.

[0062] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0063] 10. Camera 16 Object recognition device 16A Image acquisition unit 16B Target detection section 100 Automatic driving control device 120 First Control Section 130 Recognition part 130A Reference line setting section 130B Vehicle direction estimation unit 140 Action Plan Generation Unit 160 Second Control Section

Claims

1. an image acquisition unit that acquires an image of an external space of the vehicle; a target detection unit that detects a plurality of types of targets, including road structures and moving objects, captured in the image by image processing; a reference line setting unit that selects one type of target from the plurality of types of targets and sets a reference line along a direction of the selected type of target; a vehicle direction estimation unit that estimates an angle between the reference line and a traveling direction line of the vehicle as a direction of the vehicle with respect to a lane in which the vehicle is traveling or will travel; Equipped with the reference line setting unit selects one type of target from the plurality of types of detected targets based on a predetermined priority order; The predetermined priority order is the highest for the boundary line of the lane in which the vehicle is traveling or is scheduled to travel, the second highest for the boundary line of the adjacent lane to the lane, and the third highest for other vehicles around the vehicle. Vehicle control device.

2. an image acquisition unit that acquires an image of an external space of the vehicle; a target detection unit that detects a plurality of types of targets, including road structures and moving objects, captured in the image by image processing; a reference line setting unit that selects one type of target from the plurality of types of targets based on a predetermined priority order and at least one of a three-dimensional road structure included in the plurality of types of detected objects and another vehicle existing in an adjacent lane, and sets a reference line along a direction of the target of the selected type; a vehicle direction estimation unit that estimates an angle between the reference line and a traveling direction line of the vehicle as a direction of the vehicle with respect to a lane in which the vehicle is traveling or will travel; A vehicle control device comprising:

3. Among the orientations of the target, the orientation of the road structure is the extension direction of the road structure, and the orientation of the moving body is the traveling direction of the moving body. The vehicle control device according to claim 1 or 2.

4. a driving control unit that generates a target trajectory based on the direction of the vehicle estimated by the vehicle direction estimation unit, and controls steering and acceleration / deceleration of the vehicle without relying on an operation by a driver of the vehicle so that the vehicle travels along the generated target trajectory; The vehicle control device according to any one of claims 1 to 3.

5. a driving instruction unit that generates a target trajectory based on the direction of the vehicle estimated by the vehicle direction estimation unit, and gives at least one of a steering instruction and an acceleration / deceleration instruction so that an occupant of the vehicle drives the vehicle along the generated target trajectory, The vehicle control device according to any one of claims 1 to 4.

6. The vehicle's on-board computer Acquire an image of the external space of the vehicle; Detecting a plurality of types of targets including road structures and moving objects captured in the image by image processing; Selecting one type of target from the plurality of types of targets, and setting a reference line along the azimuth of the selected type of target; estimating an angle between the reference line and a line of travel of the vehicle as an orientation of the vehicle relative to a lane in which the vehicle is traveling or will travel; selecting one type of target from the plurality of types of detected targets based on a predetermined priority order; The predetermined priority order is the highest for the boundary line of the lane in which the vehicle is traveling or is scheduled to travel, the second highest for the boundary line of the adjacent lane to the lane, and the third highest for other vehicles around the vehicle. A vehicle control method.

7. The vehicle's on-board computer Acquire an image of an external space of the vehicle; Detecting a plurality of types of targets including road structures and moving objects captured in the image by image processing; Selecting one type of target from the plurality of types of targets and setting a reference line along a bearing of the selected type of target; estimating an angle between the reference line and a traveling bearing line of the vehicle as an orientation of the vehicle with respect to a lane in which the vehicle is traveling or will travel; selecting one type of target from the plurality of types of detected targets based on a predetermined priority order; The predetermined priority order is the highest for the boundary line of the lane in which the vehicle is traveling or is scheduled to travel, the second highest for the boundary line of the adjacent lane to the lane, and the third highest for other vehicles around the vehicle. program.

8. The vehicle's on-board computer Acquire an image of the external space of the vehicle; Detecting a plurality of types of targets including road structures and moving objects captured in the image by image processing; selecting one type of target from the plurality of types of targets based on a predetermined priority order and at least one of a three-dimensional road structure included in the plurality of types of detected objects and another vehicle existing in an adjacent lane, and setting a reference line along a direction of the target of the selected type; estimating an angle between the reference line and a line of travel of the vehicle as an orientation of the vehicle relative to a lane in which the vehicle is traveling or will travel; A vehicle control method.

9. The vehicle's on-board computer Acquire an image of an external space of the vehicle; Detecting a plurality of types of targets including road structures and moving objects captured in the image by image processing; selecting one type of target from the plurality of types of targets based on a predetermined priority order and at least one of a three-dimensional road structure included in the plurality of types of detected objects and another vehicle existing in an adjacent lane, and setting a reference line along a direction of the target of the selected type; an angle between the reference line and a traveling direction line of the vehicle is estimated as a direction of the vehicle with respect to a lane in which the vehicle is traveling or is to travel; program.

Images