Vehicle control device, vehicle control method, and storage medium

Abstract

The present invention provides a vehicle control device, a vehicle control method, and a storage medium that can appropriately identify the orientation of a vehicle. A vehicle control device includes: an image acquisition unit that acquires an image obtained by photographing the external space of the vehicle; an object detection unit that detects, by image processing, a plurality of types of object targets reflected in the image, including road structures and moving objects; a reference line setting unit that selects one type of object target from the plurality of types of object targets and sets a reference line along the orientation of the selected type of object target; and a vehicle orientation estimation unit that estimates the angle formed by the reference line and the vehicle's traveling orientation line as the orientation of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel.

Description

Technical Field

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a storage medium. Background Art

[0002] There is known a technique for detecting relative information of another vehicle with respect to the own vehicle. For example, Japanese Unexamined Patent Application Publication No. 2017-161430 discloses a technique for detecting the relative position of another vehicle based on a wireless signal transmitted from the other vehicle. Summary of the Invention

[0003] However, in the case of using only relative information such as the relative position of another vehicle, for example, there is a case where the orientation of the vehicle cannot be appropriately recognized when the vehicle changes lanes.

[0004] The present invention has been made in consideration of such a situation, and one of its purposes is to provide a vehicle control device, a vehicle control method, and a storage medium that can appropriately recognize the orientation of a vehicle.

[0005] The vehicle control device of the present invention adopts the following configuration.

[0006] (1): A vehicle control device according to an aspect of the present invention includes: an image acquisition unit that acquires an image obtained by photographing an external space of the vehicle; an object detection unit that detects, by image processing, a plurality of types of object targets including road structures and moving objects reflected in the image; a reference line setting unit that selects one type of object target from the plurality of types of object targets and sets a reference line along the orientation of the selected type of object target; and a vehicle orientation estimation unit that estimates an angle formed by the reference line and a traveling orientation line of the vehicle as the orientation of the vehicle with respect to a lane in which the vehicle is traveling or is scheduled to travel.

[0007] (2): Based on the aspect of (1) above, in the orientation of the object target, the orientation of the road structure is the extending direction of the road structure, and the orientation of the moving object is the traveling direction of the moving object.

[0008] (3): Based on the aspect of (1) or (2) above, the reference line setting unit selects one type of object target from the detected plurality of types of object targets based on a specified priority order.

[0009] (4): Based on the aspect of (3) above, in the specified priority order, the boundary line of the lane in which the vehicle is traveling or is scheduled to travel is the highest, the boundary line of an adjacent lane of the lane is the second highest, and other vehicles around the vehicle are the third highest.

[0010] (5): Based on any one of the above (1) to (4), the vehicle control device further includes a driving control unit that generates a target trajectory based on the azimuth of the vehicle estimated by the vehicle azimuth estimation unit, and controls the steering and acceleration / deceleration of the vehicle without relying on the operation of the driver of the vehicle so that the vehicle travels along the generated target trajectory.

[0011] (6): Based on any one of the above (1) to (5), the vehicle control device further includes a driving instruction unit that generates a target trajectory based on the azimuth of the vehicle estimated by the vehicle azimuth estimation unit, and gives at least one of a steering instruction and an acceleration / deceleration instruction so that the occupants of the vehicle drive along the generated target trajectory.

[0012] (7): A vehicle control method according to another aspect of the present invention causes a computer mounted on a vehicle to perform the following processing: acquiring an image obtained by photographing the external space of the vehicle; detecting, through image processing, a plurality of types of object targets reflected in the image, including road structures and moving objects; selecting one type of object target from the plurality of types of object targets, and setting a reference line along the azimuth of the selected type of object target; and estimating the angle formed by the reference line and the traveling azimuth line of the vehicle as the azimuth of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel.

[0013] (8): A storage medium according to still another aspect of the present invention stores a program, wherein the program causes a computer mounted on a vehicle to perform the following processing: acquiring an image obtained by photographing the external space of the vehicle; detecting, through image processing, a plurality of types of object targets reflected in the image, including road structures and moving objects; selecting one type of object target from the plurality of types of object targets, and setting a reference line along the azimuth of the selected type of object target; and estimating the angle formed by the reference line and the traveling azimuth line of the vehicle as the azimuth of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel.

[0014] According to (1) to (8), the azimuth of the vehicle can be appropriately identified.

[0015] According to (5), the vehicle can be appropriately controlled based on the identified azimuth of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a structural diagram of a vehicle system using the vehicle control device of the embodiment.

[0017] Figure 2 is a functional structural diagram of the first control unit and the second control unit.

[0018] Figure 3 It is a diagram showing an example of the driving control of a vehicle in a comparative example.

[0019] Figure 4 It is a diagram showing an example of the priority order in which the reference line setting unit selects an object target.

[0020] Figure 5 It is a diagram showing a first scenario example of setting a reference line RL and estimating the orientation of the host vehicle M.

[0021] Figure 6 It is a diagram showing a second scenario example of setting a reference line RL and estimating the orientation of the host vehicle M.

[0022] Figure 7 It is a diagram showing a third scenario example of setting a reference line RL and estimating the orientation of the host vehicle M.

[0023] Figure 8 It is a diagram showing a fourth scenario example of setting a reference line RL and estimating the orientation of the host vehicle M.

[0024] Figure 9 It is a flowchart showing an example of the process flow executed through the cooperation of a camera, an object recognition device, and an autonomous driving control device. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the vehicle control device, the vehicle control method, and the storage medium of the present invention will be described with reference to the drawings.

[0026] [OVERALL CONFIGURATION]

[0027] Figure 1 It is a structural diagram of a vehicle system 1 using a vehicle control device in an embodiment. The vehicle equipped with the vehicle system 1 is, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source is an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination thereof. The electric motor operates using the generated power of a generator connected to the internal combustion engine, or the discharge power of a secondary battery or a fuel cell.

[0028] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU (Map Positioning Unit) 60, a driving operation member 80, an autonomous driving control device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are interconnected through multi-channel communication lines such as CAN (Controller Area Network) communication lines, serial communication lines, wireless communication networks, etc. It should be noted that Figure 1 The structure shown is just an example, and part of the structure can be omitted, or other structures can be further added.

[0029] The camera 10 is, for example, a digital camera that uses a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is installed at any part of the vehicle (hereinafter referred to as the vehicle M) on which the vehicle system 1 is mounted. When shooting forward, the camera 10 is installed on the upper part of the front windshield, the back of the in-vehicle rearview mirror, etc. The camera 10, for example, periodically and repeatedly shoots the surroundings of the vehicle M. The camera 10 can also be a stereo camera.

[0030] The radar device 12 emits radio waves such as millimeter waves to the surroundings of the vehicle M, and detects the radio waves (reflected waves) reflected by an object to detect at least the position (distance and azimuth) of the object. The radar device 12 is installed at any part of the vehicle M. The radar device 12 can also detect the position and speed of an object by the FM-CW (Frequency Modulated Continuous Wave) method.

[0031] The LIDAR 14 irradiates light (or an electromagnetic wave with a wavelength close to light) to the surroundings of the vehicle M and measures the scattered light. The LIDAR 14 detects the distance to an object based on the time from light emission to light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 is installed at any part of the vehicle M.

[0032] The object recognition device 16 performs sensor fusion processing on the detection results based on a part or all of the camera 10, the radar device 12, and the LIDAR 14, and recognizes the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition result to the automatic driving control device 100. The object recognition device 16 may output the detection results of the camera 10, the radar device 12, and the LIDAR 14 to the automatic driving control device 100 as they are. The object recognition device 16 may also be omitted from the vehicle system 1. In the present embodiment, the object recognition device 16 includes an image acquisition unit 16A and an object target detection unit 16B. The image acquisition unit 16A acquires an image of the external space of the vehicle captured by the camera 10. The object target detection unit 16B detects, through image processing, a plurality of types of object targets including road structures and moving objects reflected in the image.

[0033] The communication device 20 communicates with other vehicles existing in the vicinity of the own vehicle M or communicates with various server devices via a wireless base station, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like.

[0034] The HMI 30 presents various information to the occupants of the own vehicle M and accepts input operations based on the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, buttons, and the like.

[0035] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the own vehicle M, an acceleration sensor that detects acceleration, a yaw rate sensor that detects the angular velocity about the vertical axis, an azimuth sensor that detects the orientation of the own vehicle M, and the like.

[0036] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as an HDD (Hard Disk Drive) or a flash memory. The GNSS receiver 51 determines the position of the host vehicle M based on signals received from GNSS satellites. The position of the host vehicle M can also be determined or supplemented by an INS (Inertial Navigation System) that utilizes the output of the vehicle sensor 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. Part or all of the navigation HMI 52 can be shared with the aforementioned HMI 30. The route determination unit 53 determines, for example, a route (hereinafter referred to as a map route) from the position of the host vehicle M determined by the GNSS receiver 51 (or an arbitrary input position) to a destination input by the occupant using the navigation HMI 52 with reference to the first map information 54. The first map information 54 is information that represents the road shape, for example, by showing road segments and nodes connected by the road segments. The first map information 54 may also include information such as the curvature of the road and POI (Point Of Interest) information. The map route is output to the MPU 60. The navigation device 50 can also perform route guidance using the navigation HMI 52 based on the map route. The navigation device 50 can be implemented, for example, by the functions of a terminal device such as a smartphone or a tablet terminal held by the occupant. The navigation device 50 can also send the current position and the destination to a navigation server via the communication device 20 and obtain a route equivalent to the map route from the navigation server.

[0037] The MPU 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or a flash memory. The recommended lane determination unit 61 divides the map route provided from the navigation device 50 into a plurality of blocks (for example, divided every 100 [m] in the vehicle traveling direction) and determines a recommended lane for each block with reference to the second map information 62. The recommended lane determination unit 61 makes a determination as to which lane from the left to drive in. When there is a branch point on the map route, the recommended lane determination unit 61 determines the recommended lane so that the host vehicle M can travel on a reasonable route for traveling to the branch destination.

[0038] The second map information 62 is map information with higher precision than the first map information 54. The second map information 62 includes, for example, information on the center of a lane or information on the boundary of a lane. In addition, the second map information 62 may include road information, traffic restriction information, address information (address, postal code), facility information, telephone number information, etc. The second map information 62 can be updated at any time by communicating with other devices through the communication device 20.

[0039] The driving operation member 80 includes, for example, an accelerator pedal, a brake pedal, a shift lever, a steering wheel, a special-shaped steering device, a joystick, and other operation members. A sensor for detecting the operation amount or the presence or absence of an operation is installed on the driving operation member 80, and the detection result is output to a part or all of the autonomous driving control device 100, or the driving force output device 200, the braking device 210, and the steering device 220.

[0040] The autonomous driving control device 100 includes, for example, a first control unit 120 and a second control unit 160. The first control unit 120 and the second control unit 160 are respectively implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). In addition, some or all of these components can be implemented by hardware (including a circuit unit: circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or can be implemented by the cooperation of software and hardware. The program can be pre-stored in a storage device (a storage device having a non-transitory storage medium) such as an HDD or a flash memory of the autonomous driving control device 100, or can be stored in a removable storage medium such as a DVD or a CD-ROM, and installed in the HDD or flash memory of the autonomous driving control device 100 by being mounted on a driving device through the storage medium (non-transitory storage medium). Combining the object recognition device 16 and the autonomous driving control device 100 is an example of a "vehicle control device", and combining the action plan generation unit 140 and the second control unit 160 is an example of a "driving control unit".

[0041] Figure 2It is a functional structure diagram of the first control unit 120 and the second control unit 160. The first control unit 120 includes, for example, an identification unit 130 and an action plan generation unit 140. The first control unit 120 implements functions based on AI (Artificial Intelligence) and functions based on a pre-given model in parallel. For example, the function of "identifying an intersection" can be achieved by parallelly executing the identification of an intersection based on deep learning, etc., and the identification based on pre-given conditions (such as signals and road signs capable of pattern matching), and comprehensively evaluating by scoring both. Thereby, the reliability of autonomous driving is ensured.

[0042] Based on the information input from the camera 10, the radar device 12, and the LIDAR 14 via the object recognition device 16, the identification unit 130 identifies the position of the objects around the host vehicle M, as well as the states such as speed and acceleration. The position of the object is, for example, identified as the position on the absolute coordinates with the representative point (such as the center of gravity or the center of the drive shaft) of the host vehicle M as the origin, and is used for control. The position of the object can be represented by the representative points such as the center of gravity or the corner of the object, or can be represented by the exhibited area. The "state" of the object can also include the acceleration, jerk, or "action state" of the object (for example, whether it is changing lanes or about to change lanes).

[0043] In addition, the identification unit 130 identifies, for example, the lane (travel lane) in which the host vehicle M is traveling. For example, the identification unit 130 identifies the travel lane by comparing the pattern (for example, the arrangement of solid lines and dashed lines) of the road dividing line (hereinafter referred to as "boundary line") obtained from the second map information 62 with the pattern of the boundary line around the host vehicle M identified from the image captured by the camera 10. It should be noted that the identification unit 130 is not limited to identifying the boundary line, and can also identify the travel lane by identifying the boundary line, the travel road boundary (road boundary) including the road shoulder, curb, median strip, guardrail, etc. In this identification, the position of the host vehicle M obtained from the navigation device 50 and the processing result based on the INS can also be added. In addition, the identification unit 130 identifies the temporary stop line, obstacles, red lights, toll booths, and other road phenomena.

[0044] The identification unit 130 also includes a reference line setting unit 130A and a vehicle orientation estimation unit 130B, and estimates the orientation of the host vehicle M related to the reference line set by the reference line setting unit 130A. The details of the functions of the reference line setting unit 130A and the vehicle orientation estimation unit 130B will be described later.

[0045] The action plan generation unit 140 generates a target trajectory for the host vehicle M to travel automatically (independently of the driver's operation) in the future in such a way that it travels on the recommended lane determined by the recommended lane determination unit 61 in principle and can respond to the surrounding conditions of the host vehicle M. The target trajectory includes, for example, a speed element. For example, the target trajectory is represented as a trajectory formed by arranging in order the points (trajectory points) that the host vehicle M should reach. The trajectory points are the points that the host vehicle M should reach at regular driving distances (e.g., on the order of several [m]) along the way. In contrast, the target speed and target acceleration at regular sampling times (e.g., on the order of zero point several [sec]) are generated as part of the target trajectory. Additionally, the trajectory points can also be the positions that the host vehicle M should reach at regular sampling times. In this case, the information on the target speed and target acceleration is represented by the interval of the trajectory points.

[0046] When generating the target trajectory, the action plan generation unit 140 can set events for autonomous driving. Among the events for autonomous driving, there are events such as a constant-speed driving event, a low-speed following driving event, a lane change event, a branch event, a merging event, a takeover event, etc. The action plan generation unit 140 generates a target trajectory corresponding to the activated event.

[0047] The second control unit 160 controls the driving force output device 200, the braking device 210, and the steering device 220 so that the host vehicle M passes through the target trajectory generated by the action plan generation unit 140 at a predetermined time.

[0048] Return Figure 2 , for example, the second control unit 160 includes an acquisition unit 162, a speed control unit 164, and a steering control unit 166. The acquisition unit 162 acquires the information on the target trajectory (trajectory points) generated by the action plan generation unit 140 and stores this information in a memory (not shown). The speed control unit 164 controls the driving force output device 200 or the braking device 210 based on the speed element attached to the target trajectory stored in the memory. The steering control unit 166 controls the steering device 220 according to the curvature of the target trajectory stored in the memory. The processing of the speed control unit 164 and the steering control unit 166 is realized, for example, by a combination of feedforward control and feedback control. As an example, the steering control unit 166 combines feedforward control corresponding to the curvature of the road ahead of the host vehicle M and feedback control based on the deviation from the target trajectory and executes them.

[0049] The driving force output device 200 outputs the driving force (torque) for vehicle travel to the drive wheels. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, and a transmission, as well as an ECU (Electronic Control Unit) that controls them. The ECU controls the above-described structure according to the information input from the second control unit 160 or the information input from the driving operation member 80.

[0050] The braking device 210 includes, for example, a brake caliper, a working cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the working cylinder, and a brake ECU. The brake ECU controls the electric motor according to the information input from the second control unit 160 or the information input from the driving operation member 80, and outputs a braking torque corresponding to the braking operation to each wheel. The braking device 210 may include a mechanism that transmits the hydraulic pressure generated by the operation of the brake pedal included in the driving operation member 80 to the working cylinder via the master cylinder as a backup. It should be noted that the braking device 210 is not limited to the structure described above, and may also be an electronically controlled hydraulic braking device that controls an actuator according to the information input from the second control unit 160 and transmits the hydraulic pressure of the master cylinder to the working cylinder.

[0051] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor, for example, applies a force to a rack-pinion mechanism to change the orientation of the steering wheels. The steering ECU drives the electric motor according to the information input from the second control unit 160 or the information input from the driving operation member 80 to change the orientation of the steering wheels.

[0052] [Comparative Example]

[0053] Next, a comparative example will be described with reference to Figure 3 FIG. Figure 3 FIG. is an example showing the travel control of the vehicle m in the comparative example. The vehicle m has an autonomous driving function, but at least does not include Figure 2 the reference line setting unit 130A and the vehicle orientation estimation unit 130B shown. In Figure 3 (a) of FIG., the vehicle m is traveling in the lane L1 and plans to change lanes to the lane L2. The vehicles m1 and m2, which are surrounding vehicles of the vehicle m, are traveling straight in the lanes L2 and L3, respectively. Therefore, by maintaining or increasing the speed when the vehicle m changes lanes to the lane L2, the lane change can be carried out without problems.

[0054] On the other hand, in Figure 3 (b) of FIG., the vehicle m is also traveling in the lane L1 and plans to change lanes to the lane L2. At this time, the relative positional relationship between the vehicle m and the vehicles m1 and m2 is similar to that in Figure 3 (a) of FIG. However, inFigure 3 In (b) of this, vehicles m1 and m2 travel in the directions of leaving lanes L2 and L3 respectively, and when vehicle m changes lanes to lane L2, problems may particularly occur with vehicle m2. That is, it is sometimes inappropriate to control vehicle m only based on the relative positional relationship between vehicle m and vehicles m1 and m2.

[0055] When Figure 3 in (a) of this is compared with Figure 3 in (b) of this, although the relative positional relationship between vehicle m and vehicles m1 and m2 is similar, the angle formed by the traveling direction of vehicle m and the boundary line of lane L1 on which vehicle m travels is different. Specifically, in Figure 3 in (a) of this, the angle formed by the traveling direction of vehicle m and the extending direction of the boundary line of lane L1, that is, the azimuth of vehicle m takes a positive value. On the other hand, in Figure 3 in (b) of this, the azimuth of vehicle m is approximately zero. Therefore, in the scenario of (b) of Figure 3 this, since the azimuth of vehicle m is approximately zero, other vehicles rather than vehicle m travel in the direction of leaving the lane, which may cause problems in the execution of the lane change of vehicle m. In contrast, the own vehicle M of the present embodiment can more appropriately control the traveling of the own vehicle M by setting a reference line and estimating the azimuth of the own vehicle M related to this reference line. Hereinafter, the detailed situation thereof will be described.

[0056] [Setting of the reference line and estimation of the azimuth of the own vehicle M]

[0057] The reference line setting unit 130A selects one type of object target from the multiple types of object targets recognized by the object recognition device 16 based on a prescribed priority order, and sets an imaginary reference line RL (corresponding to the above-mentioned "extending direction of the boundary line") along the azimuth of the selected type of object target. At this time, the object targets recognized by the object recognition device 16 are represented by a camera coordinate system with the camera 10 as the origin, but the reference line setting unit 130A sets the reference line RL, for example, on the basis of transforming the coordinates of the camera coordinate system into the coordinates on an assumed plane which is a two-dimensional plane representation obtained by observing the space around the own vehicle M from above.

[0058] The vehicle azimuth estimation unit 130B estimates the angle formed by the set reference line RL and the traveling azimuth line of the own vehicle M (corresponding to the above-mentioned "traveling direction of vehicle m") as the azimuth of the own vehicle M relative to the lane in which the own vehicle M is traveling or is scheduled to travel. In the present embodiment, the traveling azimuth line is the central axis of the own vehicle M, but instead, for example, it may be the instantaneous moving direction of the own vehicle M, that is, the actual traveling direction of the own vehicle M.

[0059] Figure 4This is a diagram showing an example of the priorities for the reference line setting unit 130A to select object targets. As Figure 4 shown, the boundary lines, guardrails, and walls of the own lane are set at the first place in the priority order indicating the highest priority. Here, the boundary lines, guardrails, and walls of the own lane may be the boundary lines, guardrails, and walls of the lane in which the own vehicle M is traveling, or may also be the boundary lines, guardrails, and walls of the lane in which the own vehicle M is scheduled to travel. For example, before the own vehicle M is about to execute a lane change or during the execution of the lane change, instead of selecting the boundary lines, guardrails, or walls of the lane in which it is traveling as object targets, the boundary lines, guardrails, or walls of the lane to which the change destination is set may be selected as object targets. It should be noted that when two or more types of object targets among the boundary lines, guardrails, and walls of the own lane are recognized by the object recognition device 16, the reference line setting unit 130A may select one type of object target by any method. For example, it may also select the object target closer to the own vehicle M. A part or all of the boundary lines, guardrails, and walls are an example of a "road structure".

[0060] The boundary lines, guardrails, and walls of the adjacent lane are set at the second place in the priority order indicating the second highest priority. That is, when the boundary lines, guardrails, and walls of the own lane are not included among the object targets recognized by the object recognition device 16, the reference line setting unit 130A selects the boundary lines, guardrails, or walls of the adjacent lane as object targets. When two or more types of object targets among the boundary lines, guardrails, and walls of the adjacent lane are recognized by the object recognition device 16, the reference line setting unit 130A may select one type of object target by any method. For example, it may also select the object target closer to the own vehicle M.

[0061] Other vehicles existing in the adjacent lane are set at the third place in the priority order indicating the third highest priority. That is, when the boundary lines, guardrails, and walls of the own lane and the adjacent lane are not included among the object targets recognized by the object recognition device 16, the reference line setting unit 130A selects other vehicles existing in the adjacent lane as object targets. Here, when there are a plurality of other vehicles existing in the adjacent lane, the reference line setting unit 130A selects these plurality of other vehicles as object targets. Other vehicles existing in the adjacent lane are an example of a "moving body".

[0062] In this way, the reference line setting unit 130A is based on Figure 4Based on the priority shown below, select an object target of a certain type and set a reference line RL along the orientation of the selected object target of that type. Here, in the case of a boundary line, guardrail, or wall, the "orientation of the object target" refers to their extension direction. In the case of another vehicle in an adjacent lane, the "orientation of the object target" refers to the moving direction of that other vehicle. It should be noted that the moving direction in this case can be either the central axis of the other vehicle or the estimated moving direction of the other vehicle. Also, when multiple object targets of a certain type are selected or multiple other vehicles in adjacent lanes are selected, the reference line setting unit 130A sets a reference line RL along the average orientation of these multiple object targets.

[0063] [Scene example]

[0064] Next, with reference to Figures 5 - 8 an example of a scene for setting the reference line RL and estimating the orientation of the host vehicle M will be described. Figure 5 FIG. is a diagram showing a first scene example for setting the reference line RL and estimating the orientation of the host vehicle M. In Figure 5 this figure, the host vehicle M is traveling in lane L1 and needs to perform a lane change to lane L2 due to a reduction in the number of lanes. In this scene, the camera 10 acquires images of the boundary line BL1 and boundary line BL2 of lane L1, the boundary line BL3 of lane L2, and another vehicle M1. The object recognition device 16 detects the boundary line BL1, boundary line BL2, boundary line BL3, and another vehicle M1 as object targets through image processing. Next, based on the priority shown in Figure 4 the reference line setting unit 130A selects the boundary line BL1 and boundary line BL2 with the highest priority as object targets and sets a reference line RL along the average orientation of the boundary line BL1 and boundary line BL2. Next, the vehicle orientation estimation unit 130B estimates the angle θ formed by the set reference line RL and the traveling orientation line of the host vehicle M as the orientation of the host vehicle M relative to lane L1. Then, the action plan generation unit 140 generates a target trajectory for the host vehicle M based on the orientation of the host vehicle estimated by the vehicle orientation estimation unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the host vehicle M independently of the operation of the driver of the host vehicle M so that it travels along the generated target trajectory.

[0065] It should be noted that in the first scenario example, the reference line setting unit 130A selects the boundary lines BL1 and BL2 with the highest priority as object targets, and sets the reference line RL along their average azimuth. However, the reference line setting unit 130A can also select only one of them as the object target and set the reference line RL along its azimuth. For example, the boundary line closer to the own vehicle M among the boundary lines BL1 and BL2 can be selected as the object target. Also, the reference line setting unit 130A may not select the boundary line of the lane L1, which is the lane before the lane change, as the object target, but identify the lane L2, which is the lane after the lane change, as the lane in which the own vehicle M is scheduled to travel, and select the boundary line BL2 of the lane L2 as the object target.

[0066] Figure 6 FIG. is a diagram showing a second scenario example in which the reference line RL is set and the azimuth of the own vehicle M is estimated. In Figure 6 this case, the own vehicle M executes a lane change from the lane L1 to the lane L2 and enters the lane L2. In this scenario, the camera 10 acquires images of the boundary lines BL1 and BL3 of the lane L2, the boundary line BL2 of the lane L1, and the other vehicle M1. The object recognition device 16 detects the boundary lines BL1, BL3, BL2, and the other vehicle M1 as object targets through image processing. Then, the reference line setting unit 130A, based on Figure 4 the priority shown, selects the boundary lines BL1 and BL3 with the highest priority as object targets, and sets the reference line RL along the average azimuth of the boundary lines BL1 and BL3. Then, the vehicle azimuth estimation unit 130B estimates the angle θ formed by the set reference line RL and the traveling azimuth line of the own vehicle M as the azimuth of the own vehicle M with respect to the lane L2. Then, the action plan generation unit 140 generates a target trajectory of the own vehicle M based on the azimuth of the own vehicle estimated by the vehicle azimuth estimation unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the own vehicle M independently of the operation of the driver of the own vehicle M so as to travel along the generated target trajectory. Similarly to the Figure 5 case of, the reference line setting unit 130A can also select only one of the boundary lines BL1 and BL3 and set the reference line RL along its azimuth.

[0067] Figure 7 FIG. is a diagram showing a third scenario example in which the reference line RL is set and the azimuth of the own vehicle M is estimated. In Figure 7In this case, the vehicle M performs a lane change from lane L1 to lane L2. In this scenario, the camera 10 acquires an image of the boundary line BL3 of lane L2 and another vehicle M1. However, at this time, the boundary line BL3 is faint and intermittent, so there is a possibility that the object recognition device 16 cannot detect the boundary line BL3 as an object target through image processing. Additionally, even when the object recognition device 16 detects the boundary line BL3 as an object target, the accuracy of the intermittent boundary line as a reference line may be low. Thus, the reference line setting unit 130A selects, as the object target, another vehicle with a third priority based on the priority order shown in Figure 4 and sets a reference line RL along the direction of the other vehicle M1. Next, the vehicle direction estimating unit 130B estimates the angle θ formed by the set reference line RL and the traveling direction line of the vehicle M as the direction of the vehicle M relative to lane L2. Then, the action plan generation unit 140 generates a target trajectory of the vehicle M based on the direction of the vehicle M estimated by the vehicle direction estimating unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the vehicle M regardless of the operation of the driver of the vehicle M so that the vehicle travels along the generated target trajectory.

[0068] Figure 8 FIG. is a diagram showing a fourth scenario example in which the reference line RL is set and the direction of the vehicle M is estimated. In Figure 8 , the vehicle M is traveling in lane L1 and is about to merge into lane L2. In this scenario, the camera 10 acquires images of the boundary line BL1 of lane L1, the boundary line BL2 of lane L2, the boundary line BL3 of lane L3, and another vehicle M1, and the object recognition device 16 detects the boundary line BL1, the boundary line BL2, the boundary line BL3, and the other vehicle M1 as object targets through image processing. Next, the reference line setting unit 130A selects, as the object target, the boundary line BL1 with a first priority based on the priority order shown in Figure 4 and sets a reference line RL along the direction of the boundary line BL1. Next, the vehicle direction estimating unit 130B estimates the angle θ formed by the set reference line RL and the traveling direction line of the vehicle M as the direction of the vehicle M relative to lane L1. Then, the action plan generation unit 140 generates a target trajectory of the vehicle M based on the direction of the vehicle M estimated by the vehicle direction estimating unit 130B, and the second control unit 160 controls the steering and acceleration / deceleration of the vehicle M regardless of the operation of the driver of the vehicle M so that the vehicle travels along the generated target trajectory. It should be noted that, similarly to the case of Figure 5 , the reference line setting unit 130A may not select the boundary line of lane L1, which is the lane before merging, as the object target, but may recognize lane L2, which is the lane after merging, as the lane in which the vehicle M is scheduled to travel, and select the boundary line BL2 of lane L2 as the object target.

[0069] [Flow of the operation]

[0070] Next, with reference to Figure 9 the flow of the process executed through the cooperation of the camera 10, the object recognition device 16, and the autonomous driving control device 100 will be described. Figure 9 It is a flowchart showing an example of the flow of the process executed through the cooperation of the camera 10, the object recognition device 16, and the autonomous driving control device 100. The process of this flowchart is repeatedly executed in a prescribed control cycle during the travel of the own vehicle M.

[0071] First, the camera 10 acquires an image obtained by photographing the external space of the own vehicle M (S100). Next, the object recognition device 16 detects, through image processing, a plurality of types of object targets including road structures and moving objects reflected in the image acquired by the camera 10 (S101). Next, the reference line setting unit 130A of the autonomous driving control device 100 selects one type of object target from the detected plurality of types of object targets based on Figure 4 the priority shown (S102). Next, the reference line setting unit 130A of the autonomous driving control device 100 sets a reference line along the orientation of the selected type of object target (S103). Next, the vehicle orientation estimation unit 130B of the autonomous driving control device 100 estimates the angle formed by the set reference line and the traveling orientation line of the own vehicle M as the orientation of the own vehicle M with respect to the lane in which the own vehicle M is traveling or is scheduled to travel (S104). Next, the driving control unit of the autonomous driving control device 100 generates a target trajectory based on the orientation of the vehicle estimated by the vehicle orientation estimation unit, and controls the steering and acceleration / deceleration of the own vehicle M so as to travel along the generated target trajectory without depending on the operation of the driver of the own vehicle M (S105). Thus, the process of this flowchart ends.

[0072] According to the process of the present embodiment described as above, object targets included in the image acquired by the camera 10 are detected, one type of object target is selected from the detected object targets based on a prescribed priority, a reference line along the orientation of the selected object target is set, and the angle formed by the set reference line and the traveling orientation line of the own vehicle M is estimated as the orientation of the own vehicle M. Thereby, the orientation of the vehicle can be appropriately recognized.

[0073] Note that, in the above-described embodiments, an example in which the vehicle control device of the present invention is applied to autonomous driving has been described. However, the vehicle control device of the present invention is not limited to this structure and can also be applied to manual driving. In this case, the vehicle control device of the present invention only needs to further include a driving instruction unit instead of the driving control unit. The driving instruction unit generates a target track based on the orientation of the own vehicle M estimated by the vehicle orientation estimation unit 130B, and gives at least one of a steering instruction and an acceleration / deceleration instruction so that the occupant of the own vehicle M drives along the generated target track. The driving instruction unit can be realized, for example, as a part of the function of the navigation device 50.

[0074] The above-described embodiments can be expressed as follows.

[0075] A vehicle control device, wherein,

[0076] The vehicle control device includes:

[0077] A storage device that stores a program; and

[0078] A hardware processor,

[0079] By executing the program stored in the storage device by the hardware processor, the following processing is performed:

[0080] Obtain an image obtained by photographing the external space of the vehicle;

[0081] Detect, by image processing, a plurality of types of object targets including road structures and moving objects reflected in the image;

[0082] Select one type of object target from the plurality of types of object targets, and set a reference line along the orientation of the selected type of object target;

[0083] Estimate the angle formed by the reference line and the traveling orientation line of the vehicle as the orientation of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel.

[0084] The specific embodiments of the present invention have been described above using the embodiments, but the present invention is in no way limited to such embodiments, and various modifications and substitutions can be made without departing from the gist of the present invention.

Claims

1. A vehicle control device, wherein, the vehicle control device includes: an image acquisition unit that acquires an image obtained by photographing the external space of the vehicle; an object detection unit that detects, by image processing, a plurality of types of object targets reflected in the image, including road structures and moving bodies; a reference line setting unit that selects one type of object target from the plurality of types of object targets and sets a reference line along the orientation of the selected type of object target; and a vehicle orientation estimation unit that estimates the angle formed by the reference line and the traveling orientation line of the vehicle as the orientation of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel, the reference line setting unit selects one type of object target from the detected plurality of types of object targets based on a prescribed priority order, in the prescribed priority order, the boundary line of the lane in which the vehicle is traveling or is scheduled to travel is the highest, the boundary line of an adjacent lane of the lane is the second highest, and other vehicles in the vicinity of the vehicle are the third highest.

2. The vehicle control device according to claim 1, wherein, in the orientation of the object target, the orientation of the road structure is the extending direction of the road structure, and the orientation of the moving body is the traveling direction of the moving body.

3. The vehicle control device according to claim 1 or 2, wherein, the vehicle control device further includes a driving control unit that generates a target trajectory based on the orientation of the vehicle estimated by the vehicle orientation estimation unit and controls the steering and acceleration / deceleration of the vehicle so as to travel along the generated target trajectory without depending on the operation of the driver of the vehicle.

4. The vehicle control device according to claim 1 or 2, wherein, the vehicle control device further includes a driving instruction unit that generates a target trajectory based on the orientation of the vehicle estimated by the vehicle orientation estimation unit and gives at least one of a steering instruction and an acceleration / deceleration instruction so that the occupant of the vehicle drives along the generated target trajectory.

5. A vehicle control method, wherein, the vehicle control method causes a computer mounted on a vehicle to perform the following processing: acquire an image obtained by photographing the external space of the vehicle; detect, by image processing, a plurality of types of object targets reflected in the image, including road structures and moving bodies; select one type of object target from the plurality of types of object targets based on a prescribed priority order and set a reference line along the orientation of the selected type of object target; estimate the angle formed by the reference line and the traveling orientation line of the vehicle as the orientation of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel, in the prescribed priority order, the boundary line of the lane in which the vehicle is traveling or is scheduled to travel is the highest, the boundary line of an adjacent lane of the lane is the second highest, and other vehicles in the vicinity of the vehicle are the third highest.

6. A storage medium that stores a program, wherein, The program causes a computer mounted on a vehicle to perform the following processing: Obtain an image obtained by photographing the external space of the vehicle; Detect, by image processing, multiple types of object targets reflected in the image, including road structures and moving objects; Based on a prescribed priority order, select one type of object target from the multiple types of object targets, and set a reference line along the orientation of the selected type of object target; Estimate the angle formed by the reference line and the traveling orientation line of the vehicle as the orientation of the vehicle relative to the lane in which the vehicle is traveling or is scheduled to travel; In the prescribed priority order, the boundary line of the lane in which the vehicle is traveling or is scheduled to travel is the highest, the boundary line of the adjacent lane of the lane is the second highest, and other vehicles in the vicinity of the vehicle are the third highest.

Images