Control device, control method, and program

Abstract

To improve convenience for an occupant (for example, a driver) of a mobile object.SOLUTION: A control device includes: a recognition section that recognizes a surrounding situation of a mobile object; and a control section that causes a display section to display a lane included in the surrounding situation recognized by the recognition section, and causes the display section to display, in a case in which a target route of the mobile object is on a side of a lane adjacent to a first lane where the mobile object is traveling, an image in which an image of an object of movement display showing movement of the mobile object to the lane side is superimposed on an image of an object of the adjacent lane. In a case in which the adjacent lane is a second lane connected to the first lane and widening ahead of a location where the mobile object is traveling in a traveling direction of the mobile object, the control section causes the display section to display an image of an object of the second lane even if the recognition section has not recognized the second lane when the display section is caused to display the image of the object of the movement display.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a control device, a control method, and a program. [Background technology]

[0002] In recent years, efforts to provide sustainable transportation systems that take various situations into consideration have become more active. To achieve this, efforts are being made to further improve traffic safety and convenience through research and development of driving assistance technologies. A display control device has been disclosed that, when a lane change is recommended based on route information, displays first content that provides guidance on the lane change, and then displays second content that provides guidance on recommended driving operations after the lane change before the display of the first content ends (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-094965 Summary of the Invention [Problem to be solved by the invention]

[0004] In the conventional technology, it was not possible to display information according to the control of the moving body on the display unit, which sometimes resulted in poor user convenience. For example, when a road does not exist, it is not possible to display the road to which the lane change is to be made, which can cause delays in displaying lane change guidance or displaying guidance in a location where no road exists, which can cause discomfort to the user. This can result in poor user convenience.

[0005] The present invention has been made in consideration of the above circumstances, and an object of the present invention is to provide a control device, a control method, and a program that can improve convenience for passengers (e.g., drivers) of a moving vehicle, thereby contributing to the development of a sustainable transportation system. In concrete terms, the present invention can display guidance displays at appropriate times and without causing discomfort. [Means for solving the problem]

[0006] The control device, control method, and program according to the present invention employ the following configuration. (1): A control device according to one embodiment of the present invention comprises a recognition unit that recognizes the situation around a moving body; and a control unit that causes a display unit to display lanes included in the situation around the moving body recognized by the recognition unit, and when the target route of the moving body is on the side of a lane adjacent to a first lane in which the moving body is traveling, causes the display unit to display an image in which an image of an object in the adjacent lane is superimposed on an image of an object in the adjacent lane, the image of an object displayed as a movement representing the movement of the moving body toward the first lane; and when the adjacent lane is a second lane that connects to the first lane and increases further forward in the direction of travel of the moving body than the position in which the moving body is traveling, the control unit causes the display unit to display an image of the object in the second lane, even if the recognition unit does not recognize the second lane when causing the display unit to display the image of the object displayed as a movement.

[0007] (2): In the above aspect (1), the control unit determines, based on road information, that the second lane is in the direction of travel of the moving body and displays an image of the object of the second lane on the display unit.

[0008] (3) In the aspect (1) above, the control unit causes the display unit to simultaneously display the image of the object in the second lane and the image of the object displayed in the movement display.

[0009] (4): In any of the aspects of (1) above, the control unit causes the display unit to display an image of the object in the second lane, and then causes the display unit to display an image of the object in the moving display.

[0010] (5): In the above aspect (1), an automobile lane change control unit is further provided that executes automobile lane change control to automatically change lanes of the moving body to the second lane, and when the automobile lane change control is being executed, the control unit displays an image of the object in the second lane superimposed on an image of the object displayed in the moving state on the display unit.

[0011] (6): In the above aspect (1), even if the recognition unit cannot recognize the second lane because the moving body is located a predetermined distance before the start of the second lane, when the control unit is planning to change lanes to the second lane and is notifying the occupant of the moving body of the lane change to the second lane, the control unit causes the display unit to display an image including a first image of an object of the moving body, a second image of an object in the second lane, and a third image of an object of the moving display associated with the first image and the second image.

[0012] (7): In the aspect (6) above, when an occupant of the moving body gives an instruction to an automobile lane change control unit, which displays an image of an object in the lane recognized by the recognition unit on the display unit, and automatically changes lanes for the moving body, when the moving body reaches a position a first distance from the second lane, even if the second lane is not included in the lanes recognized by the recognition unit, the control unit displays an image of an object in the second lane on the display unit and then displays an image of an object in the moving display on the display unit indicating that the moving body will move into the second lane, based on information indicating the existence of the second lane obtained from a navigation device, or displays the image of the object in the moving display on the display unit at the timing when the image of the object in the second lane is displayed.

[0013] (8): A control method according to another aspect of the present invention includes a process in which a computer recognizes the situation around a moving body, a process in which the computer displays lanes included in the situation around the moving body on a display unit, a process in which, when the target route of the moving body is on the side of a lane adjacent to a first lane in which the moving body is traveling, the computer displays on the display unit an image in which an image of an object in the adjacent lane is superimposed on an image of an object in the adjacent lane, the image of an object in the moving display representing the movement of the moving body toward the first lane, and a process in which, when the adjacent lane is a second lane that connects to the first lane and increases further forward in the direction of travel of the moving body than the position in which the moving body is traveling, the computer displays on the display unit an image of the object in the second lane, even if the second lane is not recognized in the recognition process when the computer displays on the display unit the image of the object in the moving display.

[0014] (9): Another aspect of the present invention provides a program for causing a computer to execute the following processes: a process for recognizing the situation around a moving body; a process for displaying lanes included in the situation around the moving body on a display unit; a process for displaying, when the target route of the moving body is on the side of a lane adjacent to a first lane in which the moving body is traveling, an image in which an image of an object in the adjacent lane is superimposed on an image of an object in the adjacent lane, the image of an object in the moving display representing the movement of the moving body toward the lane; and a process for displaying, when the adjacent lane is a second lane that connects to the first lane and increases further forward in the direction of travel of the moving body than the position in which the moving body is traveling, an image of the object in the second lane on the display unit, even if the second lane is not recognized in the recognition process when the image of the object in the moving display is displayed on the display unit. [Effects of the Invention]

[0015] According to the aspects (1) to (9), it is possible to improve user convenience. By displaying the image of the object in the second lane in accordance with the timing at which the image of the object displayed in the moving display is displayed, it is possible to prevent the occupant from feeling uncomfortable when the image of the object in the destination lane is not displayed even though the image of the object displayed in the moving display is displayed.

[0016] According to the second aspect, an image of an object in the second lane can be displayed more reliably based on road information.

[0017] According to the third aspect, the image of the moving object and the image of the object in the second lane are displayed simultaneously in a linked manner, which more reliably prevents the occupant from feeling uncomfortable. For example, it is possible to prevent only the image of the object in the second lane or only the image of the moving object from being displayed.

[0018] According to the fourth aspect, the image of the object in the second lane is displayed first, and then the image of the object displayed as moving is displayed, which further reduces the sense of discomfort felt by the occupant.

[0019] According to the aspect (5), when an automatic lane change is made, the image of a moving object is prevented from being displayed in a direction where a second lane does not exist, so that the lane change is made automatically with the passengers feeling at ease.

[0020] According to the sixth aspect, when the vehicle changes lanes, appropriate information can be provided to the occupant in accordance with the change in the position of the vehicle. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a configuration diagram of a vehicle system 1 that uses a vehicle control system according to an embodiment. [Figure 2] FIG. 1 is a diagram (part 1) for explaining a comparative example. [Figure 3] FIG. 10 is a diagram (part 2) for explaining a comparative example. [Figure 4] 5A and 5B are diagrams for explaining an image displayed on a display unit of the embodiment. [Figure 5] 3 is a flowchart showing an example of the flow of processing executed by the driving assistance device 100. DETAILED DESCRIPTION OF THE INVENTION

[0022] [Overall configuration] FIG. 1 is a configuration diagram of a vehicle system 1 that uses a vehicle control system according to an embodiment. The vehicle on which the vehicle system 1 is mounted may be, for example, a two-wheeled, three-wheeled, or four-wheeled vehicle, and its drive source may be an internal combustion engine such as a diesel engine or a gasoline engine, an electric motor, or a combination of these. The electric motor operates using power generated by a generator connected to the internal combustion engine, or power discharged from a secondary battery or a fuel cell. This embodiment will be described as being applied to a vehicle, but may also be applied to other moving objects instead of a vehicle.

[0023] The vehicle system 1 includes, for example, a camera 10, a radar device 12, a LIDAR (Light Detection and Ranging) 14, an object recognition device 16, a communication device 20, an HMI (Human Machine Interface) 30, vehicle sensors 40, a navigation device 50, an MPU 60, an operator 80, a direction indicator 90, a driving assistance device 100, a driving force output device 200, a braking device 210, and a steering device 220. These devices and equipment are connected to each other via multiplex communication lines such as a CAN (Controller Area Network) communication line, serial communication lines, a wireless communication network, etc. The configuration shown in FIG. 1 is merely an example, and some of the configuration may be omitted, or other configurations may be added. The driving assistance device 100 is an example of a "controller."

[0024] The camera 10 is, for example, a digital camera using a solid-state imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 10 is attached to any location of a vehicle (hereinafter referred to as vehicle M) in which the vehicle system 1 is installed. When capturing an image of the front, the camera 10 is attached to the top of the front windshield, the back of the rearview mirror, or the like. The camera 10, for example, periodically captures images of the periphery of the vehicle M. The camera 10 may be a stereo camera.

[0025] The radar device 12 emits radio waves such as millimeter waves around the vehicle M and detects radio waves reflected by an object (reflected waves) to detect at least the position (distance and direction) of the object. The radar device 12 is attached to any location on the vehicle M. The radar device 12 may detect the position and speed of an object using an FM-CW (Frequency Modulated Continuous Wave) method.

[0026] The LIDAR 14 irradiates the surroundings of the vehicle M with light (or electromagnetic waves with wavelengths similar to light) and measures the scattered light. The LIDAR 14 detects the distance to the target based on the time between light emission and light reception. The irradiated light is, for example, pulsed laser light. The LIDAR 14 may be attached to any location on the vehicle M.

[0027] The object recognition device 16 performs sensor fusion processing on the detection results from some or all of the camera 10, the radar device 12, and the LIDAR 14 to recognize the position, type, speed, etc. of the object. The object recognition device 16 outputs the recognition results to the driving assistance device 100. The object recognition device 16 may output the detection results from the camera 10, the radar device 12, and the LIDAR 14 directly to the driving assistance device 100. The object recognition device 16 may be omitted from the vehicle system 1.

[0028] The communication device 20 communicates with other vehicles in the vicinity of the vehicle M, for example, using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), etc., or communicates with various server devices via a wireless base station.

[0029] The HMI 30 presents various information to the occupants of the vehicle M and accepts input operations by the occupants. The HMI 30 includes various display devices, speakers, buzzers, touch panels, switches, keys, etc. The HMI 30 is equipped with a display device. The display device is, for example, a so-called multi-information display, provided in the center of the instrument panel of the vehicle M, and displays various information about the vehicle M, such as a speedometer that indicates the traveling speed of the vehicle M or a tachometer that indicates the rotation speed (rotational speed) of the internal combustion engine equipped in the vehicle M.

[0030] The vehicle sensor 40 includes a vehicle speed sensor that detects the speed of the vehicle M, an acceleration sensor that detects the acceleration, a yaw rate sensor that detects the angular velocity around a vertical axis, a direction sensor that detects the direction of the vehicle M, and the like.

[0031] The navigation device 50 includes, for example, a GNSS (Global Navigation Satellite System) receiver 51, a navigation HMI 52, and a route determination unit 53. The navigation device 50 stores first map information 54 in a storage device such as a hard disk drive (HDD) or flash memory. The GNSS receiver 51 identifies the position of the vehicle M based on signals received from GNSS satellites. The position of the vehicle M may be identified or supplemented by an inertial navigation system (INS) that uses the output of the vehicle sensors 40. The navigation HMI 52 includes a display device, a speaker, a touch panel, keys, etc. The navigation HMI 52 may share some or all of the components with the HMI 30 described above. The route determination unit 53 determines, for example, a route (hereinafter, a route on a map) from the position of the vehicle M identified by the GNSS receiver 51 (or any input position) to a destination input by the occupant using the navigation HMI 52, with reference to the first map information 54. The first map information 54 is information that represents road shapes using, for example, links indicating roads and nodes connected by the links. The first map information 54 may also include information such as road curvature and POI (Point of Interest) information. The route on the map is output to the MPU 60. The navigation device 50 may provide route guidance using the navigation HMI 52 based on the route on the map. The navigation device 50 may be realized, for example, by the functions of a terminal device such as a smartphone or tablet device owned by the occupant. The navigation device 50 may transmit the current position and destination to a navigation server via the communication device 20 and obtain a route equivalent to the route on the map from the navigation server.

[0032] The MPU 60 includes, for example, a recommended lane determination unit 61 and stores second map information 62 in a storage device such as an HDD or flash memory. The recommended lane determination unit 61 divides the route on the map provided by the navigation device 50 into a plurality of blocks (for example, every 100 m in the vehicle traveling direction) and determines a recommended lane for each block by referring to the second map information 62. The recommended lane determination unit 61 determines, for example, which lane from the left the vehicle M should take. When a branch point is present on the route on the map, the recommended lane determination unit 61 determines a recommended lane so that the vehicle M can travel on a reasonable route to the branch point. For example, when the vehicle M arrives a predetermined distance before the branch road on which the vehicle M is traveling, the recommended lane determination unit 61 determines a lane connecting to the branch road as the recommended lane. The recommended lane determination unit 61 and the second map information 62 may be functional units or information included in another device, such as the driving assistance device 100.

[0033] The second map information 62 is map information with higher accuracy than the first map information 54. The second map information 62 includes, for example, information on the center of lanes or information on lane boundaries. The second map information 62 may include road information, traffic regulation information, address information (address and postal code), facility information, telephone number information, etc. The second map information 62 may be updated as needed by the communication device 20 communicating with other devices.

[0034] The operators 80 include, for example, a steering wheel 82 as well as an accelerator pedal, a brake pedal, a shift lever, and other operators. The operators 80 are fitted with sensors that detect the amount of operation or whether or not an operation is performed, and the detection results are output to the driving assistance device 100 or some or all of the driving force output device 200, the brake device 210, and the steering device 220. The steering wheel 82 does not necessarily have to be annular, and may be in the form of an irregularly shaped steering wheel, a joystick, a button, or the like. The operators 80 include a first operator 84. The direction indicator 90 turns on or off in response to operation of the first operator 84.

[0035] The first operator 84 is, for example, a turn signal lever switch. For example, when the driver operates the first operator 84, the direction indicator 90 lights up in response to the operation. When the driver performs a predetermined operation on the first operator 84, the control unit 150 activates an ALC (Auto Lane Change) function and causes the vehicle M to perform an automatic lane change. The predetermined operation is an operation that triggers activation of the ALC function. The predetermined operation is, for example, operating the turn signal lever switch in the direction of a desired lane change for a predetermined period of time, or pressing the turn signal lever switch to a predetermined position. More specifically, the predetermined operation is operating the turn signal lever switch in the direction of a desired lane change while maintaining a predetermined position for a predetermined period of time. The first operator 84 may be another embodiment, such as a button, instead of a turn signal lever switch. The ALC function may also be activated by operating another operator, such as a predetermined button.

[0036] The driving assistance device 100 includes, for example, a recognition unit 110 and a control unit 150. The recognition unit 110 and the control unit 150 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), a GPU (Graphics Processing Unit), or an SOC (System On Chip), or may be realized by a combination of software and hardware. The program may be stored in advance in a storage device (a storage device having a non-transitory storage medium) such as an HDD or flash memory of the driving assistance device 100, or may be stored in a removable storage medium such as a DVD or CD-ROM, and installed in the HDD or flash memory of the driving assistance device 100 by inserting the storage medium (non-transitory storage medium) into a drive device.

[0037] The recognition unit 110 recognizes the position, speed, acceleration, and other states of objects around the vehicle M based on information input from the camera 10, the radar device 12, and the LIDAR 14 via the object recognition device 16. The position of an object is recognized as a position on an absolute coordinate system with a representative point of the vehicle M (such as the center of gravity or the center of the drive shaft) as the origin, and is used for control. The position of an object may be represented by a representative point such as the center of gravity or a corner of the object, or may be represented by an area. The "state" of an object may include the acceleration or jerk of the object, or the "behavioral state" (for example, whether or not the object is changing lanes or is about to change lanes).

[0038] The recognition unit 110 recognizes, for example, the lane in which the vehicle M is traveling (driving lane). For example, the recognition unit 110 recognizes the driving lane by comparing the pattern of road dividing lines (e.g., an arrangement of solid lines and dashed lines) obtained from the second map information 62 with the pattern of road dividing lines around the vehicle M recognized from an image captured by the camera 10. The recognition unit 110 may recognize the driving lane by recognizing road boundaries (road boundaries) including not only road dividing lines but also road dividing lines, shoulders, curbs, medians, guardrails, etc. In this recognition, the position of the vehicle M obtained from the navigation device 50 and the processing results by the INS may be taken into consideration. The recognition unit 110 recognizes stop lines, obstacles, red lights, toll booths, and other road phenomena.

[0039] When recognizing the driving lane, the recognition unit 110 recognizes the position and orientation of the vehicle M with respect to the driving lane. For example, the recognition unit 110 may recognize the deviation of the reference point of the vehicle M from the center of the lane and the angle it forms with a line connecting the centers of the lanes in the traveling direction of the vehicle M as the relative position and orientation of the vehicle M with respect to the driving lane. Alternatively, the recognition unit 110 may recognize the position of the reference point of the vehicle M with respect to either side edge of the driving lane (a road dividing line or a road boundary) as the relative position of the vehicle M with respect to the driving lane.

[0040] The control unit 150 executes driving assistance control. For example, the control unit 150 automatically controls the driving force output device 200 and the brake device 210 without relying on the driver's operation, thereby automatically controlling the speed of the vehicle M. The control unit 150 executes so-called ACC (Adaptive Cruise Control). The control unit 150 controls the vehicle M to travel at a set speed, or to travel by following a leading vehicle at a predetermined distance from the leading vehicle.

[0041] The control unit 150 controls the steering device 220 so that the vehicle M does not deviate from the driving lane. For example, the control unit 150 controls the steering device 220 so that the vehicle M travels in the center or near the center of the driving lane recognized by the recognition unit 110. Hereinafter, this control may be referred to as "lane keeping control." The control unit 150 executes hands-on lane keeping control and hands-off lane keeping control.

[0042] Hands-on lane keeping control is a control that is executed when the driver is gripping the steering wheel (when a steering grip sensor (not shown) detects that the driver is gripping the steering wheel). The conditions under which hands-on lane keeping control can be executed are less stringent than the conditions under which hands-off lane keeping control can be executed.

[0043] Hands-off lane keeping control is a control that is executed when the driver is not gripping the steering wheel (when a steering grip sensor, not shown, does not detect that the driver is gripping the steering wheel). Hands-off lane keeping control can be executed, for example, when the following conditions are met: the speed of vehicle M is equal to or greater than a predetermined speed, vehicle M is traveling on a predetermined road (for example, a road or type of road that has been set in advance as one for which hands-off lane keeping control can be executed), and the driver is monitoring the road ahead. When the driver is monitoring the road ahead, hands-off lane keeping control is executed, and when the driver is not monitoring the road ahead, hands-off lane keeping control is not executed or is stopped.

[0044] The above-described conditions under which the hands-on lane keeping control and the hands-off lane keeping control can be executed are merely examples, and other conditions (for example, vehicle M following a vehicle ahead) may be included, or some conditions may be omitted. The conditions under which the hands-on lane keeping control can be executed may be looser than the conditions under which the hands-off lane keeping control can be executed (the conditions under which the hands-off lane keeping control can be executed may be stricter than the conditions under which the hands-on lane keeping control can be executed). Whether the driver is monitoring the road ahead is recognized by driving assistance device 100 based on an image captured by a camera (not shown) that captures an image of the driver.

[0045] The control unit 150 includes a processing unit 152 and an assistance control unit 154 (lane change control unit). The control unit 150 automatically changes lanes of the vehicle M. Details of these will be described later. The automatic lane change control may be conditioned on the execution of hands-off lane keeping control or hands-on lane keeping control.

[0046] The driving force output device 200 outputs a driving force (torque) to the driving wheels for driving the vehicle M. The driving force output device 200 includes, for example, a combination of an internal combustion engine, an electric motor, a transmission, etc., and an ECU that controls these. The ECU controls the above components in accordance with information input from the driving assistance device 100 or information input from the operator 80.

[0047] The brake device 210 includes, for example, a brake caliper, a cylinder that transmits hydraulic pressure to the brake caliper, an electric motor that generates hydraulic pressure in the cylinder, and a brake ECU. The brake ECU controls the electric motor in accordance with information input from the driving assistance device 100 or information input from the operator 80, so that a brake torque corresponding to the braking operation is output to each wheel.

[0048] The steering device 220 includes, for example, a steering ECU and an electric motor. The electric motor changes the direction of the steered wheels by applying a force to, for example, a rack and pinion mechanism. The steering ECU drives the electric motor to change the direction of the steered wheels in accordance with information input from the driving assistance device 100 or information input from the operator 80.

[0049] [Comparative Example] FIG. 2 is a diagram (part 1) for explaining a comparative example. For example, a vehicle M is traveling on a first lane L1. The vehicle M is scheduled to change lanes to a second lane L2 adjacent to the first lane L1. The second lane L2 is a branch lane branching from the first lane L1. For example, the following control is executed when the vehicle M is set to travel on the second lane L2 as a target route. The following control is also executed when an occupant (driver) of the vehicle M performs an operation to instruct the vehicle M to change lanes to the second lane L2 a predetermined distance before the junction point where the first lane L1 and the second lane L2 connect. The instructing operation is, for example, any operation such as operating a predetermined lane change reservation button or operating the first operator 84. For example, branching assistance control is executed in which a lane change to the branch lane is automatically performed by operating the reservation button.

[0050] For example, when approaching a branching lane, the assistance control unit 154 automatically decelerates, turns on the turn indicator 90 at an appropriate timing, and automatically controls the steering without the driver's operation to change the vehicle M into the branching lane. This assistance control may be performed, for example, when hands-off lane keeping control or hands-on lane keeping control is being executed.

[0051] The processing unit 152 displays the lane marks included in the recognition result of the recognition unit 110 (for example, an image captured by the camera 10) on the display unit of the HMI 30. The display unit is, for example, a display unit provided in a position that is easily visible to the driver. The processing unit 152 converts, for example, the environment and objects around the vehicle M, such as the lane marks and leading vehicles recognized by the recognition unit 110, into images, and displays the converted images of the environment and objects on the display unit. The display unit is, for example, a meter display or multi-information display of the HMI 30, or a display unit provided in a position that is easily visible to the driver.

[0052] Time T is the timing when vehicle M arrives a first predetermined distance before the connection position (the position at time T+3). Time T+1 is the timing when vehicle M arrives a second predetermined distance before the connection position. The second predetermined distance is shorter than the first predetermined distance. Time T+2 is the timing when vehicle M arrives a third predetermined distance before the connection position. The third predetermined distance is shorter than the second predetermined distance. At time T+2, the turn signal 90 is on. Time T+3 is the timing when vehicle M starts to change lanes to the second lane L2 (or the timing just before starting).

[0053] The period from time T to time T+1 is a period in which the driving assistance device 100 starts guiding the vehicle M to change lanes to the lane change destination. The period from time T+1 to time T+2 is a period in which the driving assistance device 100 decelerates the vehicle M in advance to change lanes to the branch road.

[0054] Before time T, image IM1 is displayed on the display unit. Between time T and time T+1, image IM2 is displayed on the display unit, and then image IM3 is displayed. Between time T+1 and time T+2, images IM4 and IM5 are displayed on the display unit, in that order. After time T+2, images IM6, IM7, and IM8 are displayed on the display unit, in that order.

[0055] The vehicle included in the image is an example of an image of an object of vehicle M. The lane in which the vehicle included in the image is traveling is an example of an image of an object of the first lane L1. The lane adjacent to the lane in which the vehicle included in the image is traveling (the lane adjacent to the right) is an example of an image of an object of the second lane L2. The arrow extending from the vehicle included in image IM3 (image IM3# described below) or image IM5 (image IM5# described below) is an example of an image of an object displayed as moving.

[0056] Image IM1 includes information indicating the first lane L1, the third lane L3, vehicle M (the vehicle at the bottom of the image) recognized by the recognition unit 110, and the future trajectory of vehicle M. The third lane L3 is a lane adjacent to the first lane L1. When image IM1 is displayed, navigation linkage is in effect. Navigation linkage means that the processing unit 152 is using information from the navigation device 50 to display an image on the display unit, or is capable of displaying an image on the display unit.

[0057] Image IM2 is an image displayed using navigation linkage. The same applies to images IM3-IM10, which will be described later. When the processing unit 152 determines from the navigation device 50 that a position where vehicle M will change lanes is approaching, it causes the display unit to display information indicating that vehicle M will change lanes. Image IM2 includes, for example, an arrow indicating the direction of the lane change. After image IM2 is displayed, when vehicle M approaches the connection position, the processing unit 152 causes image IM3 to be displayed on the display unit. Image IM3 is an image that includes information that associates information (arrow) indicating the direction in which vehicle M will change lanes with vehicle M in image IM1.

[0058] Image IM4 includes information indicating the direction in which vehicle M will change lanes and information indicating that vehicle M will decelerate in order to exit into the branching lane, thereby enabling the driver to recognize that vehicle M will decelerate. Image IM5 is an image similar to image IM3.

[0059] In addition to the information of images IM3 and IM6, image IM6 includes an image of the second lane L2 recognized by the recognition unit 110. As the vehicle M approaches the second lane L2, the recognition unit 110 becomes able to recognize the second lane L2, and therefore the image of the second lane L2 is displayed on the display unit.

[0060] Image IM7 is an image of image IM6 in which the turn signals of vehicle M are on. Image IM8 is an image of image IM7 in which the trajectory of vehicle M is shown moving from the first lane L1 to the second lane L2.

[0061] As described above, the period up to time T+2 and from time T+4 onwards is a period in which the occupant can selectively change the image displayed on the display unit. From time T+2 to time T+4, the occupant can select content (preset content) different from the content shown in Figure 2 or the content shown in Figure 3 to display on the display unit.

[0062] FIG. 3 is a diagram (part 2) for explaining a comparative example. FIG. 3 shows a scene after time T+2 shown in FIG. 2. At time T+3, image IM8 is displayed. At this time, or thereafter, vehicle M starts changing lanes. At time T+4, vehicle M changes lanes and moves to the second lane L2, and image IM9 is displayed. Image IM9 is the same as image IM1. At time T+5, an image similar to image IM9 or image IM10 is displayed. Image IM10 is an image that includes the lane in which vehicle M is traveling and the vehicle ahead. After image IM10 is displayed, when vehicle M approaches a position where the curvature of the second lane L2 increases, image IM11 is displayed. Image IM11 is information indicating that acceleration of vehicle M is being suppressed in accordance with the curvature and information indicating that acceleration suppression can be released by an access operation.

[0063] In the image IM3 or IM5 described above, the lateral movement display is displayed in an area of ​​the image where no lane exists. This may cause discomfort to the occupant. In response to this, in this embodiment, the lateral movement display is superimposed on the lane as follows, thereby reducing the discomfort felt by the occupant. This improves user convenience.

[0064] [Display in this embodiment] The processing unit 152 causes the display unit to display lanes included in the surrounding situation of the vehicle M recognized by the recognition unit 110. When the target route of the vehicle M is on the side of a lane adjacent to the first lane on which the vehicle M is traveling, the processing unit 152 causes the display unit to display an image in which an image of an object in the adjacent lane is superimposed with an image of an object in the adjacent lane, the image representing the movement of the vehicle M toward the lane.

[0065] When the adjacent lane is a second lane that connects to the first lane and increases further forward in the direction of travel of vehicle M than the position at which vehicle M is traveling, processing unit 152 causes the display unit to display an image of the object in the second lane even if recognition unit 110 does not recognize the second lane when displaying an image of the moving object on the display unit.

[0066] For example, even if the recognition unit 110 cannot recognize the second lane because the vehicle M is located a predetermined distance before the start of the second lane, when the processing unit 152 is planning to change lanes to the second lane and is notifying the occupants of the vehicle M of the lane change to the second lane, the processing unit 152 causes the display unit to display an image including an image of an object of the vehicle M, an image of an object of the second lane, and an image of a moving object associated with the image of the object of the vehicle M and the image of the object of the second lane.

[0067] Based on the road information, the processing unit 152 determines that the second lane is in the traveling direction and displays the second lane on the display unit. The road information is, for example, information provided by the navigation device 50 or map information including information about roads (position information of branching roads) stored in the storage unit or provided via communication.

[0068] The processing unit 152 may display an image of the object in the second lane and an image of the object in the moving display on the display unit simultaneously, or may display an image of the object in the second lane on the display unit and then display an image of the object in the moving display on the display unit.

[0069] In a case where an occupant of vehicle M has instructed assistance control unit 154 to automatically change lanes, even if the second lane is not included in the lanes recognized by recognition unit 110 when vehicle M reaches a position a first distance from the second lane, processing unit 152 displays an image of an object in the second lane on the display unit, and then displays an image of an object in the second lane indicating that vehicle M will move into the second lane, based on information indicating the existence of the second lane obtained from navigation device 50, or displays an image of an object in the second lane indicating movement on the display unit at the timing when the image of the object in the second lane is displayed. "When vehicle M reaches a position a first distance from the second lane" refers to the timing when image IM3# or image IM5# in FIG. 4, which will be described later, is displayed.

[0070] 4 is a diagram for explaining images displayed on the display unit of the embodiment. Explanations similar to those of FIG. 2 or 3 will be omitted. In this embodiment, image IM3# is displayed instead of image IM3, and image IM5# is displayed instead of image IM5.

[0071] Image IM3# displays, for example, information about image IM3 as well as an image of an object in the second lane L2 and an image of an object on the road dividing line Ld in the second lane L2. The image of the second lane L2 is not the second lane L2 recognized by the recognition unit 110, but information acquired by the processing unit 152 from the first map information 54 in cooperation with the navigation device 50 (or information acquired from the map information in the storage unit). Even if the recognition unit 110 does not recognize the second lane L2, the processing unit 152 displays the image of the second lane L2 based on the timing of displaying the movement display.

[0072] As described above, the processing unit 152 causes the display unit to display the second lane L2, which is not recognized by the recognition unit 110, at an appropriate timing, thereby preventing a movement display from being displayed in an area of ​​the image where no lane exists. This prevents the occupants of the vehicle M from feeling uncomfortable, improving convenience for the occupants of the vehicle M.

[0073] For example, the above processing may be performed when the driver of the vehicle manually changes lanes, or may be applied when the vehicle M automatically changes lanes. For example, the processing of this embodiment may be applied when the vehicle M automatically decelerates along the target route, turns on the turn indicators 90, and the assistance control unit 154 automatically executes automated lane change control to cause the vehicle M to change lanes. In this case, when automated lane change control is executed, the processing unit 152 displays an image of the object in the second lane and an image of the object displayed as moving superimposed on each other on the display unit. This prevents the image of the object displayed as moving from being displayed in a direction where the second lane does not exist when the lane is changed automatically, thereby improving the sense of security of the occupants.

[0074] [flowchart] 5 is a flowchart showing an example of the flow of processing executed by the driving assistance device 100. First, the processing unit 152 acquires position information of the vehicle M (step S100). Next, the processing unit 152 acquires navigation information (step S102). The navigation information includes information on the destination set in the navigation device 50, information on the target route for the vehicle M guided by the navigation device 50, and the structure of roads around the target route. The processing of steps S100 and S102 is repeatedly executed at predetermined intervals.

[0075] Next, the processing unit 152 determines whether or not a target route has been set for a branch route based on the information acquired in steps S100 and S102 (step S104). If the target route has not been set for a branch route, the process returns to step S100. If the target route has been set for a branch route, the processing unit 152 determines whether or not the vehicle M has arrived a predetermined distance before the branch route, which is the target route (step S106). If the vehicle M has not arrived a predetermined distance before the branch route, which is the target route, the process returns to step S100.

[0076] When the vehicle M arrives a predetermined distance before the branch road, which is the target traveling route, the processing unit 152 causes the display unit to display an image corresponding to the position of the vehicle M relative to the position of the branch road (step S108). For example, the processing unit 152 causes the image to be displayed as described with reference to FIGS. 2 to 4. At the timing when the movement display is displayed on the display unit, the processing unit 152 uses the information on the branch road (branch lane) in the navigation information to display the movement display together with the image of the second lane L2.

[0077] Next, the processing unit 152 determines whether the recognition unit 110 has recognized a branch road (step S110). If the recognition unit 110 has not recognized a branch road, the process returns to step S108. If the recognition unit 110 has recognized a branch road, the processing unit 152 displays an image based on the recognition result of the recognition unit 110 (step S112). For example, the processing unit 152 displays a movement display and also displays information about the branch road included in the recognition result on the display unit.

[0078] Next, the processing unit 152 determines whether or not a predetermined condition is satisfied (step S114). The predetermined condition may be that the vehicle M has moved onto a branching road (moved to the center of the branching road), that a lane change to the branching road has been completed, or that the vehicle M has traveled a predetermined distance after moving onto the branching road or after changing lanes. If the predetermined condition is not satisfied, the processing returns to step S112. If the predetermined condition is satisfied, the processing unit 152 erases the movement display (step S116). This ends the processing of one routine of this flowchart.

[0079] Instead of the processing of step S110 described above, it may be determined whether a condition is satisfied. The condition may be any condition, such as the vehicle M reaching a predetermined position or the state of the vehicle M satisfying a condition. The state of the vehicle M satisfying a condition may mean, for example, that the predetermined condition described above is satisfied or that any condition, such as the turn signal being turned off, is satisfied.

[0080] As described above, the driving assistance device 100 can improve user convenience by using navigation information to display branching roads in accordance with the movement display on the image.

[0081] According to the embodiment described above, when the lane adjacent to the first lane is a second lane that connects to the first lane ahead of the position where vehicle M is traveling in the direction of travel of vehicle M, even if the recognition unit 110 does not recognize the second lane when displaying an image of an object in a moving display on the display unit, the driving assistance device 100 can improve user convenience by displaying an image of the object in the second lane on the display unit.

[0082] The above-described embodiment can be expressed as follows. a storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, Recognize the situation around the moving object, displaying the lanes included in the surrounding situation on a display unit; When the target route of the moving object is on a lane adjacent to a first lane on which the moving object is traveling, an image in which an image of an object in the adjacent lane is superimposed on an image of an object in the adjacent lane, the image representing the moving object's movement toward the first lane, is displayed on the display unit; When the adjacent lane is a second lane that connects to the first lane and increases ahead of the position where the moving body is traveling in the traveling direction of the moving body, Even if the recognition unit does not recognize the second lane when the image of the object of the movement display is displayed on the display unit, the image of the object of the second lane is displayed on the display unit. The control device is configured as follows.

[0083] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0084] 1 Vehicle Systems 10 Camera 80 Controls 84 1st controller 90 Turn signal 100 Driving assistance device 110 Recognition part 150 control section 152 Processing section 154 Support control section

Claims

1. a recognition unit that recognizes the surrounding situation of the moving object; displaying, on a display unit, lanes included in the surrounding situation recognized by the recognition unit; a control unit that, when the target route of the moving object is on a lane adjacent to a first lane on which the moving object is traveling, causes the display unit to display an image in which an image of an object in the adjacent lane is superimposed on an image of an object in the adjacent lane, the image representing the moving object's movement toward the lane, When the adjacent lane is a second lane that is connected to the first lane and increases forward in the traveling direction of the moving body from the position where the moving body is traveling, The control unit causes the display unit to display the image of the object of the second lane even if the recognition unit has not recognized the second lane when the control unit causes the display unit to display the image of the object of the movement display. Control device.

2. the control unit, based on road information, specifies that the second lane exists in the traveling direction of the moving object, and causes the display unit to display an image of an object in the second lane. The control device according to claim 1 .

3. the control unit causes the display unit to simultaneously display an image of the object in the second lane and an image of the object in the movement display. The control device according to claim 1 .

4. the control unit causes the display unit to display an image of the object in the second lane, and then causes the display unit to display an image of the object in the moving display. The control device according to claim 1 .

5. an automatic lane change control unit that executes automatic lane change control to automatically change the moving object to the second lane; the control unit causes the display unit to superimpose an image of the object in the second lane and an image of the object in the moving display when the automotive lane change control is being executed. The control device according to claim 1 .

6. The control unit Even if the recognition unit cannot recognize the second lane because the moving body is located a predetermined distance before the start point of the second lane, when the moving body plans to change lanes to the second lane and notifies an occupant of the moving body of the lane change to the second lane, an image including a first image of an object of the moving body, a second image of an object in the second lane, and a third image of an object of the movement display associated with the first image and the second image is displayed on the display unit. The control device according to claim 1 .

7. The control unit displaying an image of the lane object recognized by the recognition unit on a display unit; When an occupant of the moving body gives an instruction to automatically change lanes to an automobile lane change control unit that automatically changes lanes of the moving body, Even if the second lane is not included in the lanes recognized by the recognition unit when the vehicle reaches a position at a first distance from the second lane, Based on information indicating the existence of the second lane obtained from the navigation device, After displaying an image of the object in the second lane on a display unit, displaying an image of a moving display object indicating that the moving object is moving into the second lane on the display unit, or displaying an image of the moving object on the display unit at the same time as displaying the image of the object in the second lane; The control device according to claim 6.

8. The computer A process of recognizing the surrounding situation of the moving object; A process of displaying lanes included in the surrounding situation on a display unit; When the target route of the moving object is on a lane adjacent to a first lane on which the moving object is traveling, a process of displaying on the display unit an image in which an image of an object in the adjacent lane is superimposed with an image of an object in the adjacent lane, the object representing the moving object's movement toward the first lane; When the adjacent lane is a second lane that is connected to the first lane and increases forward in the traveling direction of the moving body from the position where the moving body is traveling, a process of displaying an image of the object of the second lane on the display unit even if the second lane has not been recognized in the recognition process when the image of the object of the movement display is displayed on the display unit; and A control method for performing

9. On the computer, A process of recognizing the surrounding situation of the moving object; A process of displaying lanes included in the surrounding situation on a display unit; When the target route of the moving object is on a lane adjacent to a first lane on which the moving object is traveling, a process of displaying on the display unit an image in which an image of an object in the adjacent lane is superimposed with an image of an object in the adjacent lane, the object representing the moving object's movement toward the first lane; When the adjacent lane is a second lane that is connected to the first lane and increases forward in the traveling direction of the moving body from the position where the moving body is traveling, a process of displaying an image of the object of the second lane on the display unit even if the second lane has not been recognized in the recognition process when the image of the object of the movement display is displayed on the display unit; and A program to execute.

Images