Vehicle control device, vehicle control method and program

Abstract

To provide a vehicle control device, a vehicle control method, and a program capable of enabling a vehicle occupant who views a display device during execution of a driving support control that performs a lane change accurately to recognize a situation in an area located in the direction of vehicle movement.SOLUTION: When performing driver assistance control to cause a vehicle 12 to change lanes toward one of the left or the right, a vehicle control unit generates a virtual viewpoint image 40 representing an area located in front of and at the one side of a vehicle occupant and displays the virtual viewpoint image 40 on a display device 23 provided in the vehicle based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located on the other side of the vehicle in the left and right directions from a center of the vehicle in a width direction.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a vehicle control device, a vehicle control method, and a program. [Background technology]

[0002] The vehicle described in Patent Document 1 below is equipped with a display device that can display a surrounding image including a road image showing the road on which the vehicle is traveling and other vehicle images showing other vehicles located around the vehicle when the vehicle executes LCA (Lane Change Assist Control). This surrounding image shows subjects (e.g., the road and other vehicles) in the area located in front of the vehicle. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 038903 Summary of the Invention [Problem to be solved by the invention]

[0004] The virtual viewpoint when generating the peripheral image displayed on the display device of Patent Document 1 is located at the center in the vehicle width direction in a plan view. In other words, the image located at the center in the vehicle width direction of the peripheral image represents a subject located in front of the center in the vehicle width direction of the vehicle. Furthermore, the image located on the left edge in the vehicle width direction of the peripheral image represents a subject located to the left of the left edge of the vehicle. Furthermore, the image located on the right edge in the vehicle width direction of the peripheral image represents a subject located to the right of the right edge of the vehicle.

[0005] As described above, the display device of Patent Document 1 displays a wide area in front of the vehicle, so Patent Document 1 has room for improvement in terms of allowing a vehicle occupant who looks at the display device while LCA is being performed to accurately recognize the situation in the area in the direction of vehicle movement.

[0006] Taking the above facts into consideration, the present invention aims to provide a vehicle control device, a vehicle control method, and a program that enable a vehicle occupant who views a display device while executing driving assistance control for lane changes to accurately recognize the situation in the area located in the direction of vehicle movement. [Means for solving the problem]

[0007] The vehicle control device of claim 1 includes a control unit that, when executing driving assistance control to change lanes of a vehicle to one of the left and right directions, generates a virtual viewpoint image representing an area located in front of and on the one side of an occupant of the vehicle based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located on the other side of the center of the vehicle in the left and right directions in a vehicle width direction, and displays the virtual viewpoint image on a display device provided in the vehicle; The virtual viewpoint is located at the same position as the front seat of the vehicle in the longitudinal direction of the vehicle or further forward than the front seat in the longitudinal direction of the vehicle in a plan view. .

[0008] The control unit of the vehicle control device of claim 1 displays a virtual viewpoint image on a display device when driving assistance control is executed to cause the vehicle to change lanes to one side, either left or right. The virtual viewpoint image is an image representing an area located in front of and to one side of a vehicle occupant, generated based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located to the other side, either left or right, of the center of the vehicle in the vehicle width direction. Therefore, the vehicle control device of claim 1 can allow a vehicle occupant who looks at the display device while driving assistance control for changing lanes is being executed to accurately recognize the situation of an area located in the direction of vehicle movement.

[0009] The vehicle control device of claim 2 is the same as claim 1, wherein the display device is located forward of the seat of the vehicle, the display device has a display area in which the virtual viewpoint image can be displayed, and the control unit displays the virtual viewpoint image in the area on one side of the center of the display area in the left-right direction.

[0010] The control unit of the vehicle control device of claim 2 displays the virtual viewpoint image in an area on one side of the center in the horizontal direction of the display area of ​​the display device. In this way, a vehicle occupant looking at the display device can easily intuitively recognize that the virtual viewpoint image displayed in the display area represents an area in the vehicle's traveling direction (lane change direction). Therefore, the vehicle control device of claim 2 makes it easy for a vehicle occupant looking at the display device to accurately recognize the situation in the area located in the vehicle's traveling direction.

[0011] A vehicle control device according to a third aspect of the present invention is the device according to the first or second aspect, wherein the control unit causes the display device to display an image representing the vehicle.

[0012] The control unit of the vehicle control device of claim 3 displays an image representing a vehicle on the display device. Therefore, the vehicle control device of claim 3 notifies the occupant who looks at the display device of the positional relationship between the vehicle in which he or she is riding and the other vehicle displayed on the display device. Easy to recognize.

[0013] Claim 4 a vehicle control method according to the present invention, when executing driving assistance control to change lanes to one side in a leftward or rightward direction, generating a virtual viewpoint image representing an area located in front of and on one side of an occupant of the vehicle based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located on the other side in a leftward or rightward direction from a center of the vehicle in a vehicle width direction, and displaying the virtual viewpoint image on a display device provided in the vehicle; The virtual viewpoint is located at the same position as the front seat of the vehicle in the longitudinal direction of the vehicle or further forward than the front seat in the longitudinal direction of the vehicle in a plan view. .

[0014] Claim 5 The program according to the present invention causes a computer to execute, when executing driving assistance control to change lanes to one side in a left or right direction, a process of generating a virtual viewpoint image representing an area located in front of and on one side of an occupant of the vehicle, based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located on the other side in a left or right direction from a center of the vehicle in a vehicle width direction, and a process of displaying the virtual viewpoint image on a display device provided in the vehicle; The virtual viewpoint is located at the same position as the front seat of the vehicle in the longitudinal direction of the vehicle or further forward than the front seat in the longitudinal direction of the vehicle in a plan view. . [Effects of the Invention]

[0015] As described above, the vehicle control device, vehicle control method, and program according to the present invention have the excellent effect of enabling a vehicle occupant who looks at the display device while driving assistance control for changing lanes is being executed to accurately recognize the situation in the area located in the direction of vehicle movement. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a diagram showing the interior of a vehicle equipped with a vehicle control device according to an embodiment; [Figure 2] FIG. 2 is a plan view of the vehicle shown in FIG. [Figure 3] FIG. 2 is a side view of the vehicle shown in FIG. [Figure 4] FIG. 2 is a diagram illustrating a hardware configuration of the vehicle shown in FIG. [Figure 5] FIG. 5 is a functional block diagram of the ECU shown in FIG. 4. [Figure 6] FIG. 2 is a plan view of the vehicle, surrounding vehicles, and road shown in FIG. [Figure 7] FIG. 10 shows a display device displaying an image representing surrounding vehicles and roads when LCA is not being performed. [Figure 8] FIG. 10 is a diagram showing the display device when LCA for changing lanes to the right is being executed. [Figure 9] FIG. 10 is a diagram showing the display device when LCA for changing lanes to the left is being executed. [Figure 10] 4 is a flowchart showing a process executed by a CPU of an ECU. DETAILED DESCRIPTION OF THE INVENTION

[0017] Hereinafter, embodiments of a vehicle control device, a vehicle control method, and a program according to the present invention will be described with reference to the drawings. In the drawings, an arrow FR indicates the front side in the longitudinal direction of the vehicle, an arrow LH indicates the left side in the lateral direction of the vehicle, and an arrow UP indicates the upper side in the vertical direction of the vehicle.

[0018] 1, a vehicle 12 equipped with the vehicle control device 10 includes an instrument panel 14 and a front windshield 15. A steering column 16 is provided on the instrument panel 14, and a steering wheel 18 is rotatably supported on the steering column 16. Furthermore, a turn signal lever 20 is movably supported on the right side of the steering column 16.

[0019] The turn signal lever 20 can rotate upward (counterclockwise) and downward (clockwise) relative to the steering column 16 around its base end (left end). The position shown in FIG. 1 is the initial position of the turn signal lever 20. When the driver (occupant; not shown) of the vehicle 12 applies an external force to the turn signal lever 20, the turn signal lever 20 rotates to a left LCA operating position above the initial position or to a right LCA operating position below the initial position (not shown). Furthermore, when the external force applied to the turn signal lever 20 in the left LCA operating position or right LCA operating position is released, the turn signal lever 20 automatically moves back to its initial position. Furthermore, the turn signal lever 20 can rotate to a left-side on position above the left-side LCA operating position and to a right-side on position below the right-side LCA operating position.

[0020] 1, a sensor unit 21 is provided on an upper portion of the interior surface of the front windshield 15. The sensor unit 21 includes a front center camera (camera) 21A that captures an image of a subject located in front of the front windshield 15 through the front windshield 15, a millimeter wave radar (not shown) that transmits a detection wave and receives a reflected wave, and a lidar (laser imaging detection and ranging) (not shown) that scans the area ahead of the vehicle 12.

[0021] 2, the vehicle 12 is further equipped with a front left camera (camera) 21B, a front right camera (camera) 21C, a left side camera (camera) 21D, and a right side camera (camera) 21E. The front central camera 21A, the front left camera 21B, and the front right camera 21C capture images of subjects located in an area forward of their respective positions. The left side camera 21D captures images of subjects located in an area to the left of the vehicle 12. The right side camera 21E captures images of subjects located in an area to the right of the vehicle 12. The front central camera 21A, the front left camera 21B, the front right camera 21C, the left side camera 21D, and the right side camera 21E each have a predetermined angle of view.

[0022] 4, the vehicle 12 has a GPS (Global Positioning System) receiver 22. The GPS receiver 22 receives GPS signals transmitted from GPS satellites to obtain information about the location where the vehicle 12 is traveling (hereinafter referred to as "location information").

[0023] As shown in Figures 1 and 4, the instrument panel 14 is provided with a display device 23. As shown in Figures 1 and 7 to 9, the display device 23 has a display area 24 that is slightly smaller than its outer shape. The display device 23 is located forward of the left and right front seats (seats) 13L, 13R of the vehicle 12. In other words, the display device 23 is located forward of the heads (eyeballs) of the occupants seated in the front seats 13L, 13R.

[0024] As shown in FIGS. 1 and 4, the instrument panel 14 is provided with a driving assistance operation device 25. The driving assistance operation device 25 is a device for causing the vehicle 12 to execute driving assistance control, which will be described later. When the driving assistance operation device 25 is in the ON state, the vehicle 12 can execute driving assistance control. When the driving assistance operation device 25 is in the OFF state, the vehicle 12 cannot execute driving assistance control.

[0025] As shown in FIG. 4, the vehicle 12 has an ECU (Electronic Control Unit) 26 as a hardware configuration.

[0026] The ECU 26 includes a CPU (Central Processing Unit) (controller) (computer) 26A, a ROM (Read Only Memory) (non-temporary recording medium) (recording medium) 26B, a RAM (Random Access Memory) 26C, a storage (non-temporary recording medium) (recording medium) 26D, a communication I / F 26E, and an input / output I / F 26F. The CPU 26A, the ROM 26B, the RAM 26C, the storage 26D, the communication I / F 26E, and the input / output I / F 26F are connected to each other via an internal bus 26Z so as to be able to communicate with each other.

[0027] The CPU 26A is a central processing unit that executes various programs and controls each part. The CPU 26A reads programs from the ROM 26B or the storage 26D and executes the programs using the RAM 26C as a work area. The CPU 26A controls each component and performs various arithmetic processing in accordance with the programs recorded in the ROM 26B or the storage 26D.

[0028] The ROM 26B stores various programs and various data. The RAM 26C temporarily stores programs or data as a working area. The storage 26D is configured with a storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and stores various programs and various data. For example, a navigation application having map data is installed in the ROM 26B or the storage 26D. In other words, the vehicle 12 is equipped with a navigation system. Furthermore, the ROM 26B or the storage 26D stores vehicle-related image data. The vehicle-related image data will be described later.

[0029] The communication I / F 26E is an interface for connecting via an external bus (not shown) to an ECU (not shown) other than the ECU 26. The interface uses a communication standard such as the CAN protocol.

[0030] The input / output I / F 26F is an interface for communicating with various devices, such as the front center camera 21A, the front left camera 21B, the front right camera 21C, the left side camera 21D, the right side camera 21E, millimeter wave radar, a lidar, a GPS receiver 22, a display device 23, a driving assistance operation device 25, and a group of actuators (described later).

[0031] 5 is a block diagram showing an example of the functional configuration of the ECU 26. The ECU 26 has, as its functional configuration, a turn signal control unit 261, a driving assistance control unit 262, and an image display control unit 263. The turn signal control unit 261, the driving assistance control unit 262, and the image display control unit 263 are realized by the CPU 26A reading and executing programs stored in the ROM 26B.

[0032] The turn signal control unit 261 controls left and right turn signals (not shown) according to the position of the turn signal lever 20. That is, when the turn signal lever 20 is in the left LCA operating position or the left lighting position, the left turn signal, which is a lighting fixture provided at the front end of the vehicle 12, is turned on by the control of the turn signal control unit 261. Also, when the turn signal lever 20 is in the right LCA operating position or the right lighting position, the right turn signal, which is a lighting fixture provided at the front end of the vehicle 12, is turned on by the control of the turn signal control unit 261.

[0033] When the driving assist operation device 25 is in the ON state, the driving assist control unit 262 uses a group of sensors and a group of actuators (not shown) provided in the vehicle 12 to cause the vehicle 12 to execute driving assist control at driving levels 1 to 5 defined by the Society of Automotive Engineers (SAE). Furthermore, when the driving assist operation device 25 is in the ON state, an occupant of the vehicle 12 can operate the driving assist operation device 25 to select the driving level and the driving assist control to be executed. The driving assist control in this embodiment includes, for example, adaptive cruise control (ACC), lane trace assist (LTA), and lane change assist (LCA). The group of sensors provided in the vehicle 12 includes a sensor unit 21 (front center camera 21A), a front left camera 21B, a front right camera 21C, a left side camera 21D, and a right side camera 21E. The group of actuators provided on the vehicle 12 includes a brake device, an electric power steering device having a steering wheel 18, various electric actuators for driving the internal combustion engine which is the drive source, and an electric motor which is the drive source.

[0034] Here, a brief explanation of LCA will be given. Like LTA, LCA is a lateral position control (lane width direction) of the vehicle 12 relative to the lane. LCA is initiated when driving assistance control of levels 1 to 3 is selected and the turn signal lever 20 is moved to the left LCA operating position or the right LCA operating position while LTA and ACC are being performed. Furthermore, when driving assistance control of level 5 (fully automated driving) is selected and a planned route for the vehicle 12 is set by the navigation system, LCA is initiated when the driving assistance control unit 262 determines that a lane change is necessary. When LCA is initiated, a predetermined LCA execution condition is met.

[0035] When LCA is initiated, the CPU 26A (driving assistance control unit 262) monitors the surroundings of the vehicle 12 based on information acquired from the sensors. After determining that the vehicle 12 can safely change lanes, the CPU 26A moves the vehicle 12 to the left or right. For example, when LCA is executed by moving the turn signal lever 20 to the left LCA operating position, the actuators are controlled so that the vehicle 12 moves from the driving lane, which is the lane in which the vehicle 12 is currently traveling, to the adjacent lane, which is the lane adjacent to the driving lane on the left. When LCA is executed by moving the turn signal lever 20 to the right LCA operating position, the actuators are controlled so that the vehicle 12 moves from the driving lane, which is the lane in which the vehicle 12 is currently traveling, to the adjacent lane, which is the lane adjacent to the driving lane on the right. When the vehicle 12 moves to a predetermined position in the adjacent lane on the left or right, the CPU 26A (driving assistance control unit 262) ends the LCA.

[0036] When a predetermined interruption condition is met during the execution of LCA, the driving assistance control unit 262 interrupts the LCA. For example, the interruption condition is met when the driving assistance control unit 262 determines that the predicted time until collision (TTC) of the vehicle 12 with another vehicle is less than a predetermined threshold while the LCA is being executed. When the interruption condition is met or when the LCA is terminated, the LCA execution condition is no longer met.

[0037] The image display control unit 263 identifies the road on which the vehicle 12 is traveling based on the car navigation system (map data) and the position information. Furthermore, the image display control unit 263 reads the map data from the car navigation system and displays an image of the road on which the vehicle 12 is currently traveling on the display device 23. For example, assume that the vehicle 12 is traveling on a road 50 shown in FIG. 6. The road 50 has a first lane 51, a second lane 52, and a third lane 53. The first lane 51 and the second lane 52 are separated by a dividing line 50A, and the second lane 52 and the third lane 53 are separated by a dividing line 50B. The arrow DR shown in FIG. 6 indicates the traveling direction of the vehicle 12. In this case, as shown in FIG. 7, the road image 30 displayed on the display device 23 has a first lane image 31, a second lane image 32, and a third lane image 33. The first lane image 31 and the second lane image 32 are separated by a lane marking image 30A, and the second lane image 32 and the third lane image 33 are separated by a lane marking image 30B. The image on the display device 23 shown in Fig. 7 is an image seen diagonally forward from a virtual viewpoint (not shown) directly above the vehicle 12.

[0038] Furthermore, when the LCA execution condition is not satisfied, the image display control unit 263 uses camera images (image data) representing subjects located around the vehicle 12 acquired by the front center camera 21A, the front left camera 21B, and the front right camera 21C and a pattern matching method to determine whether the camera images include surrounding vehicles located around the vehicle 12. Note that the surrounding vehicles include not only four-wheeled vehicles but also three-wheeled and two-wheeled vehicles. Furthermore, when it is determined that the camera images include surrounding vehicles, the image display control unit 263 acquires image data representing the surrounding vehicles from the vehicle-related image data. The vehicle-related image data in this embodiment includes passenger car image data representing four-wheeled passenger cars, truck image data representing four-wheeled trucks, and driving trajectory images. For example, as shown in FIG. 6, assume that vehicle 12 and two surrounding vehicles, passenger cars 55A and 55B, located in front of vehicle 12, are traveling in lane 2 52, another surrounding vehicle, motorcycle 55C, is traveling in lane 1 51, and yet another surrounding vehicle, truck 55D, is traveling in lane 3 53. In this case, image display control unit 263 recognizes the relative positions of vehicle 12 and each surrounding vehicle based on the detection results of the sensor group and camera images. Furthermore, as shown in FIG. 7, image display control unit 263 displays passenger car image data 35A, 35B, and 35C and truck image data 35D representing each surrounding vehicle on display device 23 based on the relative positions. The passenger car image data 35A, 35B, and 35C and truck image data 35D are each a specific color. For example, passenger car image data 35A, 35B, and 35C and truck image data 35D are white. Therefore, for example, the passenger car image data 35A, 35B, 35C and the truck image data 35D are displayed as white images on the display device 23. In this way, when the LCA execution condition is not met, images (passenger car image data 35A, 35B, 35C and truck image data 35D) that are different in color and shape from the actual surrounding vehicles are displayed on the display device 23.

[0039] On the other hand, when the LCA execution condition is met, the image display control unit 263 generates a virtual viewpoint image using camera images acquired by the front central camera 21A, the front left side camera 21B, the front right side camera 21C, the left side camera 21D and the right side camera 21E.

[0040] First, we will explain the virtual viewpoint image 40 generated by LCA when the vehicle 12 moves into the adjacent lane on the right. In this case, the image display control unit 263 generates image data that is the basis of the virtual viewpoint image 40 shown in FIG. 8 by combining camera images acquired by the front center camera 21A, the front right camera 21C, and the right side camera 21E and performing viewpoint conversion processing. The virtual viewpoint EP1 in this case is located to the left of the center line (center) CL that passes through the center of the vehicle 12 in the vehicle width direction in the front-to-rear direction in a planar view as shown in FIG. 2, and is located above the top end of the vehicle 12 in a side view as shown in FIG. 3. The range of the virtual viewpoint image 40 in a planar view is the shooting area between two straight lines L1L and L1R extending from the virtual viewpoint EP1 shown in FIG. 2. Note that the angle (angle of view) formed by the straight lines L1L and L1R in a planar view is θ1. The range of the virtual viewpoint image 40 in the side view is the imaging area between two straight lines LU and LD extending from the virtual viewpoint EP1 shown in Fig. 3. The angle (angle of view) formed by the straight lines LU and LD in the side view is θ1. Most of this imaging area is located in front of and to the right of the front windshield 15 and the heads of the occupants in the plan view.

[0041] 8, the image display control unit 263 displays a virtual viewpoint image 40 based on this image data in the display area 24 of the display device 23. At this time, the image display control unit 263 divides the display area 24 into a first area 24R, which is the right half, and a second area 24L, which is the left half. The image display control unit 263 also displays the virtual viewpoint image 40 in the first area 24R. As shown in FIG. 8, the virtual viewpoint image 40 includes a part (front) of the vehicle 12. The image display control unit 263 also reads a traveling trajectory image 47 (vehicle-related image data) representing a traveling trajectory for the vehicle 12 to perform a lane change, and displays the traveling trajectory image 47 on the display device 23.

[0042] Next, a virtual viewpoint image 45 generated by LCA when the vehicle 12 moves into the adjacent lane on the left will be described. In this case, the image display control unit 263 generates image data that is the basis of the virtual viewpoint image 45 shown in FIG. 9 by combining camera images acquired by the front center camera 21A, the front left camera 21B, and the left side camera 21D and performing viewpoint conversion processing. The virtual viewpoint EP2 in this case is located at the position shown in FIGS. 2 and 3. The virtual viewpoint EP2 is located to the right of the center line CL in a plan view as shown in FIG. 2, and is located above the top end of the vehicle 12 in a side view as shown in FIG. 3. The range of the virtual viewpoint image 45 in a plan view is the captured area between two straight lines L2L and L2R extending from the virtual viewpoint EP2 shown in FIG. 2. Note that the angle (angle of view) formed by the straight lines L2L and L2R in a plan view is θ1. The range of the virtual viewpoint image 45 in a side view is the captured area between two straight lines LU and LD extending from the virtual viewpoint EP2 shown in FIG. 3. Most of this photographing area is located in front of and to the left of the front windshield 15 and the heads of the occupants in a plan view.

[0043] The positions of the virtual viewpoints EP1 and EP2 may be set rearward from the positions shown in Figures 2 and 3. For example, in a side view, the virtual viewpoints EP1 and EP2 may be located rearward from the front seats 13L and 13R, or rearward from the rearmost seat provided in the vehicle 12. In this case, the left side camera 21D and the right side camera 21E may be provided in the vehicle 12 so as to be located rearward from the positions shown in Figure 2.

[0044] 9, the image display control unit 263 causes a virtual viewpoint image 45, which is an image based on this image data, to be displayed in the second area 24L of the display area 24 of the display device 23. As shown in FIG. 9, the virtual viewpoint image 45 includes a part (front part) of the vehicle 12. Furthermore, the image display control unit 263 reads a traveling trajectory image 48 (vehicle-related image data) representing a traveling trajectory for the vehicle 12 to execute a lane change, and causes the traveling trajectory image 48 to be displayed on the display device 23.

[0045] The GPS receiver 22, the display device 23, the driving support operation device 25, the ECU 26, the above-mentioned sensors, and the above-mentioned actuators are components of the vehicle control device 10.

[0046] (Action and effect) Next, the operation and effects of this embodiment will be described.

[0047] Next, a description will be given of the processing executed by the CPU 26A of the ECU 26. The CPU 26A repeatedly executes the processing of the flowchart shown in FIG.

[0048] In step S10 (hereinafter, the letter "step" will be omitted), the CPU 26A determines whether or not the LCA execution condition is met.

[0049] If the determination in S10 is Yes, the CPU 26A proceeds to S11 and executes the LCA.

[0050] Next, the CPU 26A proceeds to S12, where it generates image data that will be the basis of the virtual viewpoint images 40 and 45 according to the position of the turn signal lever 20.

[0051] After completing the process of S12, the CPU 26A proceeds to S13, where it causes the display device 23 to display the virtual viewpoint image 40 or the virtual viewpoint image 45.

[0052] After completing the process of S13, the CPU 26A proceeds to S14 and determines whether the LCA execution condition is met.

[0053] If the determination in S14 is Yes or if the determination in S10 is No, the CPU 26A proceeds to S15 and displays the passenger car image data on the display device 23 instead of the virtual viewpoint images 40, 45.

[0054] When the process of S15 is completed, the CPU 26A temporarily ends the process of the flowchart of FIG.

[0055] As described above, in this embodiment, when the vehicle 12 performs LCA to change lanes to one side, either left or right, the virtual viewpoint images 40, 45 are displayed on the display device 23. The virtual viewpoint images 40, 45 are generated by combining image data acquired by multiple cameras, including the front center camera 21A, the front left camera 21B, the front right camera 21C, the left side camera 21D, and the right side camera 21E, and performing viewpoint conversion processing. Furthermore, for example, when the vehicle 12 changes lanes to the right, the virtual viewpoint EP1 is located to the left of the center line CL in a planar view (the other side of the left or right direction), and the virtual viewpoint EP2 when the vehicle 12 changes lanes to the left is located to the right of the center line CL in a planar view. Therefore, for example, the virtual viewpoint image 40 when the vehicle 12 changes lanes to the right represents the areas ahead and to the right as seen by the occupant. In other words, the virtual viewpoint image 40 is an image that allows the occupant to easily recognize the travel path (travel path image 47) of the vehicle 12 when it changes lanes. Therefore, the occupant who looks at the display device 23 while the LCA is being performed can accurately recognize the situation of the area located in the direction of travel of the vehicle 12.

[0056] Furthermore, when the vehicle 12 changes lanes to the right, a virtual viewpoint image 40 is displayed in a first region 24R to the right of the center in the left-right direction of the display region 24. Furthermore, when the vehicle 12 changes lanes to the left, a virtual viewpoint image 45 is displayed in a second region 24L to the left of the center in the left-right direction of the display region 24. Therefore, an occupant who looks at the display device 23 while LCA is being performed can easily intuitively recognize that the virtual viewpoint images 40, 45 displayed in the display region 24 represent the region in the traveling direction (lane changing direction) of the vehicle 12.

[0057] Furthermore, while LCA is being performed, an image showing a part of the vehicle 12 is displayed in the first area 24R or the second area 24L. Therefore, an occupant who looks at the display device 23 can recognize the positional relationship between the vehicle 12 and surrounding vehicles. Therefore, compared to when the vehicle 12 is not displayed on the display device 23, there is less chance that an occupant who looks at the display device 23 while LCA is being performed will feel uneasy.

[0058] The vehicle control device 10, the vehicle control method, and the program according to the embodiment have been described above, but these can be modified in design as appropriate within the scope of the gist of the present invention.

[0059] For example, the cameras mounted on the vehicle 12 are not limited to those described above. For example, if the angle of view of the front central camera 21A is large, the vehicle 12 does not need to include the front left camera 21B and the front right camera 21C. In this case, for example, when the vehicle 12 changes lanes to the right, a virtual viewpoint image is formed using the camera images acquired by the front central camera 21A and the right side camera 21E. Also, when the vehicle 12 changes lanes to the left, a virtual viewpoint image is formed using the camera images acquired by the front central camera 21A and the left side camera 21D.

[0060] [Note] The vehicle control device of the present invention may be any combination of the following configurations 1 to 3. <Configuration 1> A vehicle control device including a control unit that, when executing driving assistance control to change lanes to one side, either left or right, generates a virtual viewpoint image representing an area located in front of and to the one side of an occupant of the vehicle based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located to the other side, either left or right, of the center of the vehicle's width, and displays the virtual viewpoint image on a display device provided in the vehicle. <Configuration 2> A vehicle control device in which the display device is located forward of the seat of the vehicle, the display device has a display area in which the virtual viewpoint image can be displayed, and the control unit displays the virtual viewpoint image in the area on one side of the center in the left-right direction of the display area. <Configuration 3> A vehicle control device in which the control unit causes the display device to display an image representing the vehicle. Furthermore, the vehicle control method of the present invention may be a combination of the following configuration 4 and at least one of configurations 1 to 3. <Configuration 4> A vehicle control method that, when executing driving assistance control to change lanes to one side, either left or right, generates a virtual viewpoint image representing an area located in front of and to the one side of an occupant of the vehicle based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located to the other side, either left or right, of the center of the vehicle's width direction, and displays the virtual viewpoint image on a display device provided on the vehicle. Furthermore, the program of the present invention may be a combination of the following configuration 5 and at least one of configurations 1 to 3. <Configuration 5> A program that causes a computer to execute the following processes when performing driving assistance control to change lanes to one side, either left or right: generating a virtual viewpoint image that represents an area located in front of and to the one side of an occupant of the vehicle, based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located to the other side, either left or right, of the center of the vehicle's width; and displaying the virtual viewpoint image on a display device provided in the vehicle. [Explanation of symbols]

[0061] 10 Vehicle control device 12 vehicles 13L 13R Front seats (seats) 21A Front center camera (camera) 21B Front left camera (camera) 21C Front right camera (camera) 21D Left side camera (camera) 21E Right side camera (camera) 23 Display device 24 Display area 24R 1st area 24L 2nd area 26A CPU (control unit) 40 Virtual Viewpoint Images 45 Virtual Viewpoint Images EP1 EP2 Virtual Viewpoint CL center line (center)

Claims

1. a control unit that, when executing driving assistance control to change lanes to one side in a leftward or rightward direction, generates a virtual viewpoint image representing an area located in front of and on one side of an occupant of the vehicle based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located on the other side in a leftward or rightward direction from a center of the vehicle in a vehicle width direction, and displays the virtual viewpoint image on a display device provided in the vehicle; A vehicle control device in which the virtual viewpoint is located at the same position as the front seat of the vehicle in the longitudinal direction of the vehicle or further forward than the front seat in the longitudinal direction of the vehicle in a plan view.

2. The display device is located in front of the seat of the vehicle, the display device has a display area capable of displaying the virtual viewpoint image, The vehicle control device according to claim 1 , wherein the control unit displays the virtual viewpoint image in the area on the one side of the center of the display area in the left-right direction.

3. The vehicle control device according to claim 1 or 2, wherein the control unit causes the display device to display an image representing the vehicle.

4. When performing driving assistance control to change lanes to one side in the left or right direction, a virtual viewpoint image is generated based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located on the other side in the left or right direction from the center of the vehicle in the vehicle width direction, representing an area located in front of and on the one side of an occupant of the vehicle, displaying the virtual viewpoint image on a display device provided in the vehicle; A vehicle control method in which the virtual viewpoint is located at the same position as the front seat of the vehicle in the longitudinal direction of the vehicle or forward of the front seat in the longitudinal direction of the vehicle in a plan view.

5. When performing driving assistance control to change lanes to one side in the left or right direction, a process of generating a virtual viewpoint image representing an area located in front of and on one side of an occupant of the vehicle based on image data acquired by a camera mounted on the vehicle and a virtual viewpoint located on the other side in the left or right direction from the center of the vehicle in the vehicle width direction; and a process of displaying the virtual viewpoint image on a display device provided in the vehicle; on the computer, A program in which the virtual viewpoint is located in the same position as the front seat of the vehicle in the longitudinal direction of the vehicle or in front of the front seat in the longitudinal direction of the vehicle in a plan view.

Images