Display control device, display control method, and recording medium

Abstract

This display control device comprises a controller 10 that: acquires a camera image from an on-vehicle camera that images the surroundings of a vehicle; generates, on the basis of the camera image, a vehicle external image showing an outside view from the vehicle and a bird's-eye view image showing a plan view of the vehicle from a virtual viewpoint; and outputs the vehicle external image and the bird's-eye view image to a display 41. If the controller 10 causes the display 41 to display the bird's-eye view image, the controller controls the display 41 such that a first predicted course indicating a predicted course of the vehicle is superimposed on the bird's-eye view image. If the controller causes the display 41 to display the vehicle external image, the controller controls the display 41 such that a second predicted course indicating a predicted course of the vehicle is superimposed on the vehicle external image. The width of the first predicted course is greater than the width of the second predicted course.

Description

Display control device, display control method, and recording medium

[0001] The present invention relates to a display control device, a display control method, and a recording medium.

[0002] Conventionally, driving assistance devices calculate the coordinates of the predicted course of the wheels as well as the coordinates of an obstacle, and when the two coordinates intersect or are closer than a predetermined distance, they display a caution or output a sound (for example, Patent Document 1).

[0003] JP 2023-156871 A

[0004] The display control device described in Patent Document 1 does not display the predicted course of the vehicle on an overhead image of the vehicle, so there is a problem that the user cannot easily check the positional relationship between the vehicle and obstacles outside the vehicle on the display screen.

[0005] The problem to be solved by the present invention is to provide a display control device, a display control method, and a recording medium that enable easy confirmation of the positional relationship between the vehicle and an obstacle or the like outside the vehicle.

[0006] The present invention solves the above problem by controlling the display so that, when an overhead image is displayed on the display, a first predicted path that is wider than a second predicted path is superimposed on the overhead image, and by controlling the display so that, when an image outside the vehicle is displayed on the display, the second predicted path is superimposed on the image outside the vehicle.

[0007] According to the present invention, the positional relationship between the vehicle and an obstacle or the like outside the vehicle can be easily confirmed.

[0008] Fig. 1 is a block diagram showing the configuration of a display control system according to this embodiment. Fig. 2 is a plan view showing a part of an instrument panel. Fig. 3 is a diagram showing a display screen of a display. Fig. 4 is a diagram showing a display screen of a display. Fig. 5 is a diagram showing a display screen of a display. Fig. 6 is a diagram showing a display screen of a display. Fig. 7 is a diagram showing a 3D view display screen of a display. Fig. 8 is a diagram showing a display screen of a display. Fig. 9 is a flowchart of a display control method executed by a controller.

[0009] A display control system including a display control device according to an embodiment of the present invention will be described below with reference to the drawings. The display control system is a system that generates an image of the surroundings of a vehicle. FIG. 1 is a block diagram showing the configuration of the display control system according to this embodiment. The display control system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an ATCU 30, a display 41, a display 42, a vehicle speed sensor 50, and a display control device 100. The display control system 1 is installed in a vehicle. The front camera 21, the right camera 22, the left camera 23, the rear camera 24, the ATCU 30, the in-vehicle display 40, the vehicle speed sensor 50, and the display control device 100 are connected via an in-vehicle communication network such as CAN or LIN.

[0010] The front camera 21, right camera 22, left camera 23, and rear camera 24 are provided on the vehicle and capture images of the surroundings of the vehicle. The front camera 21, right camera 22, left camera 23, and rear camera 24 capture images of the surroundings of the vehicle while the vehicle is traveling. The front camera 21 captures images in front of the vehicle, the right camera 22 captures images on the right side of the vehicle, the left camera 23 captures images on the left side of the vehicle, and the rear camera 24 captures images on the rear of the vehicle. The front camera 21 is provided on the grille of the vehicle, the right camera 22 is provided on the right door mirror, the left camera 23 is provided on the left door mirror, and the rear camera 24 is provided on the back door. Note that the various cameras may be provided in other positions on the vehicle. Note that in the following description, the front camera 21, right camera 22, left camera 23, and rear camera 24 will be collectively referred to as camera 20.

[0011] The ATCU (automatic transmission control unit) 30 is a control device that controls the transmission. The ATCU 30 outputs a control signal including shift information to the display control device 100. The shift information indicates the position of the shift lever, such as drive, neutral, reverse, or parking.

[0012] The in-vehicle display 40 is a display included in an in-vehicle infotainment (IVI) system and includes a display 41 and a display 42. The displays 41 and 42 are provided on the vehicle's instrument panel. The displays 41 and 42 are touch-panel displays. FIG. 2 is a plan view showing a portion of the instrument panel. As shown in FIG. 2, the display 41 is provided in the center of the instrument panel. The display 42 is provided on the instrument panel in front of the driver. The display 41 displays, for example, a map for a navigation system, a menu screen (such as a selection screen for various applications) for an entertainment system, content video, etc. The display 42 displays, for example, meters, a range indicator showing the current range of the shift position, an icon showing the status of a seat belt, a vehicle image, an image of the vehicle's surroundings, a predicted course, a driving route, etc. The display 41 and / or the display 42 also display images generated by the display control device 100.

[0013] The displays 41 and 42 have display screens that are long horizontally along the vehicle width direction (the left-right direction on the paper surface of FIG. 2 ). As shown in FIG. 2 , the displays 41 and 42 are arranged side by side on the left and right along the vehicle width direction, with their display screens close to each other. Note that the vehicle does not need to be equipped with both the displays 41 and 42, and may be equipped with only one of the displays. The vehicle speed sensor 50 detects the vehicle speed and outputs the detected value to the controller 10.

[0014] The display control device 100 includes a controller 10 and a recording medium 19. The controller 10 has functions such as acquiring various data, such as camera images and vehicle data, and image processing functions. The controller 10 has, as functional blocks, an image acquisition unit 11, a vehicle data acquisition unit 12, and an image processing unit 13. The controller 10 stores programs for implementing various functions in the recording medium 19, and executes the control programs by causing a processor to execute the processing flow indicated by the programs. Note that the recording medium 19 storing the control programs may be stored as memory within the controller 10. The recording medium 19 may be located inside or outside the controller 10. For example, the recording medium 19 may be stored in a server outside the vehicle, and the display control device 100 may access the server and store the control program data recorded on the recording medium 19 in the memory within the controller 10. The display control device 100 may also execute the control program stored in the server via communication with the server. For example, the display control device 100 transmits camera images and commands for executing each processing step included in the control program to the server. A controller included in the server executes a control program and processes data such as camera images based on instructions from the display control device 100. The server transmits a display image obtained by processing the control program to the display control device 100. The display control device 100 transmits the display image to the displays 41 and 42, causing the image to be displayed.

[0015] The image acquisition unit 11 acquires camera images from the camera 20, etc. The vehicle data acquisition unit 12 acquires vehicle data from sensors mounted on the vehicle, etc., via an in-vehicle communication network. The vehicle data is data indicating the current state of the vehicle, and includes, for example, vehicle speed, steering angle, current vehicle position (current coordinates), and gear shift. The vehicle data acquisition unit 12 acquires vehicle speed from the vehicle speed sensor 50 and acquires steering angle from the steering angle sensor. The vehicle data acquisition unit 12 acquires current position information from a navigation system. The vehicle data acquisition unit 12 also acquires gear shift information from the ATCU 30.

[0016] The image processing unit 13 performs image processing on the camera image to display a current image of the vehicle's surroundings and / or a vehicle image on the displays 41, 42. The image processing unit 13 generates a display image to display the image on the displays 41, 42 in a display mode specified by the user or the system, and outputs the display image to the displays 41, 42. There are multiple display modes, such as a front view showing the front of the vehicle, a rear view showing the rear of the vehicle, a top view giving a bird's-eye view of the vehicle, a 3D view showing the vehicle or its surroundings from a virtual viewpoint, and a skeleton view. In the front view, an image showing the exterior of the vehicle is displayed, which is located in front of the vehicle's grille. In the rear view, an image showing the exterior of the vehicle is displayed, which is located behind the vehicle's tailgate. In the top view (around view), a bird's-eye view image of the vehicle viewed from a virtual viewpoint directly above the vehicle is displayed. In the top view, a two-dimensional image is displayed, and the vehicle image is shown as a plan view of the vehicle. If there are tall three-dimensional objects such as walls around the vehicle, an image of the surroundings is displayed, showing the three-dimensional objects as if they have fallen down. For example, if the vehicle is parked on the road, the bird's-eye image includes an image of the vehicle, an image of the road surface around the vehicle, and an image of surrounding obstacles. In the 3D view, a position around the object is set as a virtual viewpoint, and an image of the object is displayed as seen from the virtual viewpoint. The object is not limited to the vehicle, but may also be an obstacle around the vehicle. The virtual viewpoint is a viewpoint that provides a pseudo-view of the object from outside the object. The virtual viewpoint is movable. For example, when a display image (three-dimensional image) of the vehicle viewed from the virtual viewpoint is displayed on the displays 41 and 42, the three-dimensional vehicle image can be rotated by orbiting the virtual viewpoint along a plane around the vehicle. In addition, the displays 41 and 42 may display the surroundings of the vehicle in other display modes, such as a skeleton view, a front wide view, or a rear wide view. In the skeleton view, in addition to the current camera image (real image) showing the vehicle's surroundings, a virtual image showing the condition of the vehicle's underfloor is displayed.For example, when a vehicle is turning left or right at an intersection with poor visibility such as a T-junction, the front wide view can display an image of the area in front of the vehicle that is in the driver's blind spot from an image taken by a front camera 21 having a wide-angle lens.

[0017] The display mode can be specified by the user or can be changed by the vehicle system as appropriate depending on the driving state of the vehicle. For example, when the range is shifted to reverse to park the vehicle, the display mode may be automatically switched to a top view or a rear view. Alternatively, the display mode may be switched by the driver operating a switch or the like while the vehicle is moving.

[0018] The image processing unit 13 can display images in multiple display modes on multiple screens on the display 41. The image processing unit 13 may display images in different display modes on the displays 41 and 42, or may display images in the same display mode on the displays 41 and 42. The image processing unit 13 may also use both the screens of the displays 41 and 42 as a single display screen to display images in multiple display modes, or may display images in a single display mode. The following description will explain the image processing performed by the image processing unit 13 when displaying an image on the display 41, but when displaying an image on the display 42, the image processing performed on the display 41 may be applied to the image display on the display 42.

[0019] The image processing unit 13 generates a display image according to the display mode based on the multiple camera images acquired by the image acquisition unit 11, and outputs the display image to the display 41. When displaying an image in front view or rear view, the image processing unit 13 generates a display image showing the front or rear of the vehicle based on the camera image from the front camera 21 or the rear camera 24.

[0020] When displaying an image in top view, the image processing unit 13 generates an overhead image of the vehicle based on multiple camera images captured by the cameras 20. The virtual viewpoint of the overhead image is set at a predetermined position vertically away from the roof of the vehicle and outside the vehicle. For example, the image processing unit 13 performs viewpoint conversion processing on images captured by the front camera 21, right camera 22, left camera 23, and rear camera 24 to generate viewpoint-converted images that show the front, rear, left, and right regions of the vehicle from the virtual viewpoint. That is, the image processing unit 13 converts camera images corresponding to images viewed from the camera positions into viewpoint-converted images viewed from the virtual viewpoint. The viewpoint conversion from the camera positions to the virtual viewpoint is performed by calculation using viewpoint parameters, and the viewpoint parameters are stored in the recording medium 19. The image processing unit 13 fits each image after the viewpoint conversion processing to the surrounding area of ​​the vehicle. The image processing unit 13 performs blending processing on overlapping portions of the camera images from the cameras 20. The blending process is a process for maintaining continuity between two overlapping images, and any method known at the time of filing of this application can be used. The image after the blending process becomes a surrounding image showing the surroundings of the vehicle. The image processing unit 13 also combines the vehicle image located at the center of the screen with the surrounding image. The combined image becomes an overhead image, and the image processing unit 13 outputs the overhead image to the display 41 as a display image.

[0021] When displaying an image in 3D view, the image processing unit 13 generates a display image of the vehicle as viewed from a virtual viewpoint based on multiple camera images captured by the camera 20. First, the image processing unit 13 sets a virtual viewpoint based on a user specification or the driving situation. The position and line of sight of the virtual viewpoint can be arbitrarily set based on a user instruction. The user can specify the position and line of sight of the virtual viewpoint by, for example, touching the display 41. The image processing unit 13 can also set the virtual viewpoint based on the driving situation. For example, the virtual viewpoint can be set behind the vehicle when driving forward to view an image in front of the vehicle, and in front of the vehicle when driving backward to view an image behind the vehicle. Specifically, the image processing unit 13 sets the position of the virtual viewpoint when it receives shift information indicating that the gearshift position is in reverse. The image processing unit 13 can also set the virtual viewpoint based on the vehicle speed. For example, the virtual viewpoint can be set based on the vehicle speed input from the vehicle speed sensor 50 so that the virtual viewpoint is farther away from the vehicle when the vehicle speed is equal to or greater than a predetermined value, and closer to the vehicle when the vehicle speed is less than the predetermined value.

[0022] The image processing unit 13 deforms at least a part of a reference curved surface coordinate system having a predefined curved surface for projecting the camera image, according to the position of the virtual viewpoint. The reference curved surface coordinate system is stored in the recording medium 19. The reference curved surface coordinate system has, for example, a bowl shape surrounding the vehicle. The reference curved surface coordinate system has a center on an xy plane parallel to the vehicle's loading surface (travel surface), and is formed from a curved surface having a curvature from the center or near the center and a component in the vehicle's height direction (z direction).

[0023] A base parallel to the xy plane can be formed near the center of the reference curved surface coordinate system, and the base can have any shape, such as a rectangle, a triangle, an ellipse, or a circle. The curvature of the curved surface of the reference curved surface coordinate system does not have to be uniform. For example, the curvature of the curved surface near the base may be relatively large, and the curvature of the curved surface away from the base may be relatively small. The image processing unit 13 may store in advance multiple reference curved surface coordinate systems with different shapes depending on the shape (size, form) of the vehicle. When multiple reference curved surface coordinate systems are stored, the reference curved surface coordinate systems to be used may be stored in the recording medium 19 in association with the vehicle speed, the position of the vehicle shift position, etc., and the image processing unit 13 may select a reference curved surface coordinate system depending on the vehicle speed, the position of the vehicle shift position, etc.

[0024] The image processing unit 13 may read a pre-stored reference curved surface coordinate system and deform the shape of the read reference curved surface coordinate system. The image processing unit 13 may deform the shape of the reference curved surface coordinate system according to the position of the virtual viewpoint. For example, when the virtual viewpoint is located outside the outer edge of the reference curved surface coordinate system, a portion of the reference curved surface coordinate system may be deformed so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system. By deforming a portion of the reference curved surface coordinate system so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system, distortion of the shape of an obstacle with height can be suppressed when the obstacle is displayed on the screen, and an image without any missing image can be displayed on the display 41. Note that the method of deforming the reference curved surface coordinate system may use other coordinate system deformation methods known at the time of filing.

[0025] The image processing unit 13 projects multiple camera images acquired from the camera 20 onto the reference curved coordinate system. When the reference curved coordinate system is deformed, the image processing unit 13 simply projects the camera images onto the curved coordinate system after the deformation (deformable coordinate system). In order to project the camera image data onto the reference curved coordinate system or the deformed coordinate system after the deformation, the image processing unit 13 has an image conversion table that associates the coordinates of pixels included in the camera images with the coordinates of the coordinate system. The image conversion table is recorded on the recording medium 19.

[0026] A prepared vehicle image of a vehicle may also be superimposed on the reference curved coordinate system. The vehicle image may be created in advance based on the design of the vehicle and stored in the recording medium 19. By superimposing the vehicle image on the reference curved coordinate system, not only the image of the vehicle's surroundings but also the vehicle itself can be displayed in three dimensions, which helps understand the relationship between the vehicle's position and orientation and the surrounding image. In this way, the image processing unit 13 refers to the image conversion table to associate the coordinates of pixels contained in the camera image with the coordinates of the coordinate system, projects the camera image onto the reference curved coordinate system or the deformed coordinate system, and superimposes the vehicle image on the projected three-dimensional image to generate a display image (three-dimensional image) in a 3D view in which the vehicle and its surroundings are viewed from a virtual viewpoint.

[0027] The image processing unit 13 may generate display images viewed from selectable virtual viewpoints, not limited to those specified by the user or set based on the driving situation. For example, when a display image is displayed on the display 41 in a 3D view, the user can freely move the position of the virtual viewpoint by touching the display 41. When the 3D view display is selected, the image processing unit 13 generates multiple 3D view display images for each of multiple virtual viewpoints selectable by the user. Then, for example, when the user moves the virtual viewpoint by touching the display 41, a 3D image in which the vehicle image rotates around the center point of a curved coordinate system is displayed on the display 41. Note that the image processing unit 13 may set virtual viewpoints at any positions around the vehicle, not limited to those selectable by the user, and generate multiple 3D view display images for each virtual viewpoint. In other words, when a 3D image is displayed on the display 41 in a rotating display, the image processing unit 13 generates a continuous 3D image so that the image can be displayed smoothly in response to the movement of the virtual viewpoint.

[0028] The image processing method for the display images displayed in the front / rear view, top view, and 3D view display modes is not limited to the above, and other image processing methods can be used. Furthermore, the image processing method for the display images displayed in other display modes, such as skeleton view and front / rear wide view, can be any method known at the time of filing.

[0029] Next, display control by the display control device 100 will be described along with specific examples of the display screen of the display 41. First, an example of displaying an image in a top view and a front view will be described. Fig. 3 shows the display screen of the display 41. The example of Fig. 3 shows a state in which a plurality of parking spaces are lined up in a row, and a vehicle is driving between two rows of parking spaces. In the example of Fig. 3, the left screen of the display 41 displays a top view image, and the right screen of the display 41 displays a front view image.

[0030] When displaying a top-view overhead image on the display 41, the image processing unit 13 generates a top-view image 46 by superimposing predicted course lines 51 on the overhead image of the vehicle. The top-view image 46 includes an overhead image of the vehicle viewed from above and predicted course lines 51. The predicted course lines 51 are a pair of guide lines indicating the predicted course of the vehicle. The controller 10 acquires vehicle data indicating the steering angle using the vehicle data acquisition unit 12 and calculates the tire turning angle and the vehicle course from the steering angle. The image processing unit 13 draws the predicted course lines 51 in conjunction with the steering operation. When the vehicle is turning, the image processing unit 13 also draws the predicted course lines 51 so that they curve in the turning direction of the vehicle. In the example of FIG. 3 , since the steering angle is 0 degrees, the predicted course lines 51 are drawn as straight lines. Note that the method of drawing predicted course lines 52 and 59 displayed in the rear view, top view, and 3D view (described later) is the same as the method of drawing predicted course lines 51.

[0031] The predicted course lines 51 are drawn as two lines extending vertically (along the direction of travel of the vehicle) from a position several tens of centimeters (for example, 10 to 50 cm) away from the front end of the vehicle to a position several meters (for example, 2 to 4 m). The predicted course lines 51 are also drawn at a position extending the mirror width. The mirror width is the width of the vehicle including the door mirrors, and is larger than the body width of the vehicle body excluding the rearview mirrors. The predicted course lines 51 are drawn at a position extending along the direction of travel of the vehicle while maintaining the mirror width. In other words, the width (W) of the predicted course lines 51 is １) is the distance between the two predicted course lines 51, and is equal to or greater than the vehicle body width. In this case, it is desirable that the distance is equal to or greater than the vehicle body width and equal to or less than the width between the outer edges of the left door mirror and the right door mirror. In the example of FIG. 3, the width (W １ ) corresponds to the mirror width. １ ) is the width (W ２ ) In the example of FIG. 3 , predicted course lines 51 are drawn ahead of the vehicle, which is the direction of travel of the vehicle. Image processing unit 13 outputs top view image 46, in which predicted course lines 51 are superimposed on the overhead image, to display 41. Controller 10 then controls display 41 so that predicted course lines 51 are superimposed on the overhead image in the top view.

[0032] Furthermore, when displaying the vehicle exterior image in front view on display 41, image processing unit 13 generates front view image 48 by superimposing predicted course lines 52, distance guide lines 53, and vehicle width guide lines on the vehicle exterior image. Front view image 48 includes the vehicle exterior image, predicted course lines 52, distance guide lines 53, and vehicle width guide lines. The vehicle exterior image is an image of the front of the vehicle as seen from the vehicle exterior. In the front view, the vehicle exterior image corresponds to the camera image from front camera 21.

[0033] The predicted course lines 52 are a pair of lines that indicate the predicted course of the vehicle. The predicted course lines 53 are drawn as two lines in the vertical direction (along the direction of travel of the vehicle) with a length of several meters (for example, 2 to 4 m) from a position several tens of centimeters (for example, 10 to 50 cm) away from the front end of the vehicle. The width (W ２ ) is the distance between the two predicted course lines 52, and is equal to or greater than the width between the inner ends of the left and right wheels and equal to or less than the width of the vehicle body. ２ ) is the width (W １ In the example of FIG. 3, the width (W ２) corresponds to the body width, and the predicted course line 52 is drawn at a position extended along the traveling direction of the vehicle while maintaining the body width of the vehicle. The predicted course line 52 is drawn ahead of the vehicle, which is the traveling direction of the vehicle. In this embodiment, the width (W １ ) and the width (W ２ ), but the reference point for drawing the predicted course line 52 for the vehicle may be located inside the vehicle with respect to the reference point for drawing the predicted course line 51 for the vehicle. The reference point is not particularly limited, but may be a point on the predicted new course line 51 and the predicted course line 52 closest to the vehicle or a virtual point.

[0034] The distance reference lines 53 indicate the distance from the front end of the vehicle, and are drawn as multiple lines along the vehicle width direction. In the example of Fig. 3, the distance reference lines 53 indicate three different distances (e.g., 50 cm, 1 m, and 2 m). The distance reference lines 53 are drawn between a pair of predicted course lines 52. Of the multiple distance reference lines 53, the distance reference line 53 closest to the vehicle may be drawn in a color different from the other distance reference lines 53.

[0035] The vehicle width guide lines indicate the vehicle width and are represented by a pair of lines extending in the fore-and-aft direction of the vehicle (perpendicular to the direction along the vehicle width). The width of the vehicle width guide lines corresponds to the vehicle body width. In the example of FIG. 3 , since the vehicle is traveling straight, the vehicle width guide lines are hidden by predicted course lines 52 and are not displayed. Image processing unit 13 outputs to display 41 a front view image 48 in which predicted course lines 52, distance guide lines 53, and vehicle width guide lines are superimposed on the image of the vehicle exterior. Controller 10 then controls display 41 so that at least predicted course lines 52 are superimposed on the image of the vehicle exterior in the front view.

[0036] Next, an example of displaying a top-view and a rear-view image will be described. Fig. 4 shows the display screen of the display 41. The example of Fig. 4 shows a state in which the driver is turning the steering wheel clockwise and the vehicle is moving backward. In the example of Fig. 4, the left screen of the display 41 displays a top-view image, and the right screen of the display 41 displays a rear-view image.

[0037] When displaying a top-view image on the display 41, the image processing unit 13 generates a top-view image 46 by superimposing the predicted course lines 51 and predicted bulge confirmation lines 54 on the bird's-eye image of the vehicle. The top-view image 46 includes a bird's-eye image of the vehicle viewed from above, the predicted course lines 51, and the predicted bulge confirmation lines 54. The predicted course lines 51 are the same as the predicted course lines 51 shown in the top view of FIG. 3, and the width of the predicted course lines 51 is W. １ 4, the predicted course line 51 is drawn as a curve corresponding to the steering angle. The predicted course line 51 is drawn behind the vehicle.

[0038] The predicted bulge confirmation line 54 represents the predicted path of the outer edge of the vehicle body when the vehicle bulges during a turn. In the example of FIG. 4 , the right front wheel of the vehicle body turns on the outermost side, so the predicted bulge confirmation line 54 is drawn as a curve of a predetermined length at the right front wheel. The image processing unit 13 outputs to the display 41 a top-view image 46 in which the predicted course line 51 and the predicted bulge confirmation line 54 are superimposed on the overhead-view image. In this way, the controller 10 controls the display 41 so that at least the predicted course line 51 is superimposed on the overhead-view image in the top view.

[0039] Furthermore, when displaying the vehicle exterior image in rear view on the display 41, the image processing unit 13 generates a rear view image 49 by superimposing predicted course lines 52, distance guide lines 53, vehicle width guide lines 55, predicted tire course lines 56, and center line 57 on the vehicle exterior image. The rear view image 49 includes the vehicle exterior image, predicted course lines 52, distance guide lines 53, vehicle width guide lines 55, predicted tire course lines 56, and center line 57. The vehicle exterior image is an image behind the vehicle as seen from the vehicle exterior. In the front view, the vehicle exterior image corresponds to the camera image of the rear camera 21. The predicted course lines 52 are the same as the predicted course lines 52 shown in the front view of FIG. 3, and the width of the predicted course lines 52 is W ２ In the example of FIG. 4, the predicted course line 52 is drawn as a curve according to the steering angle. The predicted course line 52 is drawn behind the vehicle. The width (W ２) is the width (W １ In this embodiment, the width (W １ ) and the width (W ２ ), but the reference point for drawing the predicted course line 52 for the vehicle may be located inside the vehicle with respect to the reference point for drawing the predicted course line 51 for the vehicle. The reference point is not particularly limited, but may be a point on the predicted new course line 51 and the predicted course line 52 closest to the vehicle or a virtual point.

[0040] The distance reference lines 53 are similar to the distance reference lines 53 shown in the front view of Fig. 3. However, in the example of Fig. 4, the distance reference lines 53 are drawn so that the interval between the distance reference lines 53 on the right wheel side is wider than the interval between the distance reference lines 53 on the left wheel side in accordance with the turning of the vehicle.

[0041] The vehicle width guide lines 55 are drawn as a pair of lines behind the vehicle. The predicted tire course lines 56 are a pair of lines that indicate the predicted course of the tires. In other words, when the vehicle travels along the predicted course lines 52, the predicted tire course lines 56 become tire marks. The width of the predicted tire course lines 56 corresponds to the distance between the left and right wheels. The width of one line of the predicted tire course lines 56 corresponds to the tire width. The predicted tire course lines 56 are drawn at a position extended along the vehicle's traveling direction while maintaining a width corresponding to the distance between the left and right wheels. The predicted tire course lines 56 are drawn behind the vehicle, which is in the vehicle's traveling direction. The predicted tire course lines 56 are linked to the steering operation and are drawn as a curve that matches the turning of the vehicle. In other words, the predicted tire course lines 56 are drawn as a curve that corresponds to the steering angle. The image processing unit 13 may draw the predicted tire course lines 56 in the same manner as the predicted course lines 51, 52. The center line 57 corresponds to the center line of the pair of predicted course lines 52.

[0042] Image processing unit 13 outputs rearview image 49, in which predicted course lines 52, distance guide lines 53, vehicle width guide lines 55, predicted tire course lines 56, and center line 57 are superimposed on the vehicle exterior image, to display 41. In this way, controller 10 controls display 41 so that at least predicted course lines 52 are superimposed on the vehicle exterior image in the rearview.

[0043] Next, an example of displaying an image in a top view and a 3D view will be described. FIG. 5 shows the display screen of the display 41. In the example of FIG. 5, the vehicle is stopped on a straight road, and the left screen of the display 41 displays a top view image, while the right screen of the display 41 displays a 3D view image. When displaying a top view overhead image and a 3D view display image on the display 41, the controller 10 generates a top view image 46 by superimposing viewpoint icons 43a-43h and an automatic display start icon 44 on the overhead image of the vehicle. The top view image 46 includes an overhead image of the vehicle viewed from above, viewpoint icons 43a-43h, and the automatic display start icon 44. The viewpoint icons 43a-43h indicate the position of a virtual viewpoint and / or the direction of the line of sight and are user-selectable icons. The viewpoint icons 43a-43h are arranged around the vehicle, with the vehicle image 45 at the center. Viewpoint icon 43a is positioned above the vehicle image located at the center of the overhead image, while viewpoint icons 43c, 43e, and 43g are positioned to the right, bottom, and left of the vehicle image, respectively. Viewpoint icon 43b, viewpoint icon 43d, viewpoint icon 43f, and viewpoint icon 43h are positioned in the upper right, lower right, lower left, and lower right directions, respectively, of the vehicle image. When the user selects one of viewpoint icons 43a to 43h, a display image (three-dimensional image) of the vehicle as viewed from the selected viewpoint icon is displayed on the right side of display 41.

[0044] As shown in FIG. 5 , when the user touches the viewpoint icon 43a, the controller 10 displays the viewpoint icon 43a as a selection icon. The controller 10 displays the selection icon by, for example, changing the color of the viewpoint icon 43a. In the example of FIG. 5 , the selection icon is represented by a thick line surrounding the viewpoint icon 43a. The image processing unit 13 sets a virtual viewpoint corresponding to the viewpoint icon 43a. This allows the virtual viewpoint to be set according to the user's designation. In the example of FIG. 5 , a position from which the vehicle is viewed from the front is set as the virtual viewpoint. Then, the image processing unit 13 generates a three-dimensional image 47 of the vehicle and its surroundings viewed from the front of the vehicle based on multiple camera images. The image processing unit 13 outputs the three-dimensional image to the display 41 as a display image. The three-dimensional image 47 is displayed on the right screen of the display 41. Then, as shown in FIG. 3 , the display 41 displays the top-view image 46 and the three-dimensional image 47 side by side.

[0045] The automatic display start icon 44 is an icon for starting automatic display, which automatically switches three-dimensional images in the 3D view by moving the virtual viewpoint. The automatic display start icon 44 is displayed at the center position of the vehicle image 45 included in the top-view image 46. When automatic display is executed, predetermined positions around the vehicle become the start and end points of the virtual viewpoint, and the virtual viewpoint moves from the start point to the end point, causing the vehicle image displayed in the 3D view to rotate 360 ​​degrees along a plane parallel to the road surface. In other words, in automatic display, the center point is placed on the vehicle image, and the vehicle image and the vehicle-surrounding image rotate 360 ​​degrees around the center point. For example, in the example of FIG. 3 , when the user touches the automatic display start icon 44, the selected viewpoint icon 43 a becomes the start and end points of the virtual viewpoint. The virtual viewpoint moves from the position corresponding to the viewpoint icon 43 a to viewpoint icon 43 b in order to viewpoint icon 43 h, rotating clockwise around the vehicle, and then returns to the position corresponding to the original viewpoint icon 43 a. As the virtual viewpoint rotates, the three-dimensional image 47 in the 3D view rotates. The orientation of the vehicle image included in the three-dimensional image 47 starts from the front (front) as shown in Fig. 3, changes in the order of the right side, rear (back), and left side, and then returns to the original front (front). In other words, the three-dimensional image is displayed as if it were rotated 360 degrees (hereinafter also referred to as "360-degree rotation display"). As a result, the display image (three-dimensional image 47) is automatically rotated 360 degrees in the 3D view.

[0046] Another example of displaying images in a top view and a 3D view will be described. FIG. 6 shows the display screen of the display 41. In the example of FIG. 6, the vehicle is stopped at a T-junction, and the left screen of the display 41 displays a top-view image, while the right screen of the display 41 displays a 3D-view image. The image processing method for the top-view image 46 and the 3D image 47 is the same as in the example of FIG. 5, and the top-view image 46 also includes viewpoint icons 43a-h and an automatic display start icon 44. Unlike the example of FIG. 5, viewpoint icon 43e is selected, and the 3D view displays a 3D image 45 of the vehicle and its surroundings as seen from behind the vehicle. Note that, as shown in FIG. 6, when the vehicle is stopped at a T-junction, the controller 10 may set a virtual viewpoint corresponding to viewpoint icon 43a and display a 3D image 47 viewed from the virtual viewpoint on the display 41. This allows the state of the blind spot to be displayed on the display 41 if there is a blind spot ahead of the vehicle.

[0047] 5 and 6, while the vehicle is stopped, the user can select one of the viewpoint icons 43a to 43h and view a three-dimensional image 47 from the direction of the selected viewpoint icon 43a to 43h in a 3D view on the display 41. For example, when the user touches the position of the viewpoint icon 43c on the screen of the display 41, the image processing unit 13 switches the viewpoint icon 43c to a selected icon and moves the current virtual viewpoint (in the example of FIG. 3, the virtual viewpoint corresponding to the viewpoint icon 43a) to the virtual viewpoint corresponding to the viewpoint icon 43c. The image processing unit 13 then generates a three-dimensional image viewed from the virtual viewpoint based on the camera image and outputs the display image to the display 41 so that the three-dimensional image is displayed in a 3D view. A three-dimensional image of the vehicle viewed from the right side is displayed on the display 41.

[0048] Furthermore, while the vehicle is stopped, the display control device 100 may perform display control so that the user can not only select a virtual viewpoint by touching the viewpoint icons 43a to 43h, but also arbitrarily specify the position of the virtual viewpoint. The user touches any position on the 3D view display screen. Then, while the user's finger is touching the screen, the user moves the finger in any direction (swipe operation). While the finger is touching the screen, the position of the virtual viewpoint moves in accordance with the swipe operation, and the 3D image rotates in accordance with the movement of the virtual viewpoint. The amount of movement of the virtual viewpoint is determined by the swipe amount (corresponding to the amount the finger moves while touching the screen), and the greater the swipe amount, the greater the movement amount. Note that the amount of movement of the virtual viewpoint corresponds to the length of the arc of a circle centered on the vehicle image. For example, the user touches the 3D view display screen with their finger and moves their finger with a large swipe amount. If the swipe amount is large, the image processing unit 13 moves the virtual viewpoint around the center point of the vehicle image, thereby rotating the 3D image 47. If the swipe amount is small, image processing unit 13 rotates three-dimensional image 47 so that the virtual viewpoint moves along an arc centered on the center point of the vehicle image by an amount equivalent to the swipe amount. Note that the rotational display caused by a swipe operation is not limited to the 3D view display screen, but may also be performed by a swipe operation on a top view display screen. For example, image processing unit 13 causes display 41 to display an image in which viewpoint icons 43a to 43h have been removed from a top view image 46 as shown in FIG. 5 or 6 . Then, the user performs a swipe operation at any position on the screen displaying top view image 46. Image processing unit 13 rotates the three-dimensional image on the 3D view display screen in response to the swipe operation on the top view display screen.

[0049] 7A and 7B are diagrams illustrating the rotational display of a three-dimensional image 47 in a 3D view. Fig. 7A shows a three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43e, Fig. 7B shows a three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43f, and Fig. 7C shows a three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43g. Note that Fig. 7 is a 3D view display screen of the display 41, and shows only the vehicle image included in the three-dimensional image 47, omitting an image of the vehicle's surroundings. The virtual viewpoint that serves as the starting point for the rotational display is the position corresponding to viewpoint icon 43e.

[0050] As shown in FIG. 7A, the state in which the vehicle image included in the three-dimensional image 47 faces rearward (backward) is the starting point of the rotational display. As the virtual viewpoint moves, the vehicle image rotates 45 degrees, resulting in the state shown in FIG. 7B. Further rotation results in the state shown in FIG. 7C. The three-dimensional image 47 has rotated 90 degrees up to this point, and then rotates the remaining 270 degrees. That is, the orientation of the vehicle image included in the three-dimensional image 47 starts from the rearward (backward) side, changes in this order to the left side, front (front), and right side, and then returns to the original rearward (backward) side. Furthermore, the surrounding images located around the vehicle image also rotate primarily together with the vehicle image. As a result, the display image (three-dimensional image 47) is automatically rotated 360 degrees in the 3D view.

[0051] Another example of displaying an image in a top view and a 3D view will now be described. Fig. 8 shows the display screen of the display 41. The example of Fig. 8 shows a state in which the driver is turning the steering wheel counterclockwise to park the vehicle facing forward into a parking space located to the left front, and the vehicle is moving forward. In the example of Fig. 8, the left screen of the display 41 displays a top view image, and the right screen of the display 41 displays a 3D image.

[0052] When displaying a top-view image on the display 41, the image processing unit 13 generates a top-view image 46 by superimposing the predicted course line 51 and the predicted encirclement confirmation line 58 on the bird's-eye image of the vehicle. The top-view image 46 includes a bird's-eye image of the vehicle viewed from above, the predicted course line 51, and the predicted encirclement confirmation line 58. The predicted course line 51 is the same as the predicted course line 51 shown in the top view of FIG. 3, and the width of the predicted course line 51 is W １ 8, the predicted course line 51 is drawn as a curve corresponding to the steering angle, and is drawn ahead of the vehicle.

[0053] The predicted vehicular collision confirmation line 58 represents the predicted trajectory of the inner edge of the vehicle body during a turn. In the example of FIG. 8 , because the left rear wheel of the vehicle body turns on the innermost side, the predicted vehicular collision confirmation line 58 is drawn as a curve of a predetermined length at the left rear wheel. Furthermore, when displaying a top-view screen in association with a 3D view, the image processing unit 13 superimposes viewpoint icons 43a to 43f and an automatic display start icon 44 on the overhead image of the vehicle. In the example of FIG. 8 , viewpoint icon 43h has been selected by the user, and viewpoint icon 43h is displayed as the selected icon. The image processing unit 13 outputs to the display 41 a top-view image 46 in which the predicted course line 51 and the predicted vehicular collision confirmation line 58 are superimposed on the overhead image. In this way, the controller 10 controls the display 41 so that at least the predicted course line 51 is superimposed on the overhead image in the top view.

[0054] Furthermore, when the image processing unit 13 displays an image of the vehicle and its surroundings on the display 41 in a 3D view, it generates a three-dimensional image 47 by superimposing an entrapment confirmation forecast line 58 and a forecast course line 59 on the image. The three-dimensional image 47 includes an image of the vehicle, an image of the surroundings of the vehicle, an entrapment confirmation forecast line 58, and a forecast course line 59. The three-dimensional image 47 is an image seen from a virtual viewpoint corresponding to the viewpoint icon 43h. The forecast course line 59 is a plurality of lines indicating the forecast course of the vehicle. The forecast course line 59 is drawn in two lengths along the direction of travel of the vehicle, with a length of up to several meters from positions a predetermined distance away from the left and right front wheels and the left rear wheel. The width (W) of the two forecast course lines 59 located in front of the vehicle is１ ) corresponds to the mirror width, and the predicted course lines 59 are drawn at a position extending along the vehicle's traveling direction while maintaining the vehicle's mirror width. The image processing unit 13 outputs to the display 41 a three-dimensional image 47 in which the entanglement confirmation predicted line 58 and the predicted course lines 59 are superimposed on a three-dimensional image of the vehicle and its surroundings. The controller 10 then controls the display 41 so that at least the predicted course lines 59 are superimposed on the three-dimensional image 47 in the 3D view. Note that the predicted course lines 59 in the 3D view are drawn in front of the vehicle when the shift position is other than reverse, and are drawn behind the vehicle when the shift position is reverse. Note that in this embodiment, the width (W １ ) and the width (W ２ ), but the reference point for drawing the predicted course line 52 for the vehicle may be located inside the vehicle with respect to the reference point for drawing the predicted course line 59 for the vehicle. The reference point is not particularly limited, but may be a point on the predicted new course line 51 and the predicted course line 59 that is closest to the vehicle or a virtual point.

[0055] Next, a display control method by the controller 10 will be described with reference to Fig. 9. Fig. 9 is a flowchart of the display control method by the controller 10.

[0056] In step S1, image acquisition unit 11 acquires a camera image from camera 20. In step S2, controller 10 determines a display mode specified by a user or the system. For example, if the user or the system specifies that a top-view image be displayed on the left screen of display 41 and a front-view image be displayed on the right screen of display 41, controller 10 executes the control flow of steps S3, S4, S9, S10, and S11.

[0057] If a front view is specified, the image processing unit 13 generates a vehicle exterior image showing the area ahead of the vehicle based on the camera image in step S3. In step S4, the image processing unit 13 draws predicted course lines, distance guide lines, and vehicle width guide lines on the vehicle exterior image, thereby superimposing each line on the vehicle exterior image. This generates a front view image. The control flow proceeds to step S11.

[0058] If a rear view is specified, the image processing unit 13 generates a vehicle external image showing the area behind the vehicle based on the camera image in step S5. In step S6, the image processing unit 13 draws predicted course lines, distance guide lines, vehicle width guide lines, predicted tire course lines, and a center line on the vehicle external image, thereby superimposing each line on the vehicle external image. This generates a rear view image. The control flow proceeds to step S11.

[0059] If a top view is specified, in step S7, the image processing unit 13 generates an overhead image of the vehicle based on the camera image. In step S8, the image processing unit 13 draws the predicted course line, the encirclement check line, and the predicted bulge check line on the vehicle exterior image, thereby superimposing each line on the vehicle exterior image. In this way, a top view image is generated. The control flow proceeds to step S11.

[0060] If a top view is specified, the image processing unit 13 generates a three-dimensional image of the vehicle viewed from a virtual viewpoint based on the camera image in step S9. In step S10, the image processing unit 13 generates a 3D view display image by drawing the predicted course line on the three-dimensional image. The control flow proceeds to step S11.

[0061] In step S11, the controller 10 outputs a display image for each display mode to the display 41, so that an image corresponding to the display mode is displayed on the display 41. After the control flow of step S11 is executed, the control flow ends. The controller 10 restarts the control flow from step S1.

[0062] As described above, in this embodiment, the controller 10 acquires camera images from the camera 20, generates a vehicle exterior image and a bird's-eye view image of the vehicle based on the camera images, and outputs the vehicle exterior image and the bird's-eye view image to the display 41. Furthermore, when the controller 10 causes the display 41 to display the bird's-eye view image, the controller 10 controls the display 41 so that a first predicted course (corresponding to the "predicted course line 51") indicating the predicted course of the vehicle is superimposed on the bird's-eye view image. When the controller 10 causes the display 41 to display the vehicle exterior image, the controller 10 controls the display 41 so that a second predicted course (corresponding to the "predicted course line 52") indicating the predicted course of the vehicle is superimposed on the vehicle exterior image. Furthermore, the width of the first predicted course is made wider than the width of the second predicted course. This allows the user to clearly grasp the clearance of obstacles, etc., outside the vehicle when viewing the bird's-eye view image of the vehicle. As a result, the user can easily confirm the positional relationship between the vehicle and obstacles, etc., outside the vehicle.

[0063] In this embodiment, the width of the second predicted path is equal to or greater than the width between the inner ends of the left and right wheels and equal to or less than the vehicle body width. The width of the first predicted path is equal to or greater than the vehicle body width, or equal to or greater than the vehicle body width and equal to or less than the width between the outer end of the left door mirror and the outer end of the right door mirror. This allows the positional relationship between the vehicle and an obstacle outside the vehicle to be easily confirmed. The present invention is not limited to the above configuration, and the reference point of the second predicted path may be located between the inner end of the left wheel and the left side of the vehicle body width or less than the inner end of the right wheel and the right side of the vehicle body width, and the reference point of the first predicted path may be equal to or greater than the vehicle body width or equal to or greater than the vehicle body width and between the outer end of the left and right door mirror.

[0064] In this embodiment, the controller 10 may draw the predicted vehicle course using a pair of guide lines (two lines) or an area (region). For example, on the left screen of the display 41 in Fig. 3, the predicted course may be displayed as an area on the display 41 by using a color or hatching that is different from the color or hatching of the road surface image in the area between the pair of predicted course lines 51.

[0065] In this embodiment, the width of the predicted course lines 51, 52, and 59 may be selectable in the initial settings at the time of vehicle shipment. Also, display or non-display of course lines and guide lines such as the predicted course lines 51, 52, and 59 may be selectable at the time of initial settings, or may be selectable by the user after vehicle shipment.

[0066] In a modification of this embodiment, the controller 10 may be configured to calculate the width (W ２ ) may be changed depending on the state of the door mirrors of the vehicle. When the door mirrors are in the extended state, the width (W ２ This makes it easier to check the positional relationship between the vehicle and obstacles outside the vehicle.

[0067] In this embodiment, the controller 10 does not need to execute all of the control flows shown in FIG. 9, and does not need to execute each control flow in the order shown in FIG.

[0068] REFERENCE SIGNS LIST 1 Display control system 10 Controller 11 Image acquisition unit 12 Vehicle data acquisition unit 13 Image processing unit 19 Recording medium 20 Camera 40 In-vehicle display 43a to 43h Viewpoint icon 44 Automatic display start icon 45 Vehicle image 46 Top view image 47 Three-dimensional image 48 Front view image 49 Rear view image 51, 52, 59 Predicted course line 50 Vehicle speed sensor

Claims

1. A vehicle control device comprising: a controller that acquires camera images from an onboard camera that captures images of the surroundings of a vehicle; generates, based on the camera images, an external image of the vehicle viewed from the outside of the vehicle and an overhead image of the vehicle from a virtual viewpoint; and outputs the external image of the vehicle and the overhead image to a display; wherein the controller controls the display so that, when the overhead image is to be displayed on the display, a first predicted course indicating the predicted course of the vehicle is superimposed on the overhead image; and when the vehicle external image is to be displayed on the display, controls the display so that a second predicted course indicating the predicted course of the vehicle is superimposed on the external image of the vehicle; and the width of the first predicted course is greater than the width of the second predicted course.

2. A vehicle control device according to claim 1, wherein the width of the second predicted course is equal to or greater than the width between the inner ends of the left and right wheels and equal to or less than the body width of the vehicle.

3. A vehicle control device according to claim 1 or 2, wherein the width of the first predicted course is equal to or greater than the body width of the vehicle and equal to or less than the width between the outer edges of the left door mirror and the right door mirror.

4. A vehicle control device according to any one of claims 1 to 3, wherein the vehicle exterior image is an image in front of the vehicle or an image behind the vehicle.

5. A vehicle control device according to any one of claims 1 to 4, wherein the overhead image includes an image of the vehicle and an image of an obstacle around the vehicle.

6. A vehicle control device according to any one of claims 1 to 5, wherein the overhead image is generated based on the camera image of the vehicle-mounted camera installed on the door mirror of the vehicle.

7. A vehicle control device according to any one of claims 1 to 5, wherein the width of the second predicted course is changed depending on the deployed state and retracted state of the door mirrors of the vehicle.

8. A display control method executed by a controller for controlling the display on a display, wherein the controller: acquires camera images from an onboard camera that captures images of the surroundings of the vehicle; generates, based on the camera images, an external vehicle image viewed from the vehicle and an overhead image of the vehicle from a virtual viewpoint; outputs the external vehicle image and the overhead image to a display; when displaying the overhead image on the display, controls the display so that a first predicted course indicating the predicted course of the vehicle is superimposed on the overhead image; when displaying the external vehicle image on the display, controls the display so that a second predicted course indicating the predicted course of the vehicle is superimposed on the external vehicle image; and the width of the first predicted course is greater than the width of the second predicted course.

9. A recording medium on which a control program executed by a controller is recorded, the control program causing the controller to execute processing including the steps of: acquiring camera images from an onboard camera that captures images of the surroundings of the vehicle; generating, based on the camera images, an external vehicle image viewed from the vehicle and an overhead image of the vehicle from a virtual viewpoint; outputting the external vehicle image and the overhead image to a display; controlling the display, when displaying the overhead image on the display, so that a first predicted course indicating the predicted course of the vehicle is superimposed on the overhead image; and controlling the display, when displaying the external vehicle image on the display, so that a second predicted course indicating the predicted course of the vehicle is superimposed on the external vehicle image, wherein the width of the first predicted course is greater than the width of the second predicted course.

Images