Display control device, display control method, and recording medium

Abstract

Provided is a display control device comprising a controller 10 for acquiring camera images from a plurality of on-vehicle cameras for imaging surroundings of a vehicle, generating a display image of the vehicle as viewed from a virtual viewpoint on the basis of the plurality of camera images, and outputting the display image to a display 41. The controller 10 executes automatic display for moving the virtual viewpoint, and thus automatically switching display images either before the vehicle starts moving or after the vehicle completes parking, or both.

Description

Display control device, display control method, and recording medium

[0001] The present invention relates to a display control device, a display control method, and a recording medium.

[0002] Conventionally, there has been known a display control device that controls the display of an image showing the surroundings of a vehicle (for example, see Patent Document 1). The display control device described in Patent Document 1 includes an image generation unit that generates a three-dimensional image showing the surroundings of the vehicle based on images captured by a plurality of on-board cameras that capture the surroundings of the vehicle, and outputs a display image to be displayed on a display device based on the three-dimensional image, and a viewpoint change unit that, when a user swipes on the three-dimensional image displayed on the display device, changes a viewpoint parameter based on the position of the swipe and either the amount or speed of the swipe.

[0003] JP 2023-156871 A

[0004] The display control device described in Patent Document 1 displays a display image in response to a user's operation, regardless of the state of the vehicle, and therefore has a problem in that it is not possible to display an image that matches the state of the vehicle.

[0005] The problem to be solved by the present invention is to provide a display control device, a display control method, and a recording medium that can display an image that matches the state of the vehicle.

[0006] The present invention solves the above problem by performing automatic display in which the virtual viewpoint is moved and the displayed image is automatically switched at least either before the vehicle starts moving or after parking is completed.

[0007] According to the present invention, operability can be improved.

[0008] Fig. 1 is a block diagram showing the configuration of a display control system according to this embodiment. Fig. 2 is a plan view showing a part of an instrument panel. Fig. 3 is a diagram showing a display screen of a display. Fig. 4 is a diagram showing a display screen of a display. Fig. 5 is a diagram showing a display screen of a 3D view of the display. Fig. 6 is a diagram showing a display screen of a display. Fig. 7 is a diagram showing a display screen of a display. Fig. 8 is a flowchart of a display control method executed by a controller.

[0009] A display control system including a display control device according to an embodiment of the present invention will be described below with reference to the drawings. The display control system is a system that generates an image of the surroundings of a vehicle. FIG. 1 is a block diagram showing the configuration of the display control system according to this embodiment. The display control system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an ATCU 30, a display 41, a display 42, a vehicle speed sensor 50, and a display control device 100. The display control system 1 is installed in a vehicle. The front camera 21, the right camera 22, the left camera 23, the rear camera 24, the ATCU 30, the in-vehicle display 40, the vehicle speed sensor 50, and the display control device 100 are connected via an in-vehicle communication network such as CAN or LIN.

[0010] The front camera 21, right camera 22, left camera 23, and rear camera 24 are provided on the vehicle and capture images of the surroundings of the vehicle. The front camera 21, right camera 22, left camera 23, and rear camera 24 capture images of the surroundings of the vehicle while the vehicle is traveling. The front camera 21 captures images in front of the vehicle, the right camera 22 captures images on the right side of the vehicle, the left camera 23 captures images on the left side of the vehicle, and the rear camera 24 captures images on the rear of the vehicle. The front camera 21 is provided on the grille of the vehicle, the right camera 22 is provided on the right door mirror, the left camera 23 is provided on the left door mirror, and the rear camera 24 is provided on the back door. Note that the various cameras may be provided in other positions on the vehicle. Note that in the following description, the front camera 21, right camera 22, left camera 23, and rear camera 24 will be collectively referred to as camera 20.

[0011] The ATCU (automatic transmission control unit) 30 is a control device that controls the transmission. The ATCU 30 outputs a control signal including shift information to the display control device 100. The shift information indicates the shift position, and indicates the current state of the change lever, such as drive, neutral, reverse, or park.

[0012] The in-vehicle display 40 is a display included in an in-vehicle infotainment (IVI) system and includes a display 41 and a display 42. The displays 41 and 42 are provided on the vehicle's instrument panel. The displays 41 and 42 are touch-panel displays. FIG. 2 is a plan view showing a portion of the instrument panel. As shown in FIG. 2, the display 41 is provided in the center of the instrument panel. The display 42 is provided on the instrument panel in front of the driver. The display 41 displays, for example, a map for a navigation system, a menu screen (such as a selection screen for various applications) for an entertainment system, content video, etc. The display 42 displays, for example, meters, a range indicator showing the current range of the shift position, an icon showing the status of a seat belt, a vehicle image, an image of the vehicle's surroundings, a predicted course, a driving route, etc. The display 41 and / or the display 42 also display images generated by the display control device 100.

[0013] The displays 41 and 42 have display screens that are long horizontally along the vehicle width direction (the left-right direction on the paper surface of FIG. 2 ). As shown in FIG. 2 , the displays 41 and 42 are arranged side by side on the left and right along the vehicle width direction, with their display screens close to each other. Note that the vehicle does not need to be equipped with both the displays 41 and 42, and may be equipped with only one of the displays. The vehicle speed sensor 50 detects the vehicle speed and outputs the detected value to the controller 10.

[0014] The display control device 100 includes a controller 10 and a recording medium 19. The controller 10 has functions such as acquiring various data, such as camera images and vehicle data, and image processing functions. The controller 10 has, as functional blocks, an image acquisition unit 11, a vehicle data acquisition unit 12, and an image processing unit 13. The controller 10 stores programs for implementing various functions in the recording medium 19, and executes the control programs by causing a processor to execute the processing flow indicated by the programs. Note that the recording medium 19 storing the control programs may be stored as memory within the controller 10. The recording medium 19 may be located inside or outside the controller 10. For example, the recording medium 19 may be stored in a server outside the vehicle, and the display control device 100 may access the server and store the control program data recorded on the recording medium 19 in the memory within the controller 10. The display control device 100 may also execute the control program stored in the server via communication with the server. For example, the display control device 100 transmits camera images and commands for executing each processing step included in the control program to the server. A controller included in the server executes a control program and processes data such as camera images based on instructions from the display control device 100. The server transmits a display image obtained by processing the control program to the display control device 100. The display control device 100 transmits the display image to the displays 41 and 42, causing the image to be displayed.

[0015] The image acquisition unit 11 acquires camera images from the camera 20, etc. The vehicle data acquisition unit 12 acquires vehicle data from sensors mounted on the vehicle, etc., via an in-vehicle communication network. The vehicle data is data indicating the current state of the vehicle, and includes, for example, vehicle speed, steering angle, current vehicle position (current coordinates), and gear shift. The vehicle data acquisition unit 12 acquires vehicle speed from the vehicle speed sensor 50 and acquires steering angle from the steering angle sensor. The vehicle data acquisition unit 12 acquires current position information from a navigation system. The vehicle data acquisition unit 12 also acquires gear shift information from the ATCU 30.

[0016] The image processing unit 13 performs image processing on the camera image to display a current image of the vehicle's surroundings and / or a vehicle image on the displays 41, 42. The image processing unit 13 generates a display image to display the image on the displays 41, 42 in a display mode specified by the user or the system, and outputs the display image to the displays 41, 42. There are multiple display modes, such as a front view showing the front of the vehicle, a rear view showing the rear of the vehicle, a top view giving a bird's-eye view of the vehicle, a 3D view showing the vehicle or its surroundings from a virtual viewpoint, and a skeleton view. In the front view, an image showing the exterior of the vehicle is displayed, which is located in front of the vehicle's grille. In the rear view, an image showing the exterior of the vehicle is displayed, which is located behind the vehicle's tailgate. In the top view (around view), a bird's-eye view image of the vehicle viewed from a virtual viewpoint directly above the vehicle is displayed. In the top view, a two-dimensional image is displayed, and the vehicle image is shown as a plan view of the vehicle. If there are tall three-dimensional objects such as walls around the vehicle, an image of the surroundings is displayed, showing the three-dimensional objects as if they have fallen over. In the 3D view, a virtual viewpoint is used to display an image of the object from a virtual viewpoint. The object is not limited to the vehicle, but may also be an obstacle around the vehicle. The virtual viewpoint is a viewpoint that allows a virtual view of the object from outside the object around the object. The virtual viewpoint is movable. For example, when a display image (three-dimensional image) of the vehicle viewed from a virtual viewpoint is displayed on the displays 41 and 42, the three-dimensional vehicle image can be rotated by orbiting the virtual viewpoint along a plane around the vehicle. In addition, the displays 41 and 42 may display the surroundings of the vehicle in other display modes, such as a skeleton view, a front wide view, or a rear wide view. In the skeleton view, a virtual image showing the condition of the vehicle's underfloor is displayed in addition to a current camera image (real image) showing the surroundings of the vehicle. For example, when a vehicle is turning left or right at an intersection with poor visibility such as a T-junction, the front wide view can display an image of the area in front of the vehicle that is in the driver's blind spot from an image taken by a front camera 21 having a wide-angle lens.

[0017] The display mode can be specified by the user or can be changed by the vehicle system as appropriate depending on the driving state of the vehicle. For example, when the range is shifted to reverse to park the vehicle, the display mode may be automatically switched to a top view or a rear view. Alternatively, the display mode may be switched by the driver operating a switch or the like while the vehicle is moving.

[0018] The image processing unit 13 can display images in multiple display modes on multiple screens on the display 41. The image processing unit 13 may display images in different display modes on the displays 41 and 42, or may display images in the same display mode on the displays 41 and 42. The image processing unit 13 may also use both the screens of the displays 41 and 42 as a single display screen to display images in multiple display modes, or may display images in a single display mode. The following description will explain the image processing performed by the image processing unit 13 when displaying an image on the display 41, but when displaying an image on the display 42, the image processing performed on the display 41 may be applied to the image display on the display 42.

[0019] The image processing unit 13 generates a display image according to the display mode based on the multiple camera images acquired by the image acquisition unit 11, and outputs the display image to the display 41. When displaying an image in front view or rear view, the image processing unit 13 generates a display image showing the front or rear of the vehicle based on the camera image from the front camera 21 or the rear camera 24.

[0020] When displaying an image in top view, the image processing unit 13 generates an overhead image of the vehicle based on multiple camera images captured by the cameras 20. The virtual viewpoint of the overhead image is set at a predetermined position vertically away from the roof of the vehicle and outside the vehicle. For example, the image processing unit 13 performs viewpoint conversion processing on images captured by the front camera 21, right camera 22, left camera 23, and rear camera 24 to generate viewpoint-converted images that show the front, rear, left, and right regions of the vehicle from the virtual viewpoint. That is, the image processing unit 13 converts camera images corresponding to images viewed from the camera positions into viewpoint-converted images viewed from the virtual viewpoint. The viewpoint conversion from the camera positions to the virtual viewpoint is performed by calculation using viewpoint parameters, and the viewpoint parameters are stored in the recording medium 19. The image processing unit 13 fits each image after the viewpoint conversion processing to the surrounding area of ​​the vehicle. The image processing unit 13 performs blending processing on overlapping portions of the camera images from the cameras 20. The blending process is a process for maintaining continuity between two overlapping images, and any method known at the time of filing of this application can be used. The image after the blending process becomes a surrounding image showing the surroundings of the vehicle. The image processing unit 13 also combines the vehicle image located at the center of the screen with the surrounding image. The combined image becomes an overhead image, and the image processing unit 13 outputs the overhead image to the display 41 as a display image.

[0021] When displaying an image in 3D view, the image processing unit 13 generates a display image of the vehicle as viewed from a virtual viewpoint based on multiple camera images captured by the camera 20. First, the image processing unit 13 sets a virtual viewpoint based on a user specification or the driving situation. The position and line of sight of the virtual viewpoint can be arbitrarily set based on a user instruction. The user can specify the position and line of sight of the virtual viewpoint by, for example, touching the display 41. The image processing unit 13 can also set the virtual viewpoint based on the driving situation. For example, the virtual viewpoint can be set behind the vehicle when driving forward to view an image in front of the vehicle, and in front of the vehicle when driving backward to view an image behind the vehicle. Specifically, the image processing unit 13 sets the position of the virtual viewpoint when it receives shift information indicating that the gearshift position is in reverse. The image processing unit 13 can also set the virtual viewpoint based on the vehicle speed. For example, the virtual viewpoint can be set based on the vehicle speed input from the vehicle speed sensor 50 so that the virtual viewpoint is farther away from the vehicle when the vehicle speed is equal to or greater than a predetermined value, and the virtual viewpoint is closer to the vehicle when the vehicle speed is less than the predetermined value.

[0022] The image processing unit 13 deforms at least a part of a reference curved surface coordinate system having a predefined curved surface for projecting the camera image, according to the position of the virtual viewpoint. The reference curved surface coordinate system is stored in the recording medium 19. The reference curved surface coordinate system has, for example, a bowl shape surrounding the vehicle. The reference curved surface coordinate system has a center on an xy plane parallel to the vehicle's loading surface (travel surface), and is formed from a curved surface having a curvature from the center or near the center and a component in the vehicle's height direction (z direction).

[0023] A base parallel to the xy plane can be formed near the center of the reference curved surface coordinate system, and the base can have any shape, such as a rectangle, a triangle, an ellipse, or a circle. The curvature of the curved surface of the reference curved surface coordinate system does not have to be uniform. For example, the curvature of the curved surface near the base may be relatively large, and the curvature of the curved surface away from the base may be relatively small. The image processing unit 13 may store in advance multiple reference curved surface coordinate systems with different shapes depending on the shape (size, form) of the vehicle. When multiple reference curved surface coordinate systems are stored, the reference curved surface coordinate systems to be used may be stored in the recording medium 19 in association with the vehicle speed, the vehicle shift position, etc., and the image processing unit 13 may select a reference curved surface coordinate system depending on the vehicle speed, the vehicle shift position, etc.

[0024] The image processing unit 13 may read a pre-stored reference curved surface coordinate system and deform the shape of the read reference curved surface coordinate system. The image processing unit 13 may deform the shape of the reference curved surface coordinate system according to the position of the virtual viewpoint. For example, when the virtual viewpoint is located outside the outer edge of the reference curved surface coordinate system, a portion of the reference curved surface coordinate system may be deformed so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system. By deforming a portion of the reference curved surface coordinate system so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system, distortion of the shape of an obstacle with height can be suppressed when the obstacle is displayed on the screen, and an image without any missing image can be displayed on the display 41. Note that the method of deforming the reference curved surface coordinate system may use other coordinate system deformation methods known at the time of filing.

[0025] The image processing unit 13 projects multiple camera images acquired from the camera 20 onto the reference curved coordinate system. When the reference curved coordinate system is deformed, the image processing unit 13 simply projects the camera images onto the curved coordinate system after the deformation (deformable coordinate system). In order to project the camera image data onto the reference curved coordinate system or the deformed coordinate system after the deformation, the image processing unit 13 has an image conversion table that associates the coordinates of pixels included in the camera images with the coordinates of the coordinate system. The image conversion table is recorded on the recording medium 19.

[0026] A prepared vehicle image of a vehicle may also be superimposed on the reference curved coordinate system. The vehicle image may be created in advance based on the design of the vehicle and stored in the recording medium 19. By superimposing the vehicle image on the reference curved coordinate system, not only the image of the vehicle's surroundings but also the vehicle itself can be displayed in three dimensions, which helps understand the relationship between the vehicle's position and orientation and the surrounding image. In this way, the image processing unit 13 refers to the image conversion table to associate the coordinates of pixels contained in the camera image with the coordinates of the coordinate system, projects the camera image onto the reference curved coordinate system or the deformed coordinate system, and superimposes the vehicle image on the projected three-dimensional image to generate a display image (three-dimensional image) in a 3D view in which the vehicle and its surroundings are viewed from a virtual viewpoint.

[0027] The image processing unit 13 may generate display images viewed from selectable virtual viewpoints, not limited to those specified by the user or set based on the driving situation. For example, when a display image is displayed on the display 41 in a 3D view, the user can freely move the position of the virtual viewpoint by touching the display 41. When the 3D view display is selected, the image processing unit 13 generates multiple 3D view display images for each of multiple virtual viewpoints selectable by the user. Then, for example, when the user moves the virtual viewpoint by touching the display 41, a 3D image in which the vehicle image rotates around the center point of a curved coordinate system is displayed on the display 41. Note that the image processing unit 13 may set virtual viewpoints at any positions around the vehicle, not limited to those selectable by the user, and generate multiple 3D view display images for each virtual viewpoint. In other words, when a 3D image is displayed on the display 41 in a rotating display, the image processing unit 13 generates a continuous 3D image so that the image can be displayed smoothly in response to the movement of the virtual viewpoint.

[0028] The image processing method for the display images displayed in the front / rear view, top view, and 3D view display modes is not limited to the above, and other image processing methods can be used. Furthermore, the image processing method for the display images displayed in other display modes, such as skeleton view and front / rear wide view, can be any method known at the time of filing.

[0029] Next, display control by the display control device 100 will be described along with specific examples of the display screen of the display 41. FIG. 3 shows the display screen of the display 41. In the example of FIG. 3, the vehicle is stopped on a straight road, and the left screen of the display 41 displays a top-view image, while the right screen of the display 41 displays a 3D-view image. When displaying a top-view overhead image and a 3D-view display image on the display 41, the controller 10 generates a top-view image 46 by superimposing viewpoint icons 43a-43h and an automatic display start icon 44 on the overhead image of the vehicle. The top-view image 46 includes an overhead image of the vehicle viewed from above, viewpoint icons 43a-43h, and the automatic display start icon 44. The viewpoint icons 43a-43h indicate the position of a virtual viewpoint and / or the direction of the line of sight and are user-selectable icons. The viewpoint icons 43a-43h are arranged around the vehicle, with the vehicle image 45 at the center. Viewpoint icon 43a is positioned above the vehicle image located at the center of the overhead image, while viewpoint icons 43c, 43e, and 43g are positioned to the right, bottom, and left of the vehicle image, respectively. Viewpoint icon 43b, viewpoint icon 43d, viewpoint icon 43f, and viewpoint icon 43h are positioned in the upper right, lower right, lower left, and lower right directions, respectively, of the vehicle image. When the user selects one of viewpoint icons 43a to 43h, a display image (three-dimensional image) of the vehicle as viewed from the selected viewpoint icon is displayed on the right side of display 41.

[0030] As shown in FIG. 3 , when the user touches the viewpoint icon 43a, the controller 10 displays the viewpoint icon 43a as a selection icon. The controller 10 displays the selection icon by, for example, changing the color of the viewpoint icon 43a. In the example of FIG. 3 , the selection icon is represented by a thick line surrounding the viewpoint icon 43a. The image processing unit 13 sets a virtual viewpoint corresponding to the viewpoint icon 43a. This allows the virtual viewpoint to be set according to the user's specification. In the example of FIG. 3 , a position from which the vehicle is viewed from the front is set as the virtual viewpoint. Then, the image processing unit 13 generates a three-dimensional image 47 of the vehicle and its surroundings viewed from the front of the vehicle based on multiple camera images. The image processing unit 13 outputs the three-dimensional image to the display 41 as a display image. The three-dimensional image 47 is displayed on the right screen of the display 41. Then, as shown in FIG. 3 , the display 41 displays a top-view image 46 and the three-dimensional image 47 side by side.

[0031] The automatic display start icon 44 is an icon for starting automatic display, which automatically switches three-dimensional images in the 3D view by moving the virtual viewpoint. The automatic display start icon 44 is displayed at the center position of the vehicle image 45 included in the top-view image 46. When automatic display is executed, predetermined positions around the vehicle become the start and end points of the virtual viewpoint, and the virtual viewpoint moves from the start point to the end point, causing the vehicle image displayed in the 3D view to rotate 360 ​​degrees along a plane parallel to the road surface. In other words, in automatic display, the center point is placed on the vehicle image, and the vehicle image and the vehicle-surrounding image rotate 360 ​​degrees around the center point. For example, in the example of FIG. 3 , when the user touches the automatic display start icon 44, the selected viewpoint icon 43 a becomes the start and end points of the virtual viewpoint. The virtual viewpoint moves from the position corresponding to the viewpoint icon 43 a to viewpoint icon 43 b in order to viewpoint icon 43 h, rotating clockwise around the vehicle, and then returns to the position corresponding to the original viewpoint icon 43 a. As the virtual viewpoint rotates, the three-dimensional image 47 in the 3D view rotates. The orientation of the vehicle image included in the three-dimensional image 47 starts from the front (front) as shown in Fig. 3, changes in the order of the right side, rear (back), and left side, and then returns to the original front (front). In other words, the three-dimensional image is displayed as if it were rotated 360 degrees (hereinafter also referred to as "360-degree rotation display"). As a result, the display image (three-dimensional image 47) is automatically rotated 360 degrees in the 3D view.

[0032] FIG. 4 shows a display screen of the display 41. In the example of FIG. 4, the vehicle is stopped at a T-junction, and the left screen of the display 41 displays a top-view image, while the right screen of the display 41 displays a 3D-view image. The image processing method for the top-view image 46 and the 3D image 47 is the same as in the example of FIG. 3, and the top-view image 46 also includes viewpoint icons 43a-h and an automatic display start icon 44. Unlike the example of FIG. 3, viewpoint icon 43e is selected, and the 3D view displays a 3D image 45 of the vehicle and its surroundings as seen from behind the vehicle. Note that, as shown in FIG. 4, when the vehicle is stopped at a T-junction, the controller 10 may set a virtual viewpoint corresponding to viewpoint icon 43a and display a 3D image 47 as seen from the virtual viewpoint on the display 41. This allows the state of the blind spot to be displayed on the display 41 if there is a blind spot ahead of the vehicle.

[0033] 3 and 4, while the vehicle is stopped, the user can select one of the viewpoint icons 43a to 43h and view a three-dimensional image 47 from the direction of the selected viewpoint icon 43a to 43h in a 3D view on the display 41. For example, if the user touches the position of the viewpoint icon 43c on the screen of the display 41, the image processing unit 13 switches the viewpoint icon 43c to a selected icon and moves the current virtual viewpoint (in the example of FIG. 3, the virtual viewpoint corresponding to the viewpoint icon 43a) to the virtual viewpoint corresponding to the viewpoint icon 43c. The image processing unit 13 then generates a three-dimensional image viewed from the virtual viewpoint based on the camera image and outputs the display image to the display 41 so that the three-dimensional image is displayed in a 3D view. A three-dimensional image of the vehicle viewed from the right side is displayed on the display 41.

[0034] Furthermore, while the vehicle is stopped, the display control device 100 may perform display control so that the user can not only select a virtual viewpoint by touching the viewpoint icons 43a to 43h, but also arbitrarily specify the position of the virtual viewpoint. The user touches any position on the 3D view display screen. Then, while the user's finger is touching the screen, the user moves the finger in any direction (swipe operation). While the finger is touching the screen, the position of the virtual viewpoint moves in accordance with the swipe operation, and the 3D image rotates in accordance with the movement of the virtual viewpoint. The amount of movement of the virtual viewpoint is determined by the swipe amount (corresponding to the amount the finger moves while touching the screen), and the greater the swipe amount, the greater the movement amount. Note that the amount of movement of the virtual viewpoint corresponds to the length of the arc of a circle centered on the vehicle image. For example, the user touches the 3D view display screen with their finger and moves their finger with a large swipe amount. If the swipe amount is large, the image processing unit 13 moves the virtual viewpoint around the center point of the vehicle image, thereby rotating the 3D image 47. If the swipe amount is small, image processing unit 13 rotates three-dimensional image 47 so that the virtual viewpoint moves along an arc centered on the center point of the vehicle image by an amount equivalent to the swipe amount. Note that the rotational display caused by a swipe operation is not limited to the 3D view display screen, but may also be performed by a swipe operation on a top view display screen. For example, image processing unit 13 causes display 41 to display an image in which viewpoint icons 43a to 43h have been removed from a top view image 46 as shown in FIG. 3 or 4 . Then, the user performs a swipe operation at any position on the screen displaying top view image 46. Image processing unit 13 rotates the three-dimensional image on the 3D view display screen in response to the swipe operation on the top view display screen.

[0035] 5A and 5B are diagrams illustrating the rotational display of a three-dimensional image 47 in a 3D view. Fig. 5A shows the three-dimensional image 47 viewed from a virtual viewpoint corresponding to viewpoint icon 43e, Fig. 5B shows the three-dimensional image 47 viewed from a virtual viewpoint corresponding to viewpoint icon 43f, and Fig. 5C shows the three-dimensional image 47 viewed from a virtual viewpoint corresponding to viewpoint icon 43g. Note that Fig. 5 is a 3D view display screen of the display 41, and shows only the vehicle image included in the three-dimensional image 47, omitting an image of the vehicle's surroundings. The virtual viewpoint that serves as the starting point for the rotational display is the position corresponding to viewpoint icon 43e.

[0036] As shown in FIG. 5A, the state in which the vehicle image included in the three-dimensional image 47 faces rearward (backward) is the starting point of the rotational display. As the virtual viewpoint moves, the vehicle image rotates 45 degrees, resulting in the state shown in FIG. 5B. Further rotation results in the state shown in FIG. 5C. The three-dimensional image 47 has rotated 90 degrees so far, and then rotates the remaining 270 degrees. That is, the orientation of the vehicle image included in the three-dimensional image 47 starts from the rearward (backward) side, changes in this order to the left side, front (front), and right side, and then returns to the original rearward (backward) side. The surrounding images located around the vehicle image also rotate primarily along with the vehicle image. As a result, the display image (three-dimensional image 47) is automatically rotated 360 degrees in the 3D view.

[0037] In this embodiment, the virtual viewpoint is moved and the display image is automatically switched at least either before the vehicle starts or after parking is completed. The following describes the automatic display control that is executed before the vehicle starts and after parking is completed.

[0038] First, the control of automatic display before the vehicle starts will be described. The controller 10 determines whether the vehicle has not yet started based on the vehicle data acquired by the vehicle data acquisition unit 12. Before the vehicle starts, the vehicle is in a state where it is capable of running and is stopped (the vehicle speed is equal to or less than a predetermined threshold). The controller 10 detects the state of the vehicle's main switch from the vehicle data. The vehicle's main switch is a switch that switches whether the vehicle is capable of running. In a vehicle with an engine, the ignition switch corresponds to the main switch. The vehicle's main switch is also called a power switch or a starter switch. When the controller 10 detects that the main switch is in an on state, it determines that the vehicle has not yet started.

[0039] The controller 10 may determine that the vehicle has not yet started when it detects that the main switch has been switched from off to on. The controller 10 may also determine that the vehicle has not yet started when the main switch is on and the shift position is other than reverse. Positions other than reverse include, for example, parking, neutral, and drive. The controller 10 may also determine that the vehicle has not yet started when the main switch is on and the vehicle speed is equal to or lower than a predetermined vehicle speed threshold. Note that "not yet started" may be, for example, a time before a predetermined time (for example, several tens of seconds or several minutes) has elapsed since the main switch was switched from off to on.

[0040] If it is determined that the vehicle has not yet started moving, the image processing unit 13 performs a 360-degree automatic rotation display of the display image (three-dimensional image 47) in the 3D view. FIG. 6 shows the display screen of the display 41. The example of FIG. 6 illustrates a state in which the vehicle has temporarily stopped to be parked facing forward in a nearby parking space ahead. As shown in FIG. 6, a top-view image 46 is displayed on the left screen of the display 41, and a three-dimensional image 47 is displayed on the right screen. If the vehicle has not yet started moving and the shift position is other than reverse, the image processing unit 13 sets positions behind the vehicle as the start and end points of the virtual viewpoint. In the example of FIG. 6, the positions corresponding to the viewpoint icon 43e are the start and end points of the virtual viewpoint. The start and end points of the virtual viewpoint are the start and end points when the three-dimensional image 47 is rotated 360 degrees and displayed. The image processing unit 13 moves the position of the virtual viewpoint from the position corresponding to viewpoint icon 43e to the positions corresponding to each of the viewpoint icons 43f to 43h, 43a to 43e, in that order. That is, the virtual viewpoint moves clockwise around one circle from viewpoint icon 43e in the top-view image 46, as indicated by the dotted arrow in FIG. 6 . Note that during the rotation display, the image processing unit 13 may superimpose a guide image indicating the direction of rotation, as indicated by the dotted arrow in FIG. 6 , on the top-view image 46 and display it on the top-view screen. The image processing unit 13 generates a three-dimensional image 47 in the 3D view so that the three-dimensional image 47 rotates 360 degrees in accordance with the movement of the virtual viewpoint, and outputs the image to the display 41. When the virtual viewpoint is located at the position corresponding to viewpoint icon 43e, the image processing unit 13 generates a three-dimensional image in which the vehicle is displayed facing backward on the display 41, as shown in FIG. 6 . This allows the user to check the surroundings of the vehicle in a 3D view before starting the vehicle by automatically rotating the three-dimensional image 47 360 degrees.

[0041] Furthermore, when automatically rotating and displaying the three-dimensional image 47 by 360 degrees, the image processing unit 13 may display selection icons on the top-view screen in accordance with the movement of the virtual viewpoint. First, because the starting point of the virtual viewpoint is the position corresponding to viewpoint icon 43e, the image processing unit 13 displays viewpoint icon 43e as the selection icon. When the three-dimensional image 47 rotates by 45 degrees, the virtual viewpoint moves from the position corresponding to viewpoint icon 43e to the position corresponding to viewpoint icon 43f, so the image processing unit 13 displays viewpoint icon 43e as a normal icon and viewpoint icon 43f as the selection icon. Then, the image processing unit 13 moves the selection icons in the order of viewpoint icons 43g to 43h, and 43a to 43e in accordance with the rotation of the three-dimensional image 47 by 45 degrees. In other words, when the vehicle has not yet started moving and the shift position is other than reverse, the image processing unit 13 starts displaying the selection icons from viewpoint icon 43e, which is located behind the vehicle image 44 in the top view. 6, the image processing unit 13 moves the display of the selected icon so that it rotates 360 degrees around the vehicle image 44. This allows the user to check the position of the virtual viewpoint in a top view while the three-dimensional image 47 is being displayed in 360-degree automatic rotation.

[0042] Next, the control of the automatic display after parking is complete will be described. The controller 10 determines whether parking has been completed based on the vehicle data acquired by the vehicle data acquisition unit 12. After parking is completed, the vehicle cannot be driven and is parked and stopped (the vehicle speed is equal to or less than a predetermined threshold). The controller 10 detects the state of the vehicle's main switch and / or the shift position from the vehicle data. If the main switch is in the off state, the controller 10 determines that parking has started.

[0043] The controller 10 may determine that parking is complete when it detects that the main switch has been switched from on to off. The controller 10 may determine that parking is complete when it detects that the shift position has been switched from a position other than park to park. The controller 10 may also obtain information indicating the on / off status of the electric parking brake from vehicle data and determine that parking is complete when the electric parking brake is turned on. The term "parking is complete" may be, for example, a time before a predetermined time (e.g., several tens of seconds or several minutes) has elapsed since the shift position was switched from a position other than park to park, the main switch was switched from on to off, or the electric parking brake was switched from off to on.

[0044] If it is determined that parking has been completed, the image processing unit 13 performs a 360-degree automatic rotation display of the display image (three-dimensional image 47) in the 3D view. FIG. 7 shows the display screen of the display 41. In the example of FIG. 7, the shift position is in reverse, representing a state in which the vehicle is temporarily stopped or parked. As shown in FIG. 7, a top-view image 46 is displayed on the left screen of the display 41, and a three-dimensional image 47 is displayed on the right screen. When parking has been completed and the shift position is in reverse, the image processing unit 13 sets the positions in front of the vehicle as the start and end points of the virtual viewpoint. In the example of FIG. 7, the positions corresponding to the viewpoint icon 43a are the start and end points of the virtual viewpoint. The start and end points of the virtual viewpoint are the start and end points when the three-dimensional image 47 is rotated 360 degrees and displayed. The image processing unit 13 moves the position of the virtual viewpoint from the position corresponding to the viewpoint icon 43a to the positions corresponding to each of the viewpoint icons 43b to 43h, 43a, in that order. That is, the movement of the virtual viewpoint rotates clockwise from viewpoint icon 43a in top view image 46, as indicated by the dotted arrow in FIG. 7 . Note that, during the rotation display, image processing unit 13 may superimpose a guide image indicating the direction of rotation, as indicated by the dotted arrow in FIG. 7 , on top view image 46 and display it on the top view screen. Image processing unit 13 generates three-dimensional image 47 so that, in the 3D view, three-dimensional image 47 rotates 360 degrees in accordance with the movement of the virtual viewpoint, and outputs the generated three-dimensional image to display 41. When the virtual viewpoint is located at a position corresponding to viewpoint icon 43a, image processing unit 13 generates a three-dimensional image in which the vehicle is displayed on display 41 facing forward, as shown in FIG. 7 . This allows the user to check the surroundings of the vehicle in a 3D view after parking is complete, thanks to the automatic 360-degree rotation of three-dimensional image 47.

[0045] Next, a display control method by the controller 10 will be described with reference to Fig. 8. Fig. 8 is a flowchart of the display control method by the controller 10. The controller 10 repeatedly executes the following control flow while the left display image on the display 41 is displayed in top view and the right display image is displayed in 3D view.

[0046] In step S1, the image acquisition unit 11 acquires a camera image from the camera 20. The vehicle data acquisition unit 12 acquires vehicle data from the ATCU 30 and / or the vehicle speed sensor 50, etc. In step S2, the image processing unit 13 detects the current vehicle state based on the vehicle data. The vehicle state is detected to determine at least one of the timing before the vehicle starts and the timing after parking is completed, and is detected based on, for example, the state of the vehicle's main switch, the shift position, the vehicle speed, etc.

[0047] In step S3, the image processing unit 13 generates a top-view image of the vehicle as seen from above, based on the multiple camera images. In step S4, the image processing unit 13 generates a three-dimensional image of the vehicle as seen from a virtual viewpoint, based on the multiple camera images. In step S5, the controller 10 determines whether the vehicle has not yet started moving. If it is determined that the vehicle has not yet started moving, in step S6, the controller 10 sets the rear position of the vehicle as the start point and end point of the virtual viewpoint. The control flow proceeds to step S9.

[0048] Returning to step S5, if it is determined that the vehicle has not yet started, the controller 10 determines whether parking has been completed in step S7. If it is determined that parking has been completed, the controller 10 sets the front position of the vehicle as the start point and end point of the virtual viewpoint in step S8. The control flow proceeds to step S9.

[0049] In step S9, the controller 10 outputs the top-view image and the three-dimensional image to the display 41, and performs automatic 360-degree rotation display by moving the virtual viewpoint from the start point to the end point and automatically switching the three-dimensional image.

[0050] Returning to step S7, if controller 10 determines in step S7 that it is not the timing after parking has been completed, then in step S10, controller 10 executes normal display. In normal display, controller 10 outputs a top-view image and a three-dimensional image to display 41 so that, for example, a three-dimensional image as seen from a virtual viewpoint specified by the user and a top-view image are displayed on display 41. Then, after the control flow of steps S9 and S10 is executed, the control flow ends. Controller 10 restarts the control flow from step S1.

[0051] As described above, in this embodiment, the controller 10 acquires camera images from the camera 20, generates a display image of the vehicle as seen from a virtual viewpoint based on the multiple camera images, and outputs the display image to the display 41. The controller 10 also performs automatic display, moving the virtual viewpoint and automatically switching the display image at least either before the vehicle starts moving or after parking is completed. This enables the display 41 to display images according to the vehicle state, and provides the user with an appropriate display image that matches the vehicle state. As a result, display control of the display 41 with high operability can be achieved.

[0052] In this embodiment, the automatic display is a display in which predetermined positions around the vehicle are set as the start and end points of a virtual viewpoint, and the virtual viewpoint is moved from the start point to the end point, rotating the display image 360 ​​degrees. When the vehicle is in a state where it can be driven and the shift position is in a position other than reverse, the controller 10 determines that the vehicle has not yet started, and sets the rear position of the vehicle as the start and end point of the virtual viewpoint. When the main switch of the vehicle is turned off, the controller 10 determines that the vehicle has been parked, and sets the front position of the vehicle as the start and end point of the virtual viewpoint. This allows the occupant to view an image at a time when they want to check the situation around the vehicle.

[0053] Furthermore, in this embodiment, controller 10 outputs to display 41 a top view image 46 that includes viewpoint icons 43a-43h indicating the position of a virtual viewpoint and shows the vehicle from above, and among the multiple viewpoint icons 43a-43h arranged around vehicle image 44 included in top view image 46, controller 10 displays, as a selection icon, the viewpoint icon 43a-43h that corresponds to the virtual viewpoint of the display image currently being displayed on display 41. Before the vehicle starts moving, controller 10 starts the selection icon from the viewpoint icon located behind vehicle image 44 in the top view. Furthermore, after parking is complete, controller 10 starts the selection icon from the viewpoint icon 43a-43h located in front of vehicle image 44 in the top view. This makes it possible to confirm, in the top view, from which position the three-dimensional image displayed in the 3D view is being viewed.

[0054] In this embodiment, the selected icon moves around the vehicle image 44 in accordance with the automatic display of the three-dimensional image 47. This allows the movement of the virtual viewpoint to be confirmed in a top view.

[0055] In this embodiment, the controller 10 may automatically display the three-dimensional image 47 either before the vehicle starts or after parking is completed, or only before the vehicle starts, or only after parking is completed.

[0056] In this embodiment, the automatic display of the three-dimensional image 47 does not necessarily have to be a 360-degree rotation display, but may be a rotation display of, for example, 90 degrees, 135 degrees, or 180 degrees. For example, before the vehicle starts moving, the controller 10 may execute a rotation display in which the virtual viewpoint rotates 180 degrees in front of the vehicle by moving the position of the virtual viewpoint from the position corresponding to viewpoint icon 43g to the positions corresponding to each of the viewpoint icons 43h, 43a, 43b, 43c, in that order, in order to display the surroundings in front of the vehicle. Furthermore, after parking is completed, the controller 10 may execute a rotation display in which the virtual viewpoint rotates 180 degrees in front of the vehicle by moving the position of the virtual viewpoint from the position corresponding to viewpoint icon 43c to the positions corresponding to each of the viewpoint icons 43d, 43c, 43d, 43g, in that order, in order to display the surroundings behind the vehicle.

[0057] In this embodiment, the controller 10 does not need to execute all of the control flows shown in FIG. 8, and does not need to execute each control flow in the order shown in FIG.

[0058] REFERENCE SIGNS LIST 1 display control system 10 controller 11 image acquisition unit 12 vehicle data acquisition unit 13 image processing unit 19 recording medium 20 camera 40 in-vehicle display 43a to 43h viewpoint icon 44 automatic display start icon 45 three-dimensional image 46 top-view image 50 vehicle speed sensor 100 display control device

Claims

1. A display control device comprising: a controller that acquires camera images from multiple on-board cameras that capture images of the surroundings of a vehicle, generates a display image of the vehicle as seen from a virtual viewpoint based on the multiple camera images, and outputs the display image to a display; the controller performs automatic display by moving the virtual viewpoint and automatically switching the display image at least either before the vehicle starts moving or after parking is completed.

2. A display control device according to claim 1, wherein the automatic display is a display in which predetermined positions around the vehicle are set as the start and end points of the virtual viewpoint, the virtual viewpoint is moved from the start point to the end point, and the display image is rotated 360 degrees, and the controller determines that the vehicle has not yet started moving when the vehicle is in a state in which it can be driven and the shift position is in a position other than reverse, and sets the rear position of the vehicle as the start and end point of the virtual viewpoint, and determines that parking has been completed when the main switch of the vehicle is turned off, and sets the front position of the vehicle as the start and end point of the virtual viewpoint.

3. A display control device according to claim 1 or 2, wherein the controller includes a viewpoint icon indicating the position of the virtual viewpoint, outputs a top view image of the vehicle viewed from above to the display, and displays, as a selection icon, one of the viewpoint icons arranged around a vehicle image included in the top view image that corresponds to the virtual viewpoint of the display image currently being displayed on the display, and when the vehicle has not yet started moving, starts the selection icon from the viewpoint icon located behind the vehicle image in the top view, and when parking has been completed, starts the selection icon from the viewpoint icon located in front of the vehicle image in the top view.

4. A display control device according to claim 3, wherein the selected icon moves around the vehicle image in accordance with the automatic display.

5. A display control method executed by a controller to control what is displayed on a display, wherein the controller: acquires camera images from a plurality of on-board cameras that capture images of the surroundings of a vehicle; generates a display image of the vehicle as seen from a virtual viewpoint based on the plurality of camera images; outputs the display image to a display; and moves the virtual viewpoint to automatically switch the display image at least either before the vehicle starts moving or after parking is completed.

6. A recording medium on which a control program executed by a controller is recorded, the control program causing the controller to execute processes including the steps of: acquiring camera images from a plurality of onboard cameras that capture images of the surroundings of the vehicle; generating a display image of the vehicle as viewed from a virtual viewpoint based on the plurality of camera images; outputting the display image to a display; and moving the virtual viewpoint to automatically switch the display image at least either before the vehicle starts moving or after parking is completed.

Images