Display control device, display control method, and recording medium

Abstract

Provided is a display control device comprising a controller 10 that: acquires a camera image from a plurality of on-vehicle cameras for imaging the surroundings of a vehicle; generates, on the basis of the plurality of camera images, a display image when the vehicle is viewed from a virtual viewpoint; and outputs the display image to a display 41. The controller 10 moves the position of the virtual viewpoint by a user operation, detects the vehicle speed of the vehicle, and restricts the movement of the virtual viewpoint by the user operation in accordance with the vehicle speed that has been detected.

Description

Display control device, display control method, and recording medium

[0001] The present invention relates to a display control device, a display control method, and a recording medium.

[0002] Conventionally, there has been known a display control device that controls the display of an image showing the surroundings of a vehicle (for example, see Patent Document 1). The display control device described in Patent Document 1 includes an image generation unit that generates a three-dimensional image showing the surroundings of the vehicle based on images captured by a plurality of on-board cameras that capture the surroundings of the vehicle, and outputs a display image to be displayed on a display device based on the three-dimensional image, and a viewpoint change unit that, when a user swipes on the three-dimensional image displayed on the display device, changes a viewpoint parameter based on the position of the swipe and either the amount or speed of the swipe.

[0003] JP 2023-156871 A

[0004] The display control device described in Patent Document 1 has a problem in that it is difficult for the user to operate because it can display a display image that does not correspond to the state of the vehicle.

[0005] The problem to be solved by the present invention is to provide a display control device, a display control method, and a recording medium that can improve operability.

[0006] The present invention solves the above problem by generating a display image of a vehicle viewed from a virtual viewpoint based on a camera image, outputting the display image to a display, and limiting the movement of the virtual viewpoint through user operation depending on the vehicle speed.

[0007] According to the present invention, operability can be improved.

[0008] FIG. 1 is a block diagram showing the configuration of a display control system according to this embodiment. FIG. 2 is a plan view showing a portion of an instrument panel. FIG. 3 is a diagram showing a display screen of a display. FIG. 4 is a diagram showing a display screen of a display. FIG. 5 is a diagram showing a display screen of a 3D view of the display. FIG. 6 is a flowchart of a display control method executed by a controller. FIG. 7 is a diagram showing a display screen of a display. FIG. 8 is a diagram showing a display screen of a display. FIG. 9 is a diagram showing a display screen of a display. FIG. 10 is a diagram showing a display screen of a display. FIG. 11 is a diagram showing a display screen of a display.

[0009] A display control system including a display control device according to an embodiment of the present invention will be described below with reference to the drawings. The display control system is a system that generates an image of the surroundings of a vehicle. FIG. 1 is a block diagram showing the configuration of the display control system according to this embodiment. The display control system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an ATCU 30, a display 41, a display 42, a vehicle speed sensor 50, and a display control device 100. The display control system 1 is installed in a vehicle. The front camera 21, the right camera 22, the left camera 23, the rear camera 24, the ATCU 30, the in-vehicle display 40, the vehicle speed sensor 50, and the display control device 100 are connected via an in-vehicle communication network such as CAN or LIN.

[0010] The front camera 21, right camera 22, left camera 23, and rear camera 24 are provided on the vehicle and capture images of the surroundings of the vehicle. The front camera 21, right camera 22, left camera 23, and rear camera 24 capture images of the surroundings of the vehicle while the vehicle is traveling. The front camera 21 captures images in front of the vehicle, the right camera 22 captures images on the right side of the vehicle, the left camera 23 captures images on the left side of the vehicle, and the rear camera 24 captures images on the rear of the vehicle. The front camera 21 is provided on the grille of the vehicle, the right camera 22 is provided on the right door mirror, the left camera 23 is provided on the left door mirror, and the rear camera 24 is provided on the back door. Note that the various cameras may be provided in other positions on the vehicle. Note that in the following description, the front camera 21, right camera 22, left camera 23, and rear camera 24 will be collectively referred to as camera 20.

[0011] The ATCU (automatic transmission control unit) 30 is a control device that controls the transmission. The ATCU 30 outputs a control signal including shift information to the display control device 100. The shift information indicates the position of the shift lever, such as drive, neutral, reverse, or parking.

[0012] The in-vehicle display 40 is a display included in an in-vehicle infotainment (IVI) system and includes a display 41 and a display 42. The displays 41 and 42 are provided on the vehicle's instrument panel. The displays 41 and 42 are touch-panel displays. FIG. 2 is a plan view showing a portion of the instrument panel. As shown in FIG. 2, the display 41 is provided in the center of the instrument panel. The display 42 is provided on the instrument panel in front of the driver. The display 41 displays, for example, a map for a navigation system, a menu screen (such as a selection screen for various applications) for an entertainment system, content video, etc. The display 42 displays, for example, meters, a range indicator showing the current range of the shift position, an icon showing the status of a seat belt, a vehicle image, an image of the vehicle's surroundings, a predicted course, a driving route, etc. The display 41 and / or the display 42 also display images generated by the display control device 100.

[0013] The displays 41 and 42 have display screens that are long horizontally along the vehicle width direction (the left-right direction on the paper surface of FIG. 2 ). As shown in FIG. 2 , the displays 41 and 42 are arranged side by side on the left and right along the vehicle width direction, with their display screens close to each other. Note that the vehicle does not need to be equipped with both the displays 41 and 42, and may be equipped with only one of the displays. The vehicle speed sensor 50 detects the vehicle speed and outputs the detected value to the controller 10.

[0014] The display control device 100 includes a controller 10 and a recording medium 19. The controller 10 has functions such as acquiring various data, such as camera images and vehicle data, and image processing functions. The controller 10 has, as functional blocks, an image acquisition unit 11, a vehicle data acquisition unit 12, and an image processing unit 13. The controller 10 stores programs for implementing various functions in the recording medium 19, and executes the control programs by causing a processor to execute the processing flow indicated by the programs. Note that the recording medium 19 storing the control programs may be stored as memory within the controller 10. The recording medium 19 may be located inside or outside the controller 10. For example, the recording medium 19 may be stored in a server outside the vehicle, and the display control device 100 may access the server and store the control program data recorded on the recording medium 19 in the memory within the controller 10. The display control device 100 may also execute the control program stored in the server via communication with the server. For example, the display control device 100 transmits camera images and commands for executing each processing step included in the control program to the server. A controller included in the server executes a control program and processes data such as camera images based on instructions from the display control device 100. The server transmits a display image obtained by processing the control program to the display control device 100. The display control device 100 transmits the display image to the displays 41 and 42, causing the image to be displayed.

[0015] The image acquisition unit 11 acquires camera images from the camera 20, etc. The vehicle data acquisition unit 12 acquires vehicle data from sensors mounted on the vehicle, etc., via an in-vehicle communication network. The vehicle data is data indicating the current state of the vehicle, and includes, for example, vehicle speed, steering angle, current vehicle position (current coordinates), and gear shift. The vehicle data acquisition unit 12 acquires vehicle speed from the vehicle speed sensor 50 and acquires steering angle from the steering angle sensor. The vehicle data acquisition unit 12 acquires current position information from a navigation system. The vehicle data acquisition unit 12 also acquires gear shift information from the ATCU 30.

[0016] The image processing unit 13 performs image processing on the camera image to display a current image of the vehicle's surroundings and / or a vehicle image on the displays 41, 42. The image processing unit 13 generates a display image to display the image on the displays 41, 42 in a display mode specified by the user or the system, and outputs the display image to the displays 41, 42. There are multiple display modes, including a front view showing the front of the vehicle, a rear view showing the rear of the vehicle, a top view providing a bird's-eye view of the vehicle, a 3D view viewing the vehicle or its surroundings from a virtual viewpoint, and a skeleton view. The front view is displayed in front of the vehicle's grill and displays an image showing the exterior of the vehicle. The rear view is displayed in rear of the vehicle's back door and displays an image showing the exterior of the vehicle. The top view (around view) displays an overhead image of the vehicle from a virtual viewpoint directly above the vehicle. The top view displays a two-dimensional image, and the vehicle image is displayed as a plan view of the vehicle. If there are tall three-dimensional objects such as walls around the vehicle, an image of the three-dimensional objects appearing to have fallen over is displayed. The 3D view displays an image of an object from a virtual viewpoint, with the surroundings of the object being a virtual viewpoint. The object is not limited to a vehicle, but may also be an obstacle around the vehicle. The virtual viewpoint is a viewpoint position around the object when viewing the object virtually from outside the object. The virtual viewpoint is movable. For example, while a display image of the vehicle viewed from a virtual viewpoint is displayed on the displays 41 and 42, the 3D vehicle image can be rotated by orbiting the virtual viewpoint along a plane around the vehicle. Other display modes available on the displays 41 and 42 include a skeleton view, a front wide view, and a rear wide view. The skeleton view displays a virtual image showing the condition of the vehicle's underfloor in addition to a current camera image (real image) showing the surroundings of the vehicle. The front wide view can display an image of the area in the driver's blind spot in front of the vehicle, for example, when the vehicle is turning left or right at an intersection with poor visibility, such as a T-junction, using an image from the front camera 21 with a wide-angle lens.

[0017] The display mode can be specified by the user or can be changed by the vehicle system as appropriate depending on the driving state of the vehicle. For example, when the range is shifted to reverse to park the vehicle, the display mode may be automatically switched to a top view or a rear view. Alternatively, the display mode may be switched by the driver operating a switch or the like while the vehicle is moving.

[0018] The image processing unit 13 can display images in multiple display modes on multiple screens on the display 41. The image processing unit 13 may display images in different display modes on the displays 41 and 42, or may display images in the same display mode on the displays 41 and 42. The image processing unit 13 may also use both the screens of the displays 41 and 42 as a single display screen to display images in multiple display modes, or may display images in a single display mode. The following description will explain the image processing performed by the image processing unit 13 when displaying an image on the display 41, but when displaying an image on the display 42, the image processing performed on the display 41 may be applied to the image display on the display 42.

[0019] The image processing unit 13 generates a display image according to a display mode based on the multiple camera images acquired by the image acquisition unit 11, and outputs the display image to the display 41. When displaying an image in a front view or rear view, the image processing unit 13 generates a display image showing the front or rear of the vehicle based on the camera image from the front camera 21 or the rear camera 24.

[0020] When displaying an image in top view, the image processing unit 13 generates an overhead image of the vehicle based on multiple camera images captured by the cameras 20. The virtual viewpoint of the overhead image is set at a predetermined position vertically away from the roof of the vehicle and outside the vehicle. For example, the image processing unit 13 performs viewpoint conversion processing on images captured by the front camera 21, right camera 22, left camera 23, and rear camera 24 to generate viewpoint-converted images that show the front, rear, left, and right regions of the vehicle from the virtual viewpoint. That is, the image processing unit 13 converts camera images corresponding to images viewed from the camera positions into viewpoint-converted images viewed from the virtual viewpoint. The viewpoint conversion from the camera positions to the virtual viewpoint is performed by calculation using viewpoint parameters, and the viewpoint parameters are stored in the recording medium 19. The image processing unit 13 fits each image after the viewpoint conversion processing to the surrounding area of ​​the vehicle. The image processing unit 13 performs blending processing on overlapping portions of the camera images from the cameras 20. The blending process is a process for maintaining continuity between two overlapping images, and any method known at the time of filing of this application can be used. The image after the blending process becomes a surrounding image showing the surroundings of the vehicle. The image processing unit 13 also combines the vehicle image located at the center of the screen with the surrounding image. The combined image becomes an overhead image, and the image processing unit 13 outputs the overhead image to the display 41 as a display image.

[0021] When displaying an image in 3D view, the image processing unit 13 generates a display image of the vehicle as viewed from a virtual viewpoint based on multiple camera images captured by the camera 20. First, the image processing unit 13 sets a virtual viewpoint based on a user specification or the driving situation. The position and line of sight of the virtual viewpoint can be arbitrarily set based on a user instruction. The user can specify the position and line of sight of the virtual viewpoint by, for example, touching the display 41. The image processing unit 13 can also set the virtual viewpoint based on the driving situation. For example, the virtual viewpoint can be set behind the vehicle when driving forward to view an image ahead of the vehicle, and in front of the vehicle when driving backward to view an image behind the vehicle. The image processing unit 13 sets the position of the virtual viewpoint when it receives shift information indicating that the gearshift position is in reverse. Furthermore, the image processing unit 13 can also set the virtual viewpoint based on the vehicle speed. For example, the virtual viewpoint can be set based on the vehicle speed input from the vehicle speed sensor 50 so that the virtual viewpoint is farther away from the vehicle when the vehicle speed is equal to or greater than a predetermined value, and closer to the vehicle when the vehicle speed is less than the predetermined value.

[0022] The image processing unit 13 deforms at least a part of a reference curved surface coordinate system having a predefined curved surface for projecting the camera image, according to the position of the virtual viewpoint. The reference curved surface coordinate system is stored in the recording medium 19. The reference curved surface coordinate system has, for example, a bowl shape surrounding the vehicle. The reference curved surface coordinate system has a center on an xy plane parallel to the vehicle's loading surface (travel surface), and is formed from a curved surface having a curvature from the center or near the center and a component in the vehicle's height direction (z direction).

[0023] A base parallel to the xy plane can be formed near the center of the reference curved surface coordinate system, and the base can have any shape, such as a rectangle, a triangle, an ellipse, or a circle. The curvature of the curved surface of the reference curved surface coordinate system does not have to be uniform. For example, the curvature of the curved surface near the base may be relatively large, and the curvature of the curved surface away from the base may be relatively small. The image processing unit 13 may store in advance multiple reference curved surface coordinate systems with different shapes depending on the shape (size, form) of the vehicle. When multiple reference curved surface coordinate systems are stored, the reference curved surface coordinate systems to be used may be stored in the recording medium 19 in association with the vehicle speed, the position of the vehicle shift position, etc., and the image processing unit 13 may select a reference curved surface coordinate system depending on the vehicle speed, the position of the vehicle shift position, etc.

[0024] The image processing unit 13 may read a pre-stored reference curved surface coordinate system and deform the shape of the read reference curved surface coordinate system. The image processing unit 13 may deform the shape of the reference curved surface coordinate system according to the position of the virtual viewpoint. For example, if the virtual viewpoint is located outside the outer edge of the reference curved surface coordinate system, a portion of the reference curved surface coordinate system may be deformed so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system. By deforming a portion of the reference curved surface coordinate system so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system, distortion of the shape of an obstacle with height can be suppressed when the obstacle is displayed on the screen, and an image without any missing image can be displayed on the display 41. Note that the method of deforming the reference curved surface coordinate system may use other coordinate system deformation methods known at the time of filing.

[0025] The image processing unit 13 projects multiple camera images acquired from the camera 20 onto the reference curved coordinate system. When the reference curved coordinate system is deformed, the image processing unit 13 simply projects the camera images onto the curved coordinate system after the deformation (deformable coordinate system). In order to project the camera image data onto the reference curved coordinate system or the deformed coordinate system after the deformation, the image processing unit 13 has an image conversion table that associates the coordinates of pixels included in the camera images with the coordinates of the coordinate system. The image conversion table is recorded on the recording medium 19.

[0026] A prepared vehicle image of a vehicle may also be superimposed on the reference curved coordinate system. The vehicle image may be created in advance based on the design of the vehicle and stored in the recording medium 19. Superimposing the vehicle image on the reference curved coordinate system allows the vehicle itself, as well as an image of its surroundings, to be displayed in three dimensions, helping to understand the relationship between the vehicle's position and orientation and the surrounding image. In this way, the image processing unit 13 refers to the image conversion table to associate the coordinates of pixels contained in the camera image with the coordinates of the coordinate system, projects the camera image onto the reference curved coordinate system or the deformed coordinate system, and superimposes the vehicle image on the projected three-dimensional image to generate a display image (three-dimensional image) in a 3D view in which the vehicle and its surroundings are viewed from a virtual viewpoint.

[0027] The image processing unit 13 may generate display images viewed from selectable virtual viewpoints, not limited to those specified by the user or set based on the driving situation. For example, when a display image is displayed on the display 41 in a 3D view, the user can freely move the position of the virtual viewpoint by touching the display 41. When the 3D view display is selected, the image processing unit 13 generates multiple 3D view display images for each of multiple virtual viewpoints selectable by the user. Then, for example, when the user moves the virtual viewpoint by touching the display 41, a 3D image in which the vehicle image rotates around the center point of a curved coordinate system is displayed on the display 41. Note that the image processing unit 13 may set virtual viewpoints at any positions around the vehicle, not limited to those selectable by the user, and generate multiple 3D view display images for each virtual viewpoint. In other words, when a 3D image is displayed on the display 41 in a rotating display, the image processing unit 13 generates a continuous 3D image so that the image can be displayed smoothly in response to the movement of the virtual viewpoint.

[0028] The image processing method for the display images displayed in the front / rear view, top view, and 3D view display modes is not limited to the above, and other image processing methods can be used. Furthermore, the image processing method for the display images displayed in other display modes, such as skeleton view and front / rear wide view, can be any method known at the time of filing.

[0029] Next, display control by the display control device 100 will be described along with specific examples of the display screen of the display 41. FIG. 3 shows the display screen of the display 41. In the example of FIG. 3, the vehicle is stopped on a straight road, and the left screen of the display 41 displays a top-view image, while the right screen of the display 41 displays a 3D-view image. When displaying a top-view overhead image and a 3D-view display image on the display 41, the controller 10 generates a top-view image 46 by superimposing viewpoint icons 43a-43h and an automatic display start icon 44 on the overhead image of the vehicle. The top-view image 46 includes an overhead image of the vehicle viewed from above, viewpoint icons 43a-43h, and the automatic display start icon 44. The viewpoint icons 43a-43h indicate the position of a virtual viewpoint and / or the direction of the line of sight and are user-selectable icons. The viewpoint icons 43a-43h are arranged around the vehicle, with the vehicle image at the center. Viewpoint icon 43a is positioned above the vehicle image located at the center of the overhead image, while viewpoint icons 43c, 43e, and 43g are positioned to the right, bottom, and left of the vehicle image, respectively. Viewpoint icon 43b, viewpoint icon 43d, viewpoint icon 43f, and viewpoint icon 43h are positioned in the upper right, lower right, lower left, and lower right directions, respectively, of the vehicle image. When the user selects one of viewpoint icons 43a to 43h, a display image (three-dimensional image) of the vehicle as viewed from the selected viewpoint icon is displayed on the right side of display 41.

[0030] As shown in FIG. 3 , when the user touches the viewpoint icon 43a, the controller 10 displays the viewpoint icon 43a as a selection icon. The controller 10 displays the selection icon by, for example, changing the color of the viewpoint icon 43a. In the example of FIG. 3 , the selection icon is represented by a thick line surrounding the viewpoint icon 43a. The image processing unit 13 sets a virtual viewpoint corresponding to the viewpoint icon 43a. This allows the virtual viewpoint to be set according to the user's designation. In the example of FIG. 3 , the position from which the vehicle is viewed from the front is set as the virtual viewpoint. Then, the image processing unit 13 generates a three-dimensional image of the vehicle and its surroundings viewed from the front of the vehicle based on multiple camera images. The image processing unit 13 outputs the three-dimensional image to the display 41 as a display image. A three-dimensional image (3D view display image) 45 is displayed on the right screen of the display 41. Then, as shown in FIG. 3 , the display 41 displays a top-view image 46 and the three-dimensional image 45 side by side.

[0031] The automatic display start icon 44 is an icon for starting automatic display, which automatically switches three-dimensional images in the 3D view by moving the virtual viewpoint. The automatic display start icon 44 is displayed at the center position of the vehicle image included in the top-view image 46. When automatic display is executed, predetermined positions around the vehicle become the start and end points of the virtual viewpoint, and the virtual viewpoint moves from the start point to the end point, causing the vehicle image displayed in the 3D view to rotate 360 ​​degrees along a plane parallel to the road surface. In other words, in automatic display, the center point is placed on the vehicle image, and the vehicle image and the vehicle-surrounding image rotate 360 ​​degrees around the center point. For example, in the example of FIG. 3 , when the user touches the automatic display start icon 44, the selected viewpoint icon 43 a becomes the start and end points of the virtual viewpoint. The virtual viewpoint moves from the position corresponding to the viewpoint icon 43 a to the viewpoint icon 43 b in order to the viewpoint icon 43 h, rotating clockwise around the vehicle image, and then returns to the position corresponding to the original viewpoint icon 43 a. As the virtual viewpoint rotates, the three-dimensional image 45 in the 3D view rotates. The orientation of the vehicle image included in the three-dimensional image 45 starts from the front (front) as shown in FIG. 3 , changes in the order of the right side, rear (back), and left side, and then returns to the original front (front). In other words, the three-dimensional image is displayed as if it were rotated 360 degrees (hereinafter also referred to as "360-degree rotation display"). As a result, the display image (three-dimensional image 45) is automatically rotated 360 degrees in the 3D view.

[0032] FIG. 4 shows a display screen of the display 41. In the example of FIG. 4, the vehicle is stopped at a T-junction, and the left screen of the display 41 displays a top-view image, while the right screen of the display 41 displays a 3D-view image. The image processing method for the top-view image 46 and the 3D image 45 is the same as in the example of FIG. 3, and the top-view image 46 also includes viewpoint icons 43a-h and an automatic display start icon 44. Unlike the example of FIG. 3, viewpoint icon 43e is selected, and the 3D view displays a 3D image 45 of the vehicle and its surroundings as seen from behind the vehicle. Note that, as shown in FIG. 4, when the vehicle is stopped at a T-junction, the controller 10 may set a virtual viewpoint corresponding to viewpoint icon 43a and display a 3D image 45 as seen from the virtual viewpoint on the display 41. This allows the state of the blind spot to be displayed on the display 41 if there is a blind spot ahead of the vehicle.

[0033] 3 and 4, while the vehicle is stopped, the user can select one of the viewpoint icons 43a to 43h and view a three-dimensional image from the direction of the selected viewpoint icon 43a to 43h in a 3D view on the display 41. For example, when the user touches the position of the viewpoint icon 43c on the screen of the display 41, the image processing unit 13 switches the viewpoint icon 43c to a selected icon and moves the current virtual viewpoint (in the example of FIG. 3, the virtual viewpoint corresponding to the viewpoint icon 43a) to the virtual viewpoint corresponding to the viewpoint icon 43c. The image processing unit 13 then generates a three-dimensional image viewed from the virtual viewpoint based on the camera image and outputs the display image to the display 41 so that the three-dimensional image is displayed in a 3D view. A three-dimensional image of the vehicle viewed from the right side is displayed on the display 41.

[0034] Furthermore, while the vehicle is stopped, the controller 10 may perform display control so that the user can not only select a virtual viewpoint by touching the viewpoint icons 43a to 43h but also arbitrarily specify the position of the virtual viewpoint. The user touches any position on the 3D view display screen. Then, while the user's finger is touching the screen, the user moves the finger in any direction (swipe operation). While the finger is touching the screen, the position of the virtual viewpoint moves in accordance with the swipe operation, and the 3D image rotates in accordance with the movement of the virtual viewpoint. The amount of movement of the virtual viewpoint is determined by the swipe amount (corresponding to the amount the finger moves while touching the screen), and the greater the swipe amount, the greater the movement amount. Note that the amount of movement of the virtual viewpoint corresponds to the length of the arc of a circle centered on the vehicle image. For example, the user touches the 3D view display screen with their finger and moves their finger with a large swipe amount. If the swipe amount is large, the image processing unit 13 moves the virtual viewpoint around the center point of the vehicle image, thereby rotating the 3D image. If the swipe amount is small, image processing unit 13 rotates the three-dimensional image so that the virtual viewpoint moves along an arc centered on the center point of the vehicle image by an amount equivalent to the swipe amount. Note that the rotational display caused by a swipe operation is not limited to the 3D view display screen, but may also be performed by a swipe operation on a top view display screen. For example, image processing unit 13 causes display 41 to display an image in which viewpoint icons 43a to 43h have been removed from a top view image 46 as shown in FIG. 3 or 4 . Then, the user performs a swipe operation at any position on the screen displaying top view image 46. Image processing unit 13 rotates the three-dimensional image on the 3D view display screen in response to the swipe operation on the top view display screen.

[0035] In this embodiment, when the vehicle is stopped or parked, no restrictions are placed on the movement of the virtual viewpoint due to a swipe operation, and the virtual viewpoint can move freely according to the amount of swiping. On the other hand, when the vehicle is moving, restrictions are placed on the movement of the virtual viewpoint due to a swipe operation (hereinafter also referred to as "swipe restriction"). Display control when restrictions are placed on the movement of the virtual viewpoint will be described below.

[0036] The image processing unit 13 determines whether the vehicle is traveling based on the vehicle data acquired by the vehicle data acquisition unit 12. If the vehicle speed included in the vehicle data is greater than a parking / stopping threshold, the image processing unit 13 determines that the vehicle is traveling. The parking / stopping threshold is a preset vehicle speed threshold for determining whether the vehicle is parked or stopped (e.g., 0 km / h or several km / h). The image processing unit 13 may also determine whether the vehicle is traveling based on shift information included in the vehicle data. For example, if the shift information included in the vehicle data indicates that the current shift position (range) is in drive, the image processing unit 13 may determine that the vehicle is traveling.

[0037] When the vehicle is traveling, the image processing unit 13 restricts movement of the virtual viewpoint due to a user operation. The image processing unit 13 switches between a movement restriction mode, in which movement of the virtual viewpoint is restricted, and a free movement mode, in which the virtual viewpoint is freely moved. The free movement mode is a mode in which no restriction is imposed on movement of the virtual viewpoint. In the free movement mode, the virtual viewpoint can be freely moved according to the amount of swiping, and in the 3D view, a three-dimensional image can be displayed to rotate in accordance with the movement of the virtual viewpoint. On the other hand, in the movement restriction mode, the movement of the virtual viewpoint is limited to one unit per operation, regardless of the amount of swipe of the swipe operation. One unit in the movement of the virtual viewpoint is the smallest unit of movement of the virtual viewpoint. When the vehicle is traveling, the image processing unit 13 selects the movement restriction mode. When the vehicle is stopped or parked, the image processing unit 13 selects the free movement mode.

[0038] Referring to FIG. 5 , the rotational display of the three-dimensional image 45 when a swipe restriction is applied will be described. FIG. 5 is a diagram for explaining the rotational display of the three-dimensional image 45 in a 3D view. FIG. 5( a) shows the three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43 e, FIG. 5( b) shows the three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43 f, and FIG. 5( c) shows the three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43 g. Note that FIG. 5 only shows the 3D view display screen of the display 41. The user touches the display 41 to perform a swipe operation. The three-dimensional image 45 rotates in the direction specified by the swipe operation. The movement amount of the virtual viewpoint is independent of the swipe amount, and moves one unit for each swipe operation. For example, on a display screen such as that shown in FIG. 5( a), the user performs a swipe operation to rotate the vehicle image counterclockwise. The image processing unit 13 identifies the operation direction (swipe direction) from the user's swipe operation and moves the virtual viewpoint by one unit. With this one-unit movement of the virtual viewpoint, the three-dimensional image 45 rotates 45 degrees counterclockwise. In the example of FIG. 5 , one unit of movement of the virtual viewpoint corresponds to a rotation angle (45 degrees) of the three-dimensional image 45 and corresponds to one-percentage movement of the viewpoint icons 43a to 43f. Note that one viewpoint represents movement from the current virtual viewpoint to the adjacent virtual viewpoint. The three-dimensional image shown in FIG. 5(a) rotates with one swipe operation, becoming the three-dimensional image shown in FIG. 5(b). Then, when the user performs a similar swipe operation on the display screen shown in FIG. 5(b), the image processing unit 13 rotates the three-dimensional image 45 by 45 degrees counterclockwise. The display screen of the display 41 transitions from the display screen shown in FIG. 5(b) to the display screen shown in FIG. 5(c). That is, in the movement restriction mode, with one swipe operation, the virtual viewpoint moves by only one unit in the operation direction of the swipe operation.

[0039] When the vehicle speed falls below the parking threshold and the vehicle is stopped or parked, the image processing unit 13 cancels the movement restriction mode and selects the free movement mode. In the free movement mode, the image processing unit 13 identifies the operation direction (swipe direction) from the user's swipe operation and freely moves the virtual viewpoint according to the amount of the swipe. That is, in the free movement mode, while a finger is touching the display screen of the display 41, the virtual viewpoint moves in accordance with the movement of the finger's contact point. The three-dimensional image then rotates in accordance with the free movement of the virtual viewpoint.

[0040] When the vehicle is traveling and the vehicle speed is equal to or greater than the viewpoint restriction threshold, the virtual viewpoints that can be selected by the user are restricted according to the direction of travel of the vehicle. Below, we will explain display control when restricting the selectable virtual viewpoints (hereinafter also referred to as "virtual viewpoint selection restriction").

[0041] The image processing unit 13 determines whether the current vehicle speed is equal to or greater than the viewpoint restriction threshold based on the vehicle speed acquired by the vehicle data acquisition unit 12. The viewpoint restriction threshold is a preset vehicle speed threshold (e.g., 10 to 20 km / h) and is used to determine whether to impose a virtual viewpoint selection restriction.

[0042] When the current vehicle speed is equal to or greater than the viewpoint restriction threshold, the image processing unit 13 sets selectable and non-selectable virtual viewpoints from among multiple virtual viewpoints located around the vehicle, depending on the vehicle's traveling direction. The multiple virtual viewpoints located around the vehicle are viewpoint positions when viewing the vehicle virtually from outside the vehicle. The multiple virtual viewpoints correspond to viewpoint icons 43a to 43h. For example, when the vehicle speed is equal to or greater than the viewpoint restriction threshold and the vehicle is moving forward, the image processing unit 13 limits the selectable virtual viewpoints to viewpoints from behind the vehicle. On the other hand, when the vehicle speed is equal to or greater than the viewpoint restriction threshold and the vehicle is moving backward, the image processing unit 13 limits the selectable virtual viewpoints to viewpoints from in front of the vehicle. In other words, when the vehicle speed is equal to or greater than the viewpoint restriction threshold, a virtual viewpoint selection restriction is imposed so that the vehicle image included in the three-dimensional image faces the traveling direction.

[0043] For example, when a three-dimensional image 45 such as that shown in FIG. 3 is displayed in a 3D view while the vehicle is moving forward, the vehicle image is oriented so as to face the driver, and the state of the moving forward vehicle and the display in the 3D view are not consistent. Furthermore, the direction of travel is difficult to see in the 3D view display as shown in FIG. 3. In this embodiment, when the vehicle is moving forward, only three virtual viewpoints from the rear can be selected. Therefore, when a three-dimensional image 45 viewed from limited virtual viewpoints is displayed in the 3D view, the vehicle image is oriented so as to represent the vehicle moving forward, making it easier to see the direction of travel. Furthermore, when the vehicle is moving backward, only three virtual viewpoints from the front can be selected.

[0044] Next, the display of the top view when virtual viewpoint selection restriction is applied will be described. In the top view, viewpoint icons 43a to 43h and the automatic display start icon 44 are superimposed on an overhead image of the vehicle. As shown in FIGS. 3 and 4 , when the vehicle is stopped, the viewpoint icons 43a to 43h and the automatic display start icon 44 are selectable. On the other hand, when the vehicle speed is equal to or greater than the viewpoint restriction threshold and the vehicle is moving forward, the image processing unit 13 makes the viewpoint icons 43a to 43c, 43g, and 43h and the automatic display start icon 44 unselectable. To indicate that the icons are unselectable, the image processing unit 13 may, for example, gray out or use gray hatching on the icons, or may erase the viewpoint icons 43a to 43c, 43g, and 43h. The image processing unit 13 also makes the viewpoint icons 43d to 43f selectable.

[0045] Furthermore, when the vehicle speed is equal to or greater than the viewpoint restriction threshold and the vehicle is reversing, the image processing unit 13 makes the viewpoint icons 43c to 43g and the automatic display start icon 44 unselectable, and makes the viewpoint icons 43a, 43b, and 43h selectable.

[0046] Next, a display control method by the controller 10 will be described with reference to Fig. 6. Fig. 6 is a flowchart of the display control method by the controller 10. The controller 10 repeatedly executes the following control flow while the display image on the left side of the display 41 is displayed in top view and the display image on the right side is displayed in 3D view.

[0047] In step S1, the image acquisition unit 11 acquires a camera image from the camera 20. The vehicle data acquisition unit 12 acquires vehicle data from the ATCU 30 and / or the vehicle speed sensor 50. In step S2, the image processing unit 13 detects the current vehicle state based on the vehicle data. In this flow, the vehicle state is detected from the vehicle speed. In step S3, the image processing unit 13 determines whether the current vehicle speed is equal to or lower than the parking / stopping threshold, whether the current vehicle speed is higher than the parking / stopping threshold but lower than the viewpoint restriction threshold, or whether the current vehicle speed is equal to or higher than the viewpoint restriction threshold.

[0048] If the current vehicle speed is equal to or less than the parking / stopping threshold, the vehicle state is determined to be parked, and the image processing unit 13 does not impose any restrictions on movement of the virtual viewpoint due to swipe restrictions or restrictions on selection of the virtual viewpoint (step S4). In step S5, the image processing unit 13 generates a top-view image of the vehicle as seen from above based on multiple camera images. The top-view image is an image in which viewpoint icons 43a to 43h and an automatic display start icon 44 are superimposed on an overhead image of the vehicle and its surroundings. Note that the display of the viewpoint icons 43a to 43h and the automatic display start icon 44 is selectable. In step S6, the image processing unit 13 generates a three-dimensional image of the vehicle as seen from a virtual viewpoint based on multiple camera images. Note that the virtual viewpoint is specified by the user touching an icon, etc., while the control flow of FIG. 6 is repeatedly executed.

[0049] Then, in step S7, image processing unit 13 outputs an image including a top-view image and a three-dimensional image to display 41 as a display image. The display screen displayed on display 41 in accordance with the control flow of step S7 is shown in FIG. 7 . As shown in FIG. 7 , a top-view image is displayed on the left screen of display 41, and viewpoint icons 43a to 43h and automatic display start icon 44 are selectable. In the example of FIG. 7 , viewpoint icon 43h has been selected by the user, and a three-dimensional image 45 (a virtual image viewed from the left front of the vehicle) viewed from the virtual viewpoint corresponding to viewpoint icon 43h is displayed on the right screen of display 41. Note that if, for example, viewpoint icon 43g is selected by the user while display 41 is displaying the screen shown in FIG. 7 , image processing unit 13 moves the position of the virtual viewpoint to the position corresponding to viewpoint icon 43g and outputs the three-dimensional image viewed from the moved virtual viewpoint to display 41.

[0050] Returning to step S3, if the current vehicle speed is higher than the parking / stopping threshold but lower than the viewpoint restriction threshold, the controller 10 determines that the vehicle is in a traveling state and executes the control flow of steps S8 to S11. In step S8, the image processing unit 13 executes a swipe restriction to restrict movement of the virtual viewpoint by a user operation. No virtual viewpoint selection restriction is imposed. In step S9, the image processing unit 13 generates a top-view image based on multiple camera images. The viewpoint icons 43a to 43h are set to a selectable state, and the automatic display start icon 44 is set to an unselectable state. Note that no virtual viewpoint selection restriction is imposed, but the automatic display start icon 44 is set to an unselectable state. In step S10, the image processing unit 13 generates a three-dimensional image of the vehicle as viewed from a virtual viewpoint based on multiple camera images. Note that the virtual viewpoint is specified by the user touching an icon, etc., while the control flow of FIG. 6 is repeatedly executed.

[0051] Then, in step S11, the image processing unit 13 outputs an image including a top-view image and a three-dimensional image to the display 41 as a display image. The display screen displayed on the display 41 according to the control flow of step S11 is shown in FIG. 8 . As shown in FIG. 8 , a top-view image is displayed on the left screen of the display 41, the viewpoint icons 43a to 43h are selectable, and the automatic display start icon 44 is not selectable. In the example of FIG. 7 , the user selects viewpoint icon 43h, and the right screen of the display 41 displays a three-dimensional image viewed from the virtual viewpoint corresponding to viewpoint icon 43h (a virtual image viewed from the left front of the vehicle). As shown in the example of FIG. 8 , when swipe restriction is implemented, automatic display, in which the three-dimensional image is automatically rotated 360 degrees, may be prohibited. The vehicle image included in the three-dimensional image 45 is an image viewed from a virtual viewpoint specified by the user, and the orientation of the vehicle image does not need to correspond to the vehicle's traveling direction.

[0052] Returning to step S3, if the current vehicle speed is equal to or greater than the viewpoint restriction threshold, the controller 10 determines that the vehicle is moving (moving at a speed higher than a slow-moving speed) and executes the control flow of steps S12 to S15. In step S12, the image processing unit 13 restricts swiping and restricts virtual viewpoint selection. In step S13, the image processing unit 13 generates a top-view image based on multiple camera images. Of the viewpoint icons 43a to 43h, one icon 43a to 43h is selectable depending on the vehicle's direction of travel, while the other icons 43a to 43h are unselectable. For example, when the vehicle is moving forward, the viewpoint icons 43d to 43f are selectable, while the other icons 43a to 43c, 43g, and 43h are unselectable. Furthermore, the automatic display start icon 44 is unselectable. In step S14, the image processing unit 13 generates a three-dimensional image of the vehicle as viewed from a virtual viewpoint based on multiple camera images.

[0053] Then, in step S14, image processing unit 13 outputs an image including a top-view image and a three-dimensional image to display 41 as a display image. The display screen displayed on display 41 by the control flow of step S14 is shown in FIG. 9 . As shown in FIG. 9 , a top-view image is displayed on the left screen of display 41, viewpoint icons 43d to 43f are selectable, and viewpoint icons 43a to 43c, 44g, and 44h and automatic display start icon 44 are not selectable. The orientation of the vehicle image included in three-dimensional image 45 corresponds to the traveling direction of the vehicle. In other words, the virtual viewpoint is set to one of three viewpoints located behind the vehicle (virtual viewpoints corresponding to viewpoint icons 43d to 43f). In the example of FIG. 9 , the virtual viewpoint of three-dimensional image 45 is set to a position behind the vehicle and in the center of the vehicle.

[0054] After the control flow of steps S7, S11, and S15 is executed, the control flow ends. The controller 10 restarts the control flow from step S1.

[0055] Note that, while repeatedly executing the control flow of FIG. 6 , if the current vehicle speed exceeds the viewpoint restriction threshold and a new restriction on selectable virtual viewpoints is imposed, the image processing unit 13 may automatically switch the virtual viewpoint to face the vehicle's traveling direction when executing the control flow of step S14. For example, when the vehicle is moving forward, the image processing unit 13 sets the virtual viewpoint to a central position behind the vehicle. When the vehicle transitions from a slow-moving state to a state where the vehicle speed is higher than the slow-moving state, the image processing unit 13 automatically switches the virtual viewpoint so that the viewpoint reflects the vehicle's traveling direction in the 3D view. For example, when the vehicle is moving forward slowly and the vehicle speed is higher than the parking / stopping threshold but less than the viewpoint restriction threshold, the display 41 displays a screen such as that shown in FIG. 8 . If the vehicle speed increases from the state of FIG. 8 and the current vehicle speed exceeds the viewpoint restriction threshold, the image processing unit 13 moves the virtual viewpoint to a position corresponding to the viewpoint icon 43 e, generates a three-dimensional image so that the vehicle image in the 3D view faces the vehicle's traveling direction, and outputs the three-dimensional image to the display 41. The display screen of the display 41 automatically switches from the display screen shown in FIG. 8 to the display screen shown in FIG.

[0056] Furthermore, when the vehicle decelerates from a state in which the vehicle is moving forward and the vehicle speed is equal to or greater than the viewpoint restriction threshold to a state in which the current vehicle speed falls below the viewpoint restriction threshold, the image processing unit 13 makes the viewpoint icons 43a to 43h, which were previously unselectable, selectable. For example, when the display 41 is displaying a screen such as that shown in Fig. 9 and the vehicle speed falls below the viewpoint restriction threshold, the image processing unit 13 executes the control flow of steps S8 to S11 to make the viewpoint icons 43a to 43h selectable so that a top-view image 46 such as that shown in Fig. 10 is displayed on the display 41. In the top view of Fig. 9, the viewpoint icons 43a to 43c, 43g, and 43h were unselectable, but in the top view of Fig. 10, the viewpoint icons 43a to 43c, 43g, and 43h are switched to a selectable state.

[0057] While the control flow of each step has been described above with reference to FIGS. 7 to 10 using a specific example in which the vehicle is traveling forward, the control flow of steps S1 to S15 can also be applied to display control of the display 41 while the vehicle is reversing. For example, if the current vehicle speed exceeds the viewpoint restriction threshold while the controller 10 is repeatedly executing the control flow of FIG. 6 while the vehicle is reversing, the image processing unit 13 generates a top-view image 46 and a three-dimensional image 45 so that a display image such as that shown in FIG. 11 is displayed on the display 41. Of the multiple viewpoint icons 43a to 43h superimposed on the top-view image 46, one of the viewpoint icons 43a, 43b, and 43h that allows selection of a virtual viewpoint in front of the vehicle is selectable, while the other viewpoint icons 43c to 43g are unselectable. Furthermore, the orientation of the vehicle image included in the three-dimensional image 45 corresponds to the vehicle's traveling direction. When the vehicle is reversing, the virtual viewpoint is set to one of three viewpoints located behind the vehicle (the virtual viewpoints corresponding to the viewpoint icons 43a, 43b, and 43h). In the example of FIG. 11, the virtual viewpoint of the three-dimensional image 45 is set at a position in front of the vehicle and at the center of the vehicle.

[0058] Furthermore, when the vehicle transitions from a slow-moving state to a state where the vehicle speed is higher than the slow-moving state while the vehicle is reversing, the image processing unit 13 may automatically switch the virtual viewpoint to a viewpoint that shows the vehicle's traveling direction in a 3D view. For example, when the vehicle is reversing slowly and the vehicle speed is higher than the parking threshold and lower than the viewpoint restriction threshold, the image processing unit 13 makes the viewpoint icons 43a to 43h selectable to generate a top-view image and generate a 3D image as seen from the virtual viewpoint specified by the user. When the vehicle speed exceeds the viewpoint restriction threshold while the vehicle is reversing, the image processing unit 13 moves the virtual viewpoint to a position corresponding to the viewpoint icon 43a, generates a 3D image so that the vehicle image in the 3D view is oriented in the vehicle's traveling direction, and outputs the 3D image to the display 41. This automatically switches the orientation of the vehicle image included in the 3D image.

[0059] In this embodiment, the controller 10 executes different control flows depending on whether the vehicle speed is equal to or less than the parking threshold, whether the vehicle speed is higher than the parking threshold but lower than the viewpoint restriction threshold, or whether the vehicle speed is equal to or greater than the viewpoint restriction threshold. That is, the control content of the display control by the controller 10 differs depending on the vehicle speed condition. Below, we will explain the control content when the vehicle state transitions and the vehicle speed satisfies a different vehicle speed condition. In the following explanation, the state of the vehicle when the vehicle speed is equal to or less than the parking threshold is referred to as a "parking state," the state of the vehicle when the vehicle speed is higher than the parking threshold but lower than the viewpoint restriction threshold is referred to as a "slow-moving state," and the state of the vehicle when the vehicle speed is equal to or greater than the viewpoint restriction threshold is referred to as a "normal driving state."

[0060] When the vehicle transitions from the "slow-travel state" to the "normal traveling state" while traveling forward, the controller 10 performs the following display control. When the vehicle is traveling in the "slow-travel state" and the virtual viewpoint is set to one of the three viewpoints from behind the vehicle (virtual viewpoints corresponding to viewpoint icons 43d to 43f) and the vehicle speed exceeds the viewpoint restriction threshold, the image processing unit 13 maintains the position of the virtual viewpoint. On the other hand, when the virtual viewpoint is set to a virtual viewpoint other than the three viewpoints from behind the vehicle (virtual viewpoints corresponding to viewpoint icons 43d to 43f) during the "slow-travel state" and the vehicle speed exceeds the viewpoint restriction threshold, the image processing unit 13 automatically switches the position of the virtual viewpoint to one of the three viewpoints from behind the vehicle. Note that when the vehicle transitions from the "slow-travel state" to the "normal traveling state" while traveling backward, the "three viewpoints from behind the vehicle" may be replaced with "three viewpoints from in front of the vehicle," and similar control content may be performed. In other words, the image processing unit 13 automatically switches the virtual viewpoint to a viewpoint that shows the vehicle's traveling direction in a 3D view.

[0061] In the "normal driving state," when the virtual viewpoint is set to one of three viewpoints from behind the vehicle (virtual viewpoints corresponding to viewpoint icons 43d to 43f) and the vehicle speed falls below the viewpoint restriction threshold, the image processing unit 13 maintains the position of the virtual viewpoint. In addition, the image processing unit 13 switches viewpoint icons 43a to 43h that were previously "unselectable" to "selectable."

[0062] When the vehicle is in a "parked state" and the vehicle speed exceeds the parking threshold while the 3D image is displayed in 360-degree rotation, the controller 10 maintains the 360-degree rotation until the user inputs a touch operation. However, if the vehicle speed exceeds the viewpoint restriction threshold while the 360-degree rotation is being displayed, the controller 10 stops the 360-degree rotation and automatically switches the virtual viewpoint to a viewpoint that shows the vehicle's traveling direction in a 3D view. When the vehicle is moving forward, the virtual viewpoint is switched to one of three virtual viewpoints from behind the vehicle or a central position behind the vehicle. When the vehicle is reversing, the virtual viewpoint is switched to one of three virtual viewpoints from in front of the vehicle or a central position in front of the vehicle.

[0063] When the vehicle is in a "parked / stopped state" and the vehicle speed becomes higher than the parking / stopping threshold while the virtual viewpoint is being moved by a swipe operation by the user, the controller 10 imposes a swipe restriction.

[0064] In addition, if the vehicle is in a "parked state" and the three-dimensional image is displayed in a 360-degree rotational display, and the shift position is moved to a position other than "parking," the controller 10 may stop the 360-degree rotational display and automatically switch the virtual viewpoint to a viewpoint that shows the direction of travel of the vehicle in a 3D view.

[0065] As described above, in this embodiment, the controller 10 acquires camera images from the camera 20, generates a display image of the vehicle as seen from a virtual viewpoint based on the multiple camera images, and outputs the display image to the display 41. The controller 10 also moves the position of the virtual viewpoint in response to a user operation, detects the vehicle speed, and limits the movement of the virtual viewpoint in response to the detected vehicle speed. This enables the display 41 to display images in accordance with the vehicle state, and provides the user with an appropriate display image that matches the vehicle state. As a result, display control of the display 41 with high operability can be achieved.

[0066] In a display control system that displays a display image of a vehicle viewed from a virtual viewpoint on the display 41, the user can freely move the virtual viewpoint, allowing the user to check the vehicle and its surroundings through various display images. However, if the user is able to freely manipulate the position of the virtual viewpoint while the vehicle is traveling at high speed, for example, the user may concentrate on the screen or touch operations of the display 41 rather than on driving operations. In this embodiment, the movement of the virtual viewpoint by user operations is limited in accordance with the detected vehicle speed, thereby preventing the user from concentrating more on operating the display 41 than on driving operations.

[0067] In this embodiment, the controller 10 selects at least one virtual viewpoint from among a plurality of virtual viewpoints positioned around the vehicle, generates a display image so that a three-dimensional image of the vehicle as seen from the selected virtual viewpoint is displayed on the display 41, and limits the virtual viewpoints selectable for displaying the three-dimensional image in accordance with the detected vehicle speed. This makes it possible to exclude virtual viewpoints that are not suitable for the vehicle's driving state from the selection candidates. Therefore, for example, when the vehicle is driving at high speed, virtual viewpoints that make it difficult to see the traveling direction can be excluded from the selection candidates, thereby preventing the display of an image that makes it difficult to see the traveling direction while driving. As a result, highly operable display control of the display 41 can be achieved.

[0068] In this embodiment, the controller 10 limits the selectable virtual viewpoints to a viewpoint from behind the vehicle when the vehicle is moving forward, and limits the selectable virtual viewpoints to a viewpoint from in front of the vehicle when the vehicle is moving backward. This makes it possible to exclude virtual viewpoints that make it difficult to see the direction of travel from the selection options, thereby preventing images that make it difficult to see the direction of travel while driving from being displayed. As a result, display control of the display 41 with high operability can be achieved.

[0069] In this embodiment, the controller 10 selects a movement restriction mode that restricts the movement of the virtual viewpoint in accordance with the detected vehicle speed. When the movement restriction mode is selected, the movement of the virtual viewpoint is restricted to one unit per swipe operation, regardless of the amount of swipe. When the movement restriction mode is released, the virtual viewpoint moves freely in accordance with the amount of swipe. This allows for easy display control of the display 41.

[0070] In this embodiment, when the vehicle transitions from a slow-moving state to a state where the vehicle speed is higher than the slow-moving state while the vehicle is moving forward or backward, the image processing unit 13 automatically switches the virtual viewpoint to a viewpoint that displays the vehicle's traveling direction in a 3D view, but when switching the virtual viewpoint, the height of the virtual viewpoint may be fixed before and after the virtual viewpoint switching. This fixes the height of the three-dimensional image when the virtual viewpoint is switched, preventing the image from shifting in the vertical direction when the image is switched.

[0071] As a modification of this embodiment, when the vehicle transitions from a stopped or parked state (parked state) to a traveling state, the image processing unit 13 may automatically switch the virtual viewpoint so that the three-dimensional image in the 3D view reflects the vehicle's traveling direction. In other words, the image processing unit 13 may automatically switch the virtual viewpoint to a viewpoint that reflects the vehicle's traveling direction in the 3D view.

[0072] In this embodiment, the controller 10 does not need to execute all of the control flows shown in FIG. 6, and does not need to execute each control flow in the order shown in FIG.

[0073] REFERENCE SIGNS LIST 1 display control system 10 controller 11 image acquisition unit 12 vehicle data acquisition unit 13 image processing unit 19 recording medium 20 camera 40 in-vehicle display 43a to 43h viewpoint icon 44 automatic display start icon 45 three-dimensional image 46 top-view image 50 vehicle speed sensor 100 display control device

Claims

1. A display control device comprising a controller that acquires camera images from multiple onboard cameras that capture images of the surroundings of a vehicle, generates a display image of the vehicle as seen from a virtual viewpoint based on the multiple camera images, and outputs the display image to a display, wherein the controller moves the position of the virtual viewpoint through user operation, detects the vehicle speed, and limits the movement of the virtual viewpoint through user operation in accordance with the detected vehicle speed.

2. A display control device according to claim 1, wherein the controller selects at least one of the virtual viewpoints located around the vehicle, generates the display image so that a three-dimensional image of the vehicle as seen from the selected virtual viewpoint is displayed on the display, and limits the virtual viewpoints that can be selected for displaying the three-dimensional image according to the detected vehicle speed.

3. A display control device according to claim 1 or 2, wherein the controller limits the selectable virtual viewpoints to a viewpoint from behind the vehicle when the vehicle is moving forward, and limits the selectable virtual viewpoints to a viewpoint from in front of the vehicle when the vehicle is moving backward.

4. A display control device according to any one of claims 1 to 3, wherein the position of the virtual viewpoint moves in response to a swipe operation on the display, and wherein the controller selects a movement restriction mode that imposes restrictions on movement of the virtual viewpoint in response to the detected vehicle speed, and when the movement restriction mode is selected, movement of the virtual viewpoint is restricted to one unit per operation regardless of the amount of swipe of the swipe operation, and when the movement restriction mode is released, the virtual viewpoint can move freely in response to the amount of swipe.

5. A display control device according to any one of claims 1 to 4, wherein the controller automatically switches the virtual viewpoint when the vehicle transitions from a stopped or parked state to a traveling state so that the displayed image becomes an image showing the direction in which the vehicle is traveling.

6. A display control method executed by a controller to control display on a display, wherein the controller: acquires camera images from a plurality of vehicle-mounted cameras that capture images of the surroundings of the vehicle; generates a display image of the vehicle as seen from a virtual viewpoint based on the plurality of camera images; outputs the display image to a display; detects the speed of the vehicle; moves the position of the virtual viewpoint through user operation; and limits the movement of the virtual viewpoint through user operation in accordance with the detected vehicle speed.

7. A recording medium on which a control program executed by a controller is recorded, the control program causing the controller to execute processes including: acquiring camera images from a plurality of vehicle-mounted cameras that capture images of the surroundings of the vehicle; generating a display image of the vehicle as seen from a virtual viewpoint based on the plurality of camera images; outputting the display image to a display; detecting the speed of the vehicle; moving the position of the virtual viewpoint through user operation; and restricting the movement of the virtual viewpoint through user operation in accordance with the detected vehicle speed.

Images