Display control device, display control method, and recording medium

Abstract

Provided is a display control device comprising a controller (10) that acquires camera images from a plurality of vehicle-mounted cameras which image the surroundings of a vehicle, that generates a bird's-eye image of the vehicle on the basis of the plurality of camera images, and that outputs the bird's-eye image to a display (41), wherein the controller (10): acquires an operation command by a user for designating a designated position on a display screen of a display (41), in a state in which the bird's-eye image is displayed on the display screen; identifies a designated point corresponding to the designated position in the surroundings of the vehicle; generates, on the basis of the plurality of camera images, a designated point surrounding image which three-dimensionally depicts the surroundings of the designated point, with the designated point at the center; and outputs the designated point surrounding image to the display (41).

Description

Display control device, display control method, and recording medium

[0001] The present invention relates to a display control device, a display control method, and a recording medium.

[0002] Conventionally, there has been known a display control device that controls the display of an image showing the surroundings of a vehicle (for example, see Patent Document 1). The display control device described in Patent Document 1 includes an image generation unit that generates a three-dimensional image showing the surroundings of the vehicle based on images captured by a plurality of on-board cameras that capture the surroundings of the vehicle, and outputs a display image to be displayed on a display device based on the three-dimensional image, and a viewpoint change unit that, when a user swipes on the three-dimensional image displayed on the display device, changes a viewpoint parameter based on the position of the swipe and either the amount or speed of the swipe.

[0003] JP 2023-156871 A

[0004] The display control device described in Patent Document 1 has a problem in that the user cannot check any location because it can only display a car image rotated around a point on the car image.

[0005] The problem to be solved by the present invention is to provide a display control method and a recording medium that allow any location to be confirmed.

[0006] The present invention solves the above problem by obtaining an operation command that specifies a specified position on the display screen of a display, identifying a specified point corresponding to the specified position around the vehicle, generating an image of the surroundings of the specified point that represents the area around the specified point in three dimensions based on multiple camera images, and outputting the image of the surroundings of the specified point to the display.

[0007] According to the present invention, any location can be confirmed.

[0008] FIG. 1 is a block diagram showing the configuration of a display control system according to this embodiment. FIG. 2 is a plan view showing a portion of an instrument panel. FIG. 3 is a diagram showing a display screen of a display. FIG. 4 is a diagram showing a display screen of a display. FIG. 5 is a diagram showing a display screen of a 3D view of the display. FIG. 6 is a diagram showing a display screen of a display. FIG. 7 is a diagram showing a display screen of a display. FIG. 8 is a diagram showing a display screen of a display. FIG. 9 is a conceptual diagram showing a reference curved surface coordinate system at the center of the vehicle and a reference curved surface coordinate system at a specified position. FIG. 10 is a flowchart of a display control method executed by a controller.

[0009] A display control system including a display control device according to an embodiment of the present invention will be described below with reference to the drawings. The display control system is a system that generates an image of the surroundings of a vehicle. FIG. 1 is a block diagram showing the configuration of the display control system according to this embodiment. The display control system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an ATCU 30, a display 41, a display 42, a vehicle speed sensor 50, and a display control device 100. The display control system 1 is installed in a vehicle. The front camera 21, the right camera 22, the left camera 23, the rear camera 24, the ATCU 30, the in-vehicle display 40, the vehicle speed sensor 50, and the display control device 100 are connected via an in-vehicle communication network such as CAN or LIN.

[0010] The front camera 21, right camera 22, left camera 23, and rear camera 24 are provided on the vehicle and capture images of the surroundings of the vehicle. The front camera 21, right camera 22, left camera 23, and rear camera 24 capture images of the surroundings of the vehicle while the vehicle is traveling. The front camera 21 captures images in front of the vehicle, the right camera 22 captures images on the right side of the vehicle, the left camera 23 captures images on the left side of the vehicle, and the rear camera 24 captures images on the rear of the vehicle. The front camera 21 is provided on the grille of the vehicle, the right camera 22 is provided on the right door mirror, the left camera 23 is provided on the left door mirror, and the rear camera 24 is provided on the back door. Note that the various cameras may be provided in other positions on the vehicle. Note that in the following description, the front camera 21, right camera 22, left camera 23, and rear camera 24 will be collectively referred to as camera 20.

[0011] The ATCU (automatic transmission control unit) 30 is a control device that controls the transmission. The ATCU 30 outputs a control signal including shift information to the display control device 100. The shift information indicates the position of the shift lever, such as drive, neutral, reverse, or parking.

[0012] The in-vehicle display 40 is a display included in an in-vehicle infotainment (IVI) system and includes a display 41 and a display 42. The displays 41 and 42 are provided on the vehicle's instrument panel. The displays 41 and 42 are touch-panel displays. FIG. 2 is a plan view showing a portion of the instrument panel. As shown in FIG. 2, the display 41 is provided in the center of the instrument panel. The display 42 is provided on the instrument panel in front of the driver. The display 41 displays, for example, a map for a navigation system, a menu screen (such as a selection screen for various applications) for an entertainment system, content video, etc. The display 42 displays, for example, meters, a range indicator showing the current range of the shift position, an icon showing the status of a seat belt, a vehicle image, an image of the vehicle's surroundings, a predicted course, a driving route, etc. The display 41 and / or the display 42 also display images generated by the display control device 100.

[0013] The displays 41 and 42 have display screens that are long horizontally along the vehicle width direction (the left-right direction on the paper surface of FIG. 2 ). As shown in FIG. 2 , the displays 41 and 42 are arranged side by side on the left and right along the vehicle width direction, with their display screens close to each other. Note that the vehicle does not need to be equipped with both the displays 41 and 42, and may be equipped with only one of the displays. The vehicle speed sensor 50 detects the vehicle speed and outputs the detected value to the controller 10.

[0014] The display control device 100 includes a controller 10 and a recording medium 19. The controller 10 has functions such as acquiring various data, such as camera images and vehicle data, and image processing functions. The controller 10 has, as functional blocks, an image acquisition unit 11, a vehicle data acquisition unit 12, and an image processing unit 13. The controller 10 stores programs for implementing various functions in the recording medium 19, and executes the control programs by causing a processor to execute the processing flow indicated by the programs. Note that the recording medium 19 storing the control programs may be stored as memory within the controller 10. The recording medium 19 may be located inside or outside the controller 10. For example, the recording medium 19 may be stored in a server outside the vehicle, and the display control device 100 may access the server and store the control program data recorded on the recording medium 19 in the memory within the controller 10. The display control device 100 may also execute the control program stored in the server via communication with the server. For example, the display control device 100 transmits camera images and commands for executing each processing step included in the control program to the server. A controller included in the server executes a control program and processes data such as camera images based on instructions from the display control device 100. The server transmits a display image obtained by processing the control program to the display control device 100. The display control device 100 transmits the display image to the displays 41 and 42, causing the image to be displayed.

[0015] The image acquisition unit 11 acquires camera images from the camera 20, etc. The vehicle data acquisition unit 12 acquires vehicle data from sensors mounted on the vehicle, etc., via an in-vehicle communication network. The vehicle data is data indicating the current state of the vehicle, and includes, for example, vehicle speed, steering angle, current vehicle position (current coordinates), and gear shift. The vehicle data acquisition unit 12 acquires vehicle speed from the vehicle speed sensor 50 and acquires steering angle from the steering angle sensor. The vehicle data acquisition unit 12 acquires current position information from a navigation system. The vehicle data acquisition unit 12 also acquires gear shift information from the ATCU 30.

[0016] The image processing unit 13 performs image processing on the camera image to display a current image of the vehicle's surroundings and / or a vehicle image on the displays 41, 42. The image processing unit 13 generates a display image to display the image on the displays 41, 42 in a display mode specified by the user or the system, and outputs the display image to the displays 41, 42. There are multiple display modes, such as a front view showing the front of the vehicle, a rear view showing the rear of the vehicle, a top view giving a bird's-eye view of the vehicle, a 3D view showing the vehicle or its surroundings from a virtual viewpoint, and a skeleton view. In the front view, an image showing the exterior of the vehicle is displayed, which is located in front of the vehicle's grille. In the rear view, an image showing the exterior of the vehicle is displayed, which is located behind the vehicle's tailgate. In the top view (around view), a bird's-eye view image of the vehicle viewed from a virtual viewpoint directly above the vehicle is displayed. In the top view, a two-dimensional image is displayed, and the vehicle image is shown as a plan view of the vehicle. If there are tall three-dimensional objects such as walls around the vehicle, an image of the surroundings is displayed, showing the three-dimensional objects as if they have fallen over. In the 3D view, a virtual viewpoint is used to display an image of the object from a virtual viewpoint. The object is not limited to the vehicle, but may also be an obstacle around the vehicle. The virtual viewpoint is a viewpoint that allows a virtual view of the object from outside the object around the object. The virtual viewpoint is movable. For example, when a display image (three-dimensional image) of the vehicle viewed from a virtual viewpoint is displayed on the displays 41 and 42, the three-dimensional vehicle image can be rotated by orbiting the virtual viewpoint along a plane around the vehicle. In addition, the displays 41 and 42 may display the surroundings of the vehicle in other display modes, such as a skeleton view, a front wide view, or a rear wide view. In the skeleton view, a virtual image showing the condition of the vehicle's underfloor is displayed in addition to a current camera image (real image) showing the surroundings of the vehicle. For example, when a vehicle is turning left or right at an intersection with poor visibility such as a T-junction, the front wide view can display an image of the area in front of the vehicle that is in the driver's blind spot from an image taken by a front camera 21 having a wide-angle lens.

[0017] The display mode can be specified by the user or can be changed by the vehicle system as appropriate depending on the driving state of the vehicle. For example, when the range is shifted to reverse to park the vehicle, the display mode may be automatically switched to a top view or a rear view. Alternatively, the display mode may be switched by the driver operating a switch or the like while the vehicle is moving.

[0018] The image processing unit 13 can display images in multiple display modes on multiple screens on the display 41. The image processing unit 13 may display images in different display modes on the displays 41 and 42, or may display images in the same display mode on the displays 41 and 42. The image processing unit 13 may also use both the screens of the displays 41 and 42 as a single display screen to display images in multiple display modes, or may display images in a single display mode. The following description will explain the image processing performed by the image processing unit 13 when displaying an image on the display 41, but when displaying an image on the display 42, the image processing performed on the display 41 may be applied to the image display on the display 42.

[0019] The image processing unit 13 generates a display image according to a display mode based on the multiple camera images acquired by the image acquisition unit 11, and outputs the display image to the display 41. When displaying an image in a front view or rear view, the image processing unit 13 generates a display image showing the front or rear of the vehicle based on the camera image from the front camera 21 or the rear camera 24.

[0020] When displaying an image in top view, the image processing unit 13 generates an overhead image of the vehicle based on multiple camera images captured by the cameras 20. The virtual viewpoint of the overhead image is set at a predetermined position vertically away from the roof of the vehicle and outside the vehicle. For example, the image processing unit 13 performs viewpoint conversion processing on images captured by the front camera 21, right camera 22, left camera 23, and rear camera 24 to generate viewpoint-converted images that show the front, rear, left, and right regions of the vehicle from the virtual viewpoint. That is, the image processing unit 13 converts camera images corresponding to images viewed from the camera positions into viewpoint-converted images viewed from the virtual viewpoint. The viewpoint conversion from the camera positions to the virtual viewpoint is performed by calculation using viewpoint parameters, and the viewpoint parameters are stored in the recording medium 19. The image processing unit 13 fits each image after the viewpoint conversion processing to the surrounding area of ​​the vehicle. The image processing unit 13 performs blending processing on overlapping portions of the camera images from the cameras 20. The blending process is a process for maintaining continuity between two overlapping images, and any method known at the time of filing of this application can be used. The image after the blending process becomes a surrounding image showing the surroundings of the vehicle. The image processing unit 13 also combines the vehicle image located at the center of the screen with the surrounding image. The combined image becomes an overhead image, and the image processing unit 13 outputs the overhead image to the display 41 as a display image.

[0021] When displaying an image in 3D view, the image processing unit 13 generates a display image of the vehicle as viewed from a virtual viewpoint based on multiple camera images captured by the camera 20. First, the image processing unit 13 sets a virtual viewpoint based on a user specification or the driving situation. The position and line of sight of the virtual viewpoint can be arbitrarily set based on a user instruction. The user can specify the position and line of sight of the virtual viewpoint by, for example, touching the display 41. The image processing unit 13 can also set the virtual viewpoint based on the driving situation. For example, the virtual viewpoint can be set behind the vehicle when driving forward to view an image in front of the vehicle, and in front of the vehicle when driving backward to view an image behind the vehicle. Specifically, the image processing unit 13 sets the position of the virtual viewpoint when it receives shift information indicating that the gearshift position is in reverse. The image processing unit 13 can also set the virtual viewpoint based on the vehicle speed. For example, the virtual viewpoint can be set based on the vehicle speed input from the vehicle speed sensor 50 so that the virtual viewpoint is farther away from the vehicle when the vehicle speed is equal to or greater than a predetermined value, and closer to the vehicle when the vehicle speed is less than the predetermined value.

[0022] The image processing unit 13 deforms at least a part of a reference curved surface coordinate system having a predefined curved surface for projecting the camera image, according to the position of the virtual viewpoint. The reference curved surface coordinate system is stored in the recording medium 19. The reference curved surface coordinate system has, for example, a bowl shape surrounding the vehicle. The reference curved surface coordinate system has a center on an xy plane parallel to the vehicle's loading surface (travel surface), and is formed from a curved surface having a curvature from the center or near the center and a component in the vehicle's height direction (z direction).

[0023] A base parallel to the xy plane can be formed near the center of the reference curved surface coordinate system, and the base can have any shape, such as a rectangle, a triangle, an ellipse, or a circle. The curvature of the curved surface of the reference curved surface coordinate system does not have to be uniform. For example, the curvature of the curved surface near the base may be relatively large, and the curvature of the curved surface away from the base may be relatively small. The image processing unit 13 may store in advance multiple reference curved surface coordinate systems with different shapes depending on the shape (size, form) of the vehicle. When multiple reference curved surface coordinate systems are stored, the reference curved surface coordinate systems to be used may be stored in the recording medium 19 in association with the vehicle speed, the position of the vehicle shift position, etc., and the image processing unit 13 may select a reference curved surface coordinate system depending on the vehicle speed, the position of the vehicle shift position, etc.

[0024] The image processing unit 13 may read a pre-stored reference curved surface coordinate system and deform the shape of the read reference curved surface coordinate system. The image processing unit 13 may deform the shape of the reference curved surface coordinate system according to the position of the virtual viewpoint. For example, when the virtual viewpoint is located outside the outer edge of the reference curved surface coordinate system, a portion of the reference curved surface coordinate system may be deformed so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system. By deforming a portion of the reference curved surface coordinate system so that the virtual viewpoint is located inside the outer edge of the reference curved surface coordinate system, distortion of the shape of an obstacle with height can be suppressed when the obstacle is displayed on the screen, and an image without any missing image can be displayed on the display 41. Note that the method of deforming the reference curved surface coordinate system may use other coordinate system deformation methods known at the time of filing.

[0025] The image processing unit 13 projects multiple camera images acquired from the camera 20 onto the reference curved coordinate system. When the reference curved coordinate system is deformed, the image processing unit 13 simply projects the camera images onto the curved coordinate system after the deformation (deformable coordinate system). In order to project the camera image data onto the reference curved coordinate system or the deformed coordinate system after the deformation, the image processing unit 13 has an image conversion table that associates the coordinates of pixels included in the camera images with the coordinates of the coordinate system. The image conversion table is recorded on the recording medium 19.

[0026] A prepared vehicle image of a vehicle may also be superimposed on the reference curved coordinate system. The vehicle image may be created in advance based on the design of the vehicle and stored in the recording medium 19. By superimposing the vehicle image on the reference curved coordinate system, not only the image of the vehicle's surroundings but also the vehicle itself can be displayed in three dimensions, which helps understand the relationship between the vehicle's position and orientation and the surrounding image. In this way, the image processing unit 13 refers to the image conversion table to associate the coordinates of pixels contained in the camera image with the coordinates of the coordinate system, projects the camera image onto the reference curved coordinate system or the deformed coordinate system, and superimposes the vehicle image on the projected three-dimensional image to generate a display image (three-dimensional image) in a 3D view in which the vehicle and its surroundings are viewed from a virtual viewpoint.

[0027] The image processing unit 13 may generate display images viewed from selectable virtual viewpoints, not limited to those specified by the user or set based on the driving situation. For example, when a display image is displayed on the display 41 in a 3D view, the user can freely move the position of the virtual viewpoint by touching the display 41. When the 3D view display is selected, the image processing unit 13 generates multiple 3D view display images for each of multiple virtual viewpoints selectable by the user. Then, for example, when the user moves the virtual viewpoint by touching the display 41, a 3D image in which the vehicle image rotates around the center point of a curved coordinate system is displayed on the display 41. Note that the image processing unit 13 may set virtual viewpoints at any positions around the vehicle, not limited to those selectable by the user, and generate multiple 3D view display images for each virtual viewpoint. In other words, when a 3D image is displayed on the display 41 in a rotating display, the image processing unit 13 generates a continuous 3D image so that the image can be displayed smoothly in response to the movement of the virtual viewpoint.

[0028] The image processing method for the display images displayed in the front / rear view, top view, and 3D view display modes is not limited to the above, and other image processing methods can be used. Furthermore, the image processing method for the display images displayed in other display modes, such as skeleton view and front / rear wide view, can be any method known at the time of filing.

[0029] Next, display control by the display control device 100 will be described along with specific examples of the display screen of the display 41. FIG. 3 shows the display screen of the display 41. In the example of FIG. 3, the vehicle is stopped on a straight road, and the left screen of the display 41 displays a top-view image, while the right screen of the display 41 displays a 3D-view image. When displaying a top-view overhead image and a 3D-view display image on the display 41, the controller 10 generates a top-view image 46 by superimposing viewpoint icons 43a-43h and an automatic display start icon 44 on the overhead image of the vehicle. The top-view image 46 includes an overhead image of the vehicle viewed from above, viewpoint icons 43a-43h, and the automatic display start icon 44. The viewpoint icons 43a-43h indicate the position of a virtual viewpoint and / or the direction of the line of sight and are user-selectable icons. The viewpoint icons 43a-43h are arranged around the vehicle, with the vehicle image 45 at the center. Viewpoint icon 43a is positioned above the vehicle image located at the center of the overhead image, while viewpoint icons 43c, 43e, and 43g are positioned to the right, bottom, and left of the vehicle image, respectively. Viewpoint icon 43b, viewpoint icon 43d, viewpoint icon 43f, and viewpoint icon 43h are positioned in the upper right, lower right, lower left, and lower right directions, respectively, of the vehicle image. When the user selects one of viewpoint icons 43a to 43h, a display image (three-dimensional image) of the vehicle as viewed from the selected viewpoint icon is displayed on the right side of display 41.

[0030] As shown in FIG. 3 , when the user touches the viewpoint icon 43a, the controller 10 displays the viewpoint icon 43a as a selection icon. The controller 10 displays the selection icon by, for example, changing the color of the viewpoint icon 43a. In the example of FIG. 3 , the selection icon is represented by a thick line surrounding the viewpoint icon 43a. The image processing unit 13 sets a virtual viewpoint corresponding to the viewpoint icon 43a. This allows the virtual viewpoint to be set according to the user's specification. In the example of FIG. 3 , a position from which the vehicle is viewed from the front is set as the virtual viewpoint. Then, the image processing unit 13 generates a three-dimensional image 47 of the vehicle and its surroundings viewed from the front of the vehicle based on multiple camera images. The image processing unit 13 outputs the three-dimensional image to the display 41 as a display image. The three-dimensional image 47 is displayed on the right screen of the display 41. Then, as shown in FIG. 3 , the display 41 displays a top-view image 46 and the three-dimensional image 47 side by side.

[0031] The automatic display start icon 44 is an icon for starting automatic display, which automatically switches three-dimensional images in the 3D view by moving the virtual viewpoint. The automatic display start icon 44 is displayed at the center position of the vehicle image 45 included in the top-view image 46. When automatic display is executed, predetermined positions around the vehicle become the start and end points of the virtual viewpoint, and the virtual viewpoint moves from the start point to the end point, causing the vehicle image displayed in the 3D view to rotate 360 ​​degrees along a plane parallel to the road surface. In other words, in automatic display, the center point is placed on the vehicle image, and the vehicle image and the vehicle-surrounding image rotate 360 ​​degrees around the center point. For example, in the example of FIG. 3 , when the user touches the automatic display start icon 44, the selected viewpoint icon 43 a becomes the start and end points of the virtual viewpoint. The virtual viewpoint moves from the position corresponding to the viewpoint icon 43 a to viewpoint icon 43 b in order to viewpoint icon 43 h, rotating clockwise around the vehicle, and then returns to the position corresponding to the original viewpoint icon 43 a. As the virtual viewpoint rotates, the three-dimensional image 47 in the 3D view rotates. The orientation of the vehicle image included in the three-dimensional image 47 starts from the front (front) as shown in FIG. 3 , changes in the order of the right side, rear (back), and left side, and then returns to the original front (front). In other words, the three-dimensional image is displayed as if it had rotated 360 degrees (hereinafter also referred to as "360-degree rotation display"). As a result, the display image (three-dimensional image 47) is automatically rotated 360 degrees in the 3D view.

[0032] FIG. 4 shows a display screen of the display 41. In the example of FIG. 4, the vehicle is stopped at a T-junction, and the left screen of the display 41 displays a top-view image, while the right screen of the display 41 displays a 3D-view image. The image processing method for the top-view image 46 and the 3D image 47 is the same as in the example of FIG. 3, and the top-view image 46 also includes viewpoint icons 43a-h and an automatic display start icon 44. Unlike the example of FIG. 3, viewpoint icon 43e is selected, and the 3D view displays a 3D image 47 of the vehicle and its surroundings as seen from behind the vehicle. Note that, as shown in FIG. 4, when the vehicle is stopped at a T-junction, the controller 10 may set a virtual viewpoint corresponding to viewpoint icon 43a and display a 3D image 47 as seen from the virtual viewpoint on the display 41. This allows the state of the blind spot to be displayed on the display 41 if there is a blind spot ahead of the vehicle.

[0033] 3 and 4, while the vehicle is stopped, the user can select one of the viewpoint icons 43a to 43h and view a three-dimensional image 47 from the direction of the selected viewpoint icon 43a to 43h in a 3D view on the display 41. For example, if the user touches the position of the viewpoint icon 43c on the screen of the display 41, the image processing unit 13 switches the viewpoint icon 43c to a selected icon and moves the current virtual viewpoint (in the example of FIG. 3, the virtual viewpoint corresponding to the viewpoint icon 43a) to the virtual viewpoint corresponding to the viewpoint icon 43c. The image processing unit 13 then generates a three-dimensional image viewed from the virtual viewpoint based on the camera image and outputs the display image to the display 41 so that the three-dimensional image is displayed in a 3D view. A three-dimensional image of the vehicle viewed from the right side is displayed on the display 41.

[0034] Furthermore, while the vehicle is stopped, the display control device 100 may perform display control so that the user can not only select a virtual viewpoint by touching the viewpoint icons 43a to 43h, but also arbitrarily specify the position of the virtual viewpoint. The user touches any position on the 3D view display screen. Then, while the user's finger is touching the screen, the user moves the finger in any direction (swipe operation). While the finger is touching the screen, the position of the virtual viewpoint moves in accordance with the swipe operation, and the 3D image rotates in accordance with the movement of the virtual viewpoint. The amount of movement of the virtual viewpoint is determined by the swipe amount (corresponding to the amount the finger moves while touching the screen), and the greater the swipe amount, the greater the movement amount. Note that the amount of movement of the virtual viewpoint corresponds to the length of the arc of a circle centered on the vehicle image. For example, the user touches the 3D view display screen with their finger and moves their finger with a large swipe amount. If the swipe amount is large, the image processing unit 13 moves the virtual viewpoint around the center point of the vehicle image, thereby rotating the 3D image 47. If the swipe amount is small, image processing unit 13 rotates three-dimensional image 47 so that the virtual viewpoint moves along an arc centered on the center point of the vehicle image by an amount equivalent to the swipe amount. Note that the rotational display caused by a swipe operation is not limited to the 3D view display screen, but may also be performed by a swipe operation on a top view display screen. For example, image processing unit 13 causes display 41 to display an image in which viewpoint icons 43a to 43h have been removed from a top view image 46 as shown in FIG. 3 or 4 . Then, the user performs a swipe operation at any position on the screen displaying top view image 46. Image processing unit 13 rotates the three-dimensional image on the 3D view display screen in response to the swipe operation on the top view display screen.

[0035] 5A and 5B are diagrams illustrating the rotational display of a three-dimensional image 47 in a 3D view. Fig. 5A shows a three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43e, Fig. 5B shows a three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43f, and Fig. 5C shows a three-dimensional image 45 viewed from a virtual viewpoint corresponding to viewpoint icon 43g. Note that Fig. 5 is a 3D view display screen of the display 41, and shows only the vehicle image included in the three-dimensional image 47, omitting an image of the vehicle's surroundings. The virtual viewpoint that serves as the starting point for the rotational display is the position corresponding to viewpoint icon 43e.

[0036] As shown in FIG. 5A, the state in which the vehicle image included in the three-dimensional image 47 faces rearward (backward) is the starting point of the rotational display. As the virtual viewpoint moves, the vehicle image rotates 45 degrees, resulting in the state shown in FIG. 5B. Further rotation results in the state shown in FIG. 5C. The three-dimensional image 47 has rotated 90 degrees so far, and then rotates the remaining 270 degrees. That is, the orientation of the vehicle image included in the three-dimensional image 47 starts from the rearward (backward) side, changes in this order to the left side, front (front), and right side, and then returns to the original rearward (backward) side. The surrounding images located around the vehicle image also rotate primarily along with the vehicle image. As a result, the display image (three-dimensional image 47) is automatically rotated 360 degrees in the 3D view.

[0037] In this embodiment, the area around a specified point designated by the user is displayed as a three-dimensional image with the specified point as the center. The control content when displaying the area around the specified point as a three-dimensional image will be described below.

[0038] FIG. 6 shows the display screen of the display 41. In the example of FIG. 6, a vehicle is stopped on a straight road, and an obstacle 48 is located in front of and to the left of the vehicle. As shown in FIG. 6, an image of the obstacle 48 is displayed in front of and to the left of the vehicle image 45 on the top-view screen. The display control device 100 executes display control so that the user can not only select a virtual viewpoint by touching viewpoint icons 43a to 43h, but also arbitrarily specify the position of the virtual viewpoint on the top-view screen. The user touches an arbitrary position on the top-view display screen. For example, the user touches the image of the obstacle 48. The touch-panel display 41 outputs an operation command indicating the designated position specified by the user's touch operation to the controller 10. The vehicle data acquisition unit 12 acquires the user's operation command. The image processing unit 13 identifies a designated point corresponding to the designated position around the vehicle. In the example of FIG. 6, an obstacle is present at the designated position specified by the touch operation, so the position of the obstacle becomes the designated point corresponding to the designated position. In other words, the specified position is a position specified by the user on the top-view screen, and a position representing the specified position in the actual space around the vehicle is the specified point. The image processing unit 13 sets the specified point as a virtual viewpoint, generates a three-dimensional image 47 of the vehicle viewed from the virtual viewpoint based on the camera image, and outputs the three-dimensional image 47 to the display 41. As shown in Fig. 6, a three-dimensional image of the vehicle viewed from an obstacle is displayed on the left side of the display 41.

[0039] 7 and 8 show display screens of the display 41. The examples of FIGS. 7 and 8 are similar to the example of FIG. 6, except that in the example of FIG. 6, a vehicle is stopped on a straight road and an obstacle 48 is located ahead of the vehicle on the left side. The three-dimensional images in the examples of FIGS. 7 and 8 are different from those in the example of FIG. 6. Furthermore, the rotation angle of the designated point surrounding image 49 displayed in the 3D view is different in the examples of FIGS. 7 and 8. When an object is present at a specified position specified by the user, the controller 10 causes the display 41 to display the designated point surrounding image 49, which represents the periphery of the object in three dimensions with the object at the center.

[0040] As described above, when generating a three-dimensional image of the vehicle and its surroundings viewed from a virtual viewpoint, the image processing unit 13 reads out the reference curved surface coordinate system, projects the camera image onto the read-out reference curved surface coordinate system, and superimposes the vehicle image on the origin of the reference curved surface coordinate system to generate the three-dimensional image 47.

[0041] Meanwhile, the image processing unit 13 generates the designated point surrounding image 49 in the following manner. First, the image processing unit 13 measures the distance between the designated position and the vehicle position. In the examples of FIGS. 6 to 8 , the measured distance corresponds to the distance between the position of the obstacle 48 and the center of the vehicle. The image processing unit 13 shifts the position of the reference coordinate system from the center of the vehicle by the measured distance. FIG. 9 is a conceptual diagram showing a reference curved surface coordinate system at the center of the vehicle and a reference curved surface coordinate system at the designated position. The xy plane in the reference curved surface coordinate system corresponds to the ground (the surface on which the vehicle travels). In FIG. 9 , the dashed-dotted curve a represents the reference curved surface coordinate system before the shift, and the dotted curve b represents the reference curved surface coordinate system after the shift. Note that the rising portion of the reference curved surface coordinate system indicates the distance from the origin and is represented by a parameter. However, because the same parameters are used for the rising portion of the reference curved surface coordinate system before and after the shift, the shape of the reference curved surface coordinate system remains the same before and after the shift. It is not necessary to use the same parameters before and after the shift, and the shape of the reference curved surface coordinate system may be changed by changing the parameters before and after the shift.

[0042] The image processing unit 13 projects the camera image onto the shifted reference curved surface coordinate system. If an object is present at the specified position designated by the user, the image processing unit 13 superimposes an image of the object on the origin of the shifted reference curved surface coordinate system. In the examples of FIGS. 6 to 8, an image of an obstacle 48 is superimposed on the origin of the shifted reference curved surface coordinate system. This generates a specified point surroundings image 49.

[0043] The image processing unit 13 can also rotate the designated point surrounding image 49 on the 3D view display screen in response to a swipe operation on the top view display screen. For example, as shown in FIG. 7 , while the designated point surrounding image 49 is displayed, the user touches any position on the top view screen or the 3D view display screen to perform a swipe operation. The image processing unit 13 rotates the designated point surrounding image 49 by moving the image of the obstacle 48 and the surrounding image of the obstacle 48 in a circular motion around a designated point on the image of the obstacle 48. The image processing unit 13 may rotate the designated point surrounding image 49 by an angle corresponding to the amount of swipe, or may rotate the designated point surrounding image 49 by a predetermined angle (e.g., 45 degrees) for each swipe operation. For example, when the user performs a swipe operation while the designated point surrounding image 49 of the 3D view shown in FIG. 7 is displayed, the designated point surrounding image 49 rotates, and the designated point surrounding image 49 shown in FIG. 8 is displayed. In the designated point surrounding image 49 of FIG. 7, the vehicle image is positioned further back than the obstacle image, but in the designated point surrounding image 49 of FIG. 8, the vehicle image is positioned in front of the obstacle image.

[0044] The image processing unit 13 may execute a display that rotates the designated point surrounding image 49 360 degrees around the designated point. For example, in the examples of FIGS. 7 and 8 , the three-dimensional image 49 rotates around the designated point on the image of the obstacle. As the three-dimensional image 49 rotates, the image of the obstacle also rotates 360 degrees. The designated point surrounding image 49 is a three-dimensional image of the designated point as seen from a virtual viewpoint around the designated point. The image processing unit 13 rotates the designated point surrounding image 49 so that the virtual viewpoint starts at a predetermined position around the designated point, rotates 360 degrees around the designated point, and returns to the same position as the start point.

[0045] Furthermore, if the multiple camera images captured by the cameras 20 cannot capture part of the surroundings centered on the specified point, the image processing unit 13 generates a specified point surroundings image so that part of the surroundings is displayed as a virtual image. In the example of FIG. 7 , an obstacle 48 is located on the left side in front of the vehicle. The front camera 21 and the left camera 23 cannot capture the area behind the obstacle 48 (the area on the opposite side of the vehicle from the obstacle 48), which becomes a blind spot area. The blind spot area becomes part of the surroundings centered on the specified point. In this embodiment, the image of the blind spot area is displayed using a virtual image.

[0046] The virtual image may be, for example, a road surface image, a sign image such as a crosswalk, or a background image of a building, tree, or wall, and may be any of a variety of images that can serve as the surrounding image of the specified point. A reference image used as the virtual image is recorded in the recording medium 19. When generating the surrounding image 49 of the specified point, if the image processing unit 13 cannot acquire a portion of the image of the specified point from the camera image, the area where the camera image cannot be acquired is designated as a blind spot. In the examples of FIGS. 7 and 8 , the area 90 corresponds to the blind spot. The image processing unit 13 uses the reference image stored in the recording medium 19 to generate a virtual image that maintains continuity with the image surrounding the area 90. For example, in the 3D view of FIG. 7 , the road surface is captured around the area 90. Furthermore, a white line is captured on the opposite side of the area 90, separated by an obstacle 48. Based on the road surface and the white line surrounding the area 90, the image processing unit 13 generates a virtual image representing the road surface including the white line within the area 90. The image processing unit 13 then fills the area 90 with the virtual image. The image processing unit 14 outputs the designated point surrounding image 49 including the virtual image to the display 41 .

[0047] Next, a display control method by the controller 10 will be described with reference to Fig. 10. Fig. 10 is a flowchart of the display control method by the controller 10. The controller 10 repeatedly executes the following control flow while the left display image on the display 41 is displayed in top view and the right display image is displayed in 3D view.

[0048] In step S1, image acquisition unit 11 acquires camera images from camera 20. In step S2, image processing unit 13 generates a top view image of the vehicle as seen from above, based on the multiple camera images. In step S3, image processing unit 13 outputs the top view image to display 41, causing the top view image to be displayed on display 41. In step S4, when the user touches any position on the top view image, vehicle data acquisition unit 12 acquires an operation command from the user specifying the touched position as a designated position.

[0049] In step S5, the controller 10 identifies a designated point corresponding to the designated position around the vehicle. In step S6, the image generation unit 13 determines whether or not it is possible to acquire the camera image necessary for generating the designated point surrounding image. If the display range of the designated point surrounding image can be displayed in the camera image, the image generation unit 13 determines that it is possible to acquire the camera image necessary for generating the designated point surrounding image. On the other hand, if there is a blind spot area in the display range of the designated point surrounding image and the blind spot area cannot be displayed in the camera image, the image generation unit 13 determines that it is impossible to acquire the camera image necessary for generating the designated point surrounding image.

[0050] If it is determined that the camera images necessary for generating the designated point surrounding image can be acquired, the controller 10 executes the control flow of step S8. If it is determined that the camera images necessary for generating the designated point surrounding image cannot be acquired, the image generation unit 13 generates a virtual image of the area that cannot be displayed in the camera image, using a reference image recorded on the recording medium 19. The image generation unit 13 fills in the area that cannot be displayed in the camera image with the virtual image.

[0051] In step S8, the image generation unit 13 generates a designated point surrounding image based on the camera image and / or the virtual image. In step S9, the image generation unit 13 outputs the designated point surrounding image to the display 41, and causes the designated point surrounding image to be displayed on the display 41. After the control flow of step S9 is executed, the control flow ends. The controller 10 restarts the control flow from step S1.

[0052] As described above, in this embodiment, controller 10 acquires camera images from camera 20, generates an overhead image of the vehicle based on the multiple camera images, and outputs the overhead image to display 41. Furthermore, while the overhead image is displayed on the display screen of display 41, controller 10 acquires a user operation command specifying a specified position on the display screen, identifies a specified point corresponding to the specified position around the vehicle, generates a specified point surrounding image 49 that represents the area around the specified point in three dimensions based on the multiple camera images, and outputs specified point surrounding image 49 to display 41. This allows the user to check any location.

[0053] In this embodiment, the controller 10 generates a three-dimensional image of the vehicle as seen from a virtual viewpoint based on multiple camera images, outputs the three-dimensional image to the display 10, and performs automatic display by moving the virtual viewpoint and automatically switching the three-dimensional image. The controller 10 also outputs a top-view image including an automatic display start icon to the display 10. This allows the frequently used 360-degree automatic rotation display centered on the vehicle to be started by operating the icon. As a result, operational convenience can be improved.

[0054] In this embodiment, if the on-board camera cannot capture an image of the specified point and part of the surrounding area, the controller 10 generates an image of the surrounding area of ​​the specified point so that part of the surrounding area is displayed as a virtual image, allowing the user to check any location.

[0055] In this embodiment, when the camera 20 cannot capture an image of a part of the surroundings around the specified point, the controller 10 generates a specified point surrounding image 49 so that a part of the surroundings is displayed as a virtual image. This makes it possible to display an image of a part that is a blind spot due to an obstacle, for example.

[0056] In this embodiment, the controller 10 also executes a display that rotates the specified point surrounding image 49 360 degrees around the specified point, thereby allowing the user to check the surroundings of any location.

[0057] In this embodiment, the user's operation to specify the designated position is not limited to a touch operation on the display 41, but may also be a switch operation. For example, when the controller 10 detects an obstacle around the vehicle from a camera image, the controller 10 displays the detected obstacle on the display 41, and the user selects the obstacle by operating a cursor or a selection operation using a switch. The controller 10 determines the designated position of the obstacle based on the user's selection operation. Furthermore, the controller 10 may use a voice recognition function to acquire the user's designation operation for specifying the designated position on the display screen through dialogue with the user.

[0058] In this embodiment, the controller 10 does not need to execute all of the control flows shown in FIG. 9, and does not need to execute each control flow in the order shown in FIG.

[0059] REFERENCE SIGNS LIST 1 Display control system 10 Controller 11 Image acquisition unit 12 Vehicle data acquisition unit 13 Image processing unit 19 Recording medium 20 Camera 40 In-vehicle display 43a to 43h Viewpoint icon 44 Automatic display start icon 45 Vehicle image 46 Top view image 47 Three-dimensional image 48 Obstacle 49 Image of surroundings of specified point 50 Vehicle speed sensor 90 Area 100 Display control device

Claims

1. A display control device comprising a controller that acquires camera images from multiple on-board cameras that capture images of the surroundings of a vehicle, generates an overhead image of the vehicle based on the multiple camera images, and outputs the overhead image to a display, wherein the controller: acquires a user's operation command specifying a specified position on the display screen while the overhead image is displayed on the display screen of the display; identifies a specified point around the vehicle that corresponds to the specified position; generates a specified point surrounding image that shows the surroundings of the specified point in three dimensions based on the multiple camera images, with the specified point as the center; and outputs the specified point surrounding image to the display.

2. A display control device according to claim 1, wherein the controller generates a three-dimensional image of the vehicle as seen from a virtual viewpoint based on the plurality of camera images, outputs the three-dimensional image to the display, performs automatic display by moving the virtual viewpoint and automatically switching the three-dimensional image, includes an automatic display start icon for starting the automatic display, and outputs a top-view image of the vehicle as seen from above to the display.

3. A display control device according to claim 1 or 2, wherein the controller generates an image of the area around the specified point so that part of the area around the specified point is displayed as a virtual image when the vehicle-mounted camera cannot capture an image of the area around the specified point.

4. A display control device according to any one of claims 1 to 3, wherein the controller executes a display that rotates the image around the specified point by 360 degrees around the specified point.

5. A display control device according to claim 4, wherein the controller, when the user designates an object at the designated position, executes a display that rotates the object 360 degrees.

6. A display control method executed by a controller for controlling display on a display, wherein the controller: acquires camera images from a plurality of on-board cameras capturing images of the surroundings of a vehicle; generates an overhead image of the vehicle based on the plurality of camera images; outputs the overhead image to a display; acquires a user's operation command specifying a specified position on the display screen while the overhead image is displayed on the display screen of the display; identifies a specified point corresponding to the specified position around the vehicle; generates a specified point surrounding image based on the plurality of camera images, which represents the surroundings of the specified point in three dimensions with the specified point as the center; and outputs the specified point surrounding image to the display.

7. A recording medium on which a control program executed by a controller is recorded, the control program causing the controller to execute processes including the steps of: acquiring camera images from a plurality of on-board cameras that capture images of the surroundings of the vehicle; generating an overhead image of the vehicle based on the plurality of camera images; outputting the overhead image to a display; acquiring a user's operation command specifying a specified position on the display screen while the overhead image is displayed on the display screen of the display; identifying a designated point corresponding to the specified position around the vehicle; generating a designated point surrounding image based on the plurality of camera images, which represents the surroundings of the designated point in three dimensions with the designated point as the center; and outputting the designated point surrounding image to the display.

Images