Display control device, display control method, and recording medium

Abstract

A display control device has a controller (10) that acquires camera images from multiple on-board cameras that capture images of the vehicle’s surroundings, generates a display image of the vehicle from a virtual viewpoint based on the multiple camera images, and outputs the display image to a display (41). The controller (10) performs automatic display of the display image by moving the virtual viewpoint at a timed time at least once before the vehicle starts moving and after parking.

Description

Technical Field

[0001] This invention relates to a display control device, a display control method, and a recording medium. Background Technology

[0002] Conventionally, display control devices that control the display of images representing the surroundings of a vehicle are known (e.g., Patent Document 1). The display control device described in Patent Document 1 includes: an image generation unit that generates a three-dimensional image representing the surroundings of the vehicle based on images captured by multiple vehicle-mounted cameras that capture images of the surroundings of the vehicle, and outputs a display image displayed on a display device based on the three-dimensional image; and a viewpoint changing unit that, when a user slides in the three-dimensional image displayed on the display device, changes viewpoint parameters based on the position of the slide and either the amount or speed of the slide.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-156871 Summary of the Invention

[0004] The display control device described in Patent Document 1 has the following problem: since the display image is displayed based on the user's operation regardless of the vehicle's state, it is impossible to display an image that matches the vehicle's state.

[0005] The problem this invention aims to solve is to provide a display control device, a display control method, and a recording medium capable of displaying images that match the state of a vehicle.

[0006] The present invention solves the above-mentioned problem by automatically displaying the image by moving the virtual viewpoint at a timing that occurs at least once before the vehicle starts moving and after the vehicle has been parked.

[0007] The effects of the invention

[0008] According to the present invention, operability can be improved. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating the structure of the display control system according to this embodiment.

[0010] Figure 2 It is a floor plan showing a portion of the dashboard.

[0011] Figure 3 It is a diagram representing the screen displayed on the monitor.

[0012] Figure 4 It is a diagram representing the screen displayed on the monitor.

[0013] Figure 5 It is a diagram that represents a three-dimensional view of the display screen.

[0014] Figure 6 It is a diagram representing the screen displayed on the monitor.

[0015] Figure 7 It is a diagram representing the screen displayed on the monitor.

[0016] Figure 8 This is a flowchart of the display control method executed by the controller. Detailed Implementation

[0017] Hereinafter, with reference to the accompanying drawings, a display control system including a display control device according to an embodiment of the present invention will be described. The display control system is a system that generates an image of the surroundings of a vehicle. Figure 1 This is a block diagram illustrating the structure of the display control system according to this embodiment. The display control system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an ATCU 30, a display 41, a display 42, a vehicle speed sensor 50, and a display control device 100. The display control system 1 is installed in a vehicle. The front camera 21, right camera 22, left camera 23, rear camera 24, ATCU 30, vehicle display 40, vehicle speed sensor 50, and display control device 100 are connected via an in-vehicle communication network such as CAN or LIN.

[0018] A front camera 21, a right camera 22, a left camera 23, and a rear camera 24 are mounted on the vehicle to photograph the area around the vehicle. The front camera 21 photographs the front of the vehicle, the right camera 22 photographs the right side of the vehicle, the left camera 23 photographs the left side of the vehicle, and the rear camera 24 photographs the rear of the vehicle. The front camera 21 is mounted on the vehicle's grille, the right camera 22 is mounted on the right rearview mirror, the left camera 23 is mounted on the left rearview mirror, and the rear camera 24 is mounted on the rear door. Alternatively, the cameras can be mounted in other locations on the vehicle. Furthermore, in the following description, when referring to the front camera 21, right camera 22, left camera 23, and rear camera 24 collectively, they are referred to as camera 20.

[0019] The ATCU (Automatic Transmission Control Unit) 30 is a control device that controls the transmission. The ATCU 30 outputs control signals, including shift information, to the display control device 100. The shift information indicates the shift position, showing the current state of the gear lever, such as forward, neutral, reverse, and parking.

[0020] The in-vehicle display 40 is a display included in the IVI (in-vehicle infotainment) system, comprising display 41 and display 42. Display 41 and display 42 are located on the vehicle's dashboard. Display 41 and display 42 are touchscreen displays. Figure 2 This is a floor plan showing a portion of the dashboard. For example... Figure 2 As shown, display 41 is located in the center of the instrument panel. Display 42 is located in the instrument panel, positioned in front of the driver. Display 41 displays, for example, maps from the navigation system, menu screens (selection screens for various applications, etc.) or content images from the entertainment system. Display 42 displays, for example, instruments, a gear position display indicating the current gear position, icons indicating the seatbelt status, vehicle images, images of the vehicle's surroundings, predicted driving routes, driving paths, etc. In addition, display 41 and / or display 42 display images generated by the display control device 100.

[0021] Displays 41 and 42 have a width direction ( Figure 2 A horizontally elongated display screen (in the left-right direction on the paper). For example... Figure 2 As shown, displays 41 and 42 are arranged horizontally along the vehicle width direction, with their display screens close to each other. Alternatively, the vehicle may not need both displays 41 and 42; either display is sufficient. The vehicle speed sensor 50 detects the vehicle speed and outputs the detected value to the controller 10.

[0022] The display control device 100 includes a controller 10 and a recording medium 19. The controller 10 has functions such as acquiring various data including camera images and vehicle data, and image processing functions. As a functional module, the controller 10 includes an image acquisition unit 11, a vehicle data acquisition unit 12, and an image processing unit 13. The controller 10 stores programs for implementing various functions in the recording medium 19, and executes the processing flow represented by the program by having a processor execute the control program. Furthermore, the recording medium 19, which stores the control program, can also be stored as memory within the controller 10. Additionally, the recording medium 19 can be located inside or outside the controller 10. For example, the recording medium 19 can be stored on a server outside the vehicle, and the display control device 100 can access the server to store the control program data recorded in the recording medium 19 into memory within the controller 10. Furthermore, the display control device 100 can also execute the control program within the server via communication with the server. For example, the display control device 100 can send instructions for executing the processing steps included in the control program or camera images to the server. The server, containing a controller, executes a control program based on instructions from the display control device 100 to process data such as camera images. The server then sends the processed display image to the display control device 100. The display control device 100 sends the display image to the monitors 41 and 42 and displays the image on them.

[0023] The image acquisition unit 11 acquires camera images from the camera 20, etc. The vehicle data acquisition unit 12 acquires vehicle data from sensors mounted in the vehicle via the in-vehicle communication network. Vehicle data represents the current state of the vehicle, including, for example, vehicle speed, steering angle, current position (current coordinates), and gear shift. The vehicle data acquisition unit 12 acquires vehicle speed from the vehicle speed sensor 50 and steering angle from the steering angle sensor. The vehicle data acquisition unit 12 acquires current position information from the navigation system. Additionally, the vehicle data acquisition unit 12 acquires gear shift information from the ATCU 30.

[0024] The image processing unit 13 performs image processing on the camera images in order to display the current surrounding image and / or vehicle image on the displays 41 and 42. The image processing unit 13 generates display images and outputs them to the displays 41 and 42 in a manner that displays the images on the displays 41 and 42 according to a display mode specified by the user or system. Multiple display modes exist, such as a front view showing the front of the vehicle, a rear view showing the rear of the vehicle, a top view looking down at the vehicle, a three-dimensional view of the vehicle or its surroundings from a virtual viewpoint, a perspective view, etc. In the front view, an image showing the exterior of the vehicle is displayed at a position further forward than the vehicle's grille. In the rear view, an image showing the exterior of the vehicle is displayed at a position further back than the vehicle's rear doors. In the top view (surround view), a top view image is displayed, showing the vehicle from a virtual viewpoint directly above it. In the top view, a two-dimensional image is displayed, where the vehicle image is represented by a plan view of the vehicle; when there are three-dimensional objects of equal height around the vehicle, an image showing the objects tilted is displayed. The 3D view displays an image of the object as viewed from a virtual viewpoint, using the area surrounding the object as a virtual viewpoint. The object is not limited to vehicles; it can also be obstacles around the vehicle. The virtual viewpoint is a simulated viewpoint from the outside of the object, observed from its surroundings. The virtual viewpoint can be moved. For example, when a 3D image (viewing a vehicle from a virtual viewpoint) is displayed on monitors 41 and 42, the 3D-displayed vehicle image can be rotated by moving the virtual viewpoint around the vehicle along a plane. Additionally, monitors 41 and 42 can display the vehicle's perimeter using perspective, front wide-angle, and rear wide-angle views, among other display modes. In the perspective view, in addition to the current camera image (real image) showing the vehicle's perimeter, a virtual image showing the state under the vehicle's floor is also displayed. The front wide-angle view, for example, displays an image of the area in front of the vehicle that is the driver's blind spot, based on the image from the front camera 21 with a wide-angle lens, when the vehicle is making a left or right turn at a poorly visible intersection such as a T-junction.

[0025] The display mode can be appropriately changed according to the user's specifications or the vehicle's driving status via the vehicle system. For example, when the vehicle is in reverse gear to park, the display mode can automatically switch to a top view or a rear view. Alternatively, the display mode can be switched by the driver operating a switch or other means while the vehicle is in motion.

[0026] The image processing unit 13 can display images in multiple display modes on the display 41 through multiple screens. Furthermore, the image processing unit 13 can display images in different display modes on both the display 41 and the display 42, or it can display images in the same display mode on both displays 41 and 42. Additionally, the image processing unit 13 can display images in multiple display modes as a single screen on both displays 41 and 42, or it can display images in a single display mode. Furthermore, the following description focuses on the image processing performed by the image processing unit 13 when displaying an image on the display 41; however, the image processing performed on the display 41 can also be applied to the image display on the display 42 when displaying an image on the display 42.

[0027] The image processing unit 13 generates a display image corresponding to the display mode based on multiple camera images acquired by the image acquisition unit 11, and outputs the display image to the display 41. When displaying an image through a front view or a rear view, the image processing unit 13 generates a display image that displays the front or rear of the vehicle based on the camera images of the front camera 21 or the rear camera 24.

[0028] When displaying an image from a top-down view, the image processing unit 13 generates a top-down view image of the vehicle based on multiple camera images captured by the camera 20. The virtual viewpoint of the top-down view image is a predetermined position located outside the vehicle, extending vertically away from the roof of the vehicle. For example, the image processing unit 13 generates a viewpoint transformation image by performing viewpoint transformation processing on images captured by the front camera 21, right camera 22, left camera 23, and rear camera 24, allowing the vehicle to be viewed from the virtual viewpoint's front-rear and left-right areas. In other words, the image processing unit 13 transforms camera images corresponding to those viewed from the camera's position into viewpoint transformation images viewed from the virtual viewpoint. The viewpoint transformation from the camera position to the virtual viewpoint is performed using viewpoint parameters, which are stored in the recording medium 19. The image processing unit 13 adapts each viewpoint-transformed image to the surrounding area of ​​the vehicle. The image processing unit 13 performs fusion processing on the superimposed portions of the camera images from the camera 20. The fusion process is used to maintain continuity between two superimposed images, and methods known at the time of this application can be applied. Furthermore, the fused image becomes a surrounding image representing the area around the vehicle. Additionally, the image processing unit 13 composites the vehicle image located in the center of the screen with the surrounding image. The composite image becomes a top-view image, and the image processing unit 13 outputs the top-view image as a display image to the display 41.

[0029] When an image is displayed in a 3D view, the image processing unit 13 generates a display image of the vehicle viewed from a virtual viewpoint based on multiple camera images captured by the camera 20. First, the image processing unit 13 sets the virtual viewpoint according to user specifications or driving conditions. The position of the virtual viewpoint and the direction of the line of sight can be arbitrarily set based on user instructions. For example, the user can specify the position of the virtual viewpoint or the direction of the line of sight through touch operation on the display 41. Furthermore, the image processing unit 13 can set the virtual viewpoint according to driving conditions. For example, when moving forward, a virtual viewpoint can be set behind the vehicle to observe the image in front of it; when reversing, a virtual viewpoint can be set in front of the vehicle to observe the image behind it. Specifically, the image processing unit 13 sets the position of the virtual viewpoint upon receiving shift information indicating that the gear shift position has entered reverse gear. Furthermore, the image processing unit 13 can also set the virtual viewpoint according to vehicle speed. For example, the virtual viewpoint can be set according to the vehicle speed input from the vehicle speed sensor 50: when the vehicle speed is greater than or equal to a specified value, the position of the virtual viewpoint is far away from the vehicle, and when the vehicle speed is less than a specified value, the position of the virtual viewpoint is close to the vehicle.

[0030] The image processing unit 13 deforms at least a portion of a predefined reference surface coordinate system, which is used to project images from a camera, based on the position of the virtual viewpoint. The reference surface coordinate system is stored in the recording medium 19. For example, the reference surface coordinate system is shaped like a bowl surrounding the vehicle. The reference surface coordinate system has a center on the xy-plane parallel to the vehicle's mounting surface (driving surface), and is formed by a surface that has curvature from or near the center, having a component in the vehicle's height direction (z-direction).

[0031] Near the center of the reference surface coordinate system, a base surface parallel to the xy plane can be formed, and its shape can be any shape such as rectangle, triangle, ellipse, or circle. Furthermore, the curvature of the surfaces in the reference surface coordinate system does not have to be uniform. For example, the curvature of the surfaces near the base surface can be relatively large, while the curvature of the surfaces far from the base surface can be relatively small. In addition, the image processing unit 13 can pre-store multiple reference surface coordinate systems with different shapes according to the shape (size, shape) of the vehicle. When multiple reference surface coordinate systems are stored, the used reference surface coordinate system is stored in association with the vehicle speed, the vehicle's gear shift position, etc., on the recording medium 19, and the image processing unit 13 can select the reference surface coordinate system according to the vehicle speed, the vehicle's gear shift position, etc.

[0032] The image processing unit 13 can also read a pre-stored reference surface coordinate system and deform the shape of the read reference surface coordinate system. The image processing unit 13 can also deform the shape of the reference surface coordinate system according to the position of the virtual viewpoint. For example, if the virtual viewpoint is located outside the outer edge of the reference surface coordinate system, a portion of the reference surface coordinate system can be deformed so that the virtual viewpoint is located further inside the outer edge of the reference surface coordinate system. By deforming a portion of the reference surface coordinate system so that the virtual viewpoint is located further inside the outer edge of the reference surface coordinate system, and thus displaying obstacles with height on the screen, distortion of the obstacle's shape can be suppressed, and an image without image defects can be displayed on the display 41. Furthermore, other coordinate system deformation methods known at the time of application can be used for the deformation of the reference surface coordinate system.

[0033] The image processing unit 13 projects multiple camera images acquired from the camera 20 onto a reference surface coordinate system. Furthermore, if the reference surface coordinate system is deformed, the image processing unit 13 only needs to project the camera images onto the deformed surface coordinate system (deformable coordinate system). The image processing unit 13 has an image transformation table that associates the coordinates of the pixels contained in the camera images with the coordinates of the coordinate system, so as to project the camera image data onto the reference surface coordinate system or the deformed coordinate system. The image transformation table is recorded on the recording medium 19.

[0034] Alternatively, a pre-prepared vehicle image can be superimposed on a reference surface coordinate system. The vehicle image can be pre-created based on the vehicle's appearance and stored in the recording medium 19. By superimposing the vehicle image on the reference surface coordinate system, not only can the image surrounding the vehicle be displayed in three dimensions, but the vehicle itself can also be displayed in three dimensions, which helps in understanding the relationship between the vehicle's position and orientation and the surrounding images. As described above, the image processing unit 13, referring to the image transformation table, establishes a relationship between the coordinates of the pixels contained in the camera image and the coordinates of the coordinate system, projects the camera image onto the reference surface coordinate system or the deformed coordinate system, and superimposes the vehicle image onto the projected three-dimensional image. This generates a display image (three-dimensional image) in a three-dimensional view of the vehicle and its surroundings from a virtual viewpoint.

[0035] The image processing unit 13 is not limited to generating display images from virtual viewpoints set according to user specifications or driving conditions, but can also generate display images from selectable virtual viewpoints. For example, when a display image is shown on the monitor 41 in a three-dimensional view, the user can freely move the position of the virtual viewpoint through touch operation on the monitor 41. When a display in a three-dimensional view is selected, the image processing unit 13 generates multiple three-dimensional view display images for multiple virtual viewpoints selectable by the user. Furthermore, for example, when the user moves the virtual viewpoint through touch operation on the monitor 41, a three-dimensional image of the vehicle is displayed on the monitor 41 rotated around the center point of a surface coordinate system. In addition, the image processing unit 13 is not limited to multiple virtual viewpoints selectable by the user, but can also set virtual viewpoints at any position around the vehicle and generate multiple three-dimensional view display images for each virtual viewpoint. That is, when the image processing unit 13 displays a three-dimensional image on the monitor 41 in a rotated display manner, it generates a continuous three-dimensional image so that smooth image display is possible when moving relative to the virtual viewpoint.

[0036] Furthermore, the image processing methods for the displayed images in various display modes such as front / rear view, top view, and three-dimensional view are not limited to the methods described above, and other image processing methods can also be used. In addition, the image processing methods for the displayed images in other display modes such as perspective view, front wide-angle view / rear wide-angle view, etc., can appropriately use methods known at the time of application.

[0037] Next, a specific example of the display control implemented by the display control device 100 and the display screen of the display 41 will be described together. Figure 3 The display screen 41 shows the screen displayed. Figure 3In the example, when the vehicle is parking on a straight road, the left side of the display 41 shows a top-down view image, and the right side of the display 41 shows a three-dimensional view image. When the controller 10 displays the top-down view image and the three-dimensional view image on the display 41, it generates a top-down view image 46 by overlaying viewpoint icons 43a-43h and an automatic display start icon 44 onto the top-down view image of the vehicle. The top-down view image 46 includes a top-down view of the vehicle from above, viewpoint icons 43a-h, and an automatic display start icon 44. Viewpoint icons 43a-h are icons indicating the position of a virtual viewpoint and / or the direction of the line of sight, and are selectable by the user. Multiple viewpoint icons 43a-43h are arranged around the vehicle image 45, centered on the vehicle image 45. Viewpoint icon 43a is positioned above the vehicle image located at the center of the top-down view image, while viewpoint icons 43c, 43e, and 43g are positioned to the right, bottom, and left of the vehicle image, respectively. Additionally, viewpoint icons 43b, 43d, 43f, and 43h are positioned relative to the vehicle image in the upper right, lower right, lower left, and lower right directions, respectively. If the user selects viewpoint icons 43a to 43h, a display image (3D image) of the vehicle viewed from the selected viewpoint icon is displayed on the right side of the display 41.

[0038] like Figure 3 As shown, if the user touches the viewpoint icon 43a, the controller 10 causes the viewpoint icon 43a to be displayed as a selection icon. The controller 10 displays the selection icon, for example, by changing the color of the viewpoint icon 43a. Figure 3 In the example, the selection icon is represented by setting the outer perimeter of the viewpoint icon 43a to a thick line. The image processing unit 13 sets a virtual viewpoint corresponding to the viewpoint icon 43a. Thus, the virtual viewpoint is set according to the user's specification. Figure 3 In this example, the position when viewing the vehicle from the front is set as a virtual viewpoint. Furthermore, the image processing unit 13 generates a three-dimensional image 47 of the vehicle and its surroundings when viewed from the front, based on multiple camera images. The image processing unit 13 outputs the three-dimensional image as a display image to the display 41. The three-dimensional image 47 is displayed on the right side of the display 41. And, as... Figure 3 As shown, the display 41 displays a top-view image 46 and a three-dimensional image 47 arranged in a row.

[0039] The auto-display start icon 44 is an icon used in the 3D view to initiate automatic switching of the 3D image by moving the virtual viewpoint. The auto-display start icon 44 is displayed at the center of the vehicle image 45 contained in the top-view image 46. If auto-display is performed, predetermined positions around the vehicle become the start and end points of the virtual viewpoint. The virtual viewpoint moves from the start to the end point, and the vehicle image displayed in the 3D view rotates 360 degrees along a plane parallel to the road surface. In other words, during auto-display, a center point is set on the vehicle image, and the vehicle image and the images surrounding the vehicle rotate 360 ​​degrees around this center point. For example, in... Figure 3 In the example, if the user touches the automatically displayed start icon 44, the selected viewpoint icon 43a becomes the start and end point of the virtual viewpoint. The virtual viewpoint moves sequentially from viewpoint icon 43b to viewpoint icon 43h, rotating clockwise around the vehicle, starting from the position corresponding to viewpoint icon 43a, and returning to the position corresponding to the initial viewpoint icon 43a. Accompanying the rotation and movement of the virtual viewpoint, the 3D image 47 in the 3D view rotates. The orientation of the vehicle image contained in the 3D image 47 is as follows: Figure 3 Starting from the front (front view), the display changes in the order of right side, back (rear view), and left side, returning to the initial front (front view). That is, the 3D image becomes a 360-degree rotated display (hereinafter also referred to as "360-degree rotated display"). Thus, in the 3D view, the displayed image (3D image 47) is automatically rotated 360 degrees.

[0040] Figure 4 The display screen 41 shows the screen displayed. Figure 4 In the example, when the vehicle is parking on a T-shaped road, the left side of display 41 shows a top-view image, and the right side of display 41 shows a three-dimensional view image. The image processing methods for the top-view image 46 and the three-dimensional image 47 are the same as those for the top-view image 46 and the three-dimensional image 47. Figure 3 The example is the same; the top-view image 46 also includes viewpoint icons 43a~h and the automatically displayed start icon 44. (Similar to...) Figure 3 The difference in the example is that viewpoint icon 43e is selected, displaying a 3D image 45 of the vehicle and its surroundings as viewed from behind in the 3D view. Furthermore, as... Figure 4 As shown, when the vehicle is parked on a T-shaped road, the controller 10 can set a virtual viewpoint corresponding to the viewpoint icon 43a and display a 3D image 47 viewed from the virtual viewpoint on the display 41. Therefore, when there is a blind spot in front of the vehicle that is a blind spot for the driver, the status of the blind spot area can be displayed on the display 41.

[0041] In addition, such as Figure 3 and Figure 4As shown, when the vehicle is parked, the user can select viewpoint icons 43a-43h and confirm the 3D image 47 through the 3D view of the display 41 from the direction of the selected viewpoint icon 43a-43h. For example, if the user touches the position of viewpoint icon 43c on the screen of the display 41, the image processing unit 13 switches viewpoint icon 43c to a selection icon, so that the user can view the 3D image 47 from the current virtual viewpoint (in the viewpoint icon 43a-43h). Figure 3 In the example, the virtual viewpoint (corresponding to viewpoint icon 43a) moves to the virtual viewpoint corresponding to viewpoint icon 43c. Furthermore, the image processing unit 13 generates a 3D image from the virtual viewpoint based on the camera image and outputs the image to the display 41, so that the 3D image is displayed through a 3D view. The display 41 shows a 3D image of the vehicle viewed from the right side.

[0042] Additionally, when the vehicle is parked, the display control device 100 can perform display control, allowing the user to not only select a virtual viewpoint through touch operations of viewpoint icons 43a-43h, but also arbitrarily specify the position of the virtual viewpoint. The user touches any position on the display screen of the 3D view. Furthermore, while the finger is touching the screen, the user moves their finger in any direction (sliding operation). During the period when the finger is touching the screen, the position of the virtual viewpoint moves in accordance with the sliding operation, and the 3D image rotates in accordance with the movement of the virtual viewpoint. The amount of movement of the virtual viewpoint is determined by the amount of sliding (equivalent to the amount of finger movement while touching the screen); the larger the amount of sliding, the larger the amount of movement. Moreover, the amount of movement of the virtual viewpoint is equivalent to the length of the arc of a circle centered on the vehicle image. For example, the user touches the display screen of the 3D view with their finger, moving their finger with a large amount of sliding. In the case of a large amount of sliding, the image processing unit 13 rotates the display of the 3D image 47 so that the virtual viewpoint moves around the center point of the vehicle image. When the sliding amount is small, the image processing unit 13 rotates the 3D image 47 by moving the virtual viewpoint along an arc centered on the center point of the vehicle image with a movement amount equivalent to the sliding amount. Furthermore, the rotation display achieved by the sliding operation is not limited to being performed through a sliding operation on the display screen of the 3D view, but can also be performed through a sliding operation on the display screen of the top view. For example, the image processing unit 13, in situations such as... Figure 3 or Figure 4 In the top-view image 46 shown, the image with viewpoint icons 43a-43h removed is displayed on the monitor 41. Furthermore, the user can perform a sliding operation at any position on the screen displaying the top-view image 46. Based on the sliding operation on the top-view display screen, the image processing unit 13 rotates and displays the three-dimensional image on the three-dimensional view display screen.

[0043] Figure 5 This is a diagram used to illustrate the rotational display of the 3D image 47 in a 3D view. Figure 5 (a) Shows a 3D image 47 as viewed from a virtual viewpoint corresponding to viewpoint icon 43e. Figure 5 (b) Shows a 3D image 47 as viewed from a virtual viewpoint corresponding to viewpoint icon 43f. Figure 5 (c) A three-dimensional image 47 is shown when viewed from a virtual viewpoint corresponding to viewpoint icon 43g. Furthermore, Figure 5 This is a display screen of the three-dimensional view on the display screen 41, showing only the vehicle image contained in the three-dimensional image 47, omitting the illustration of the surrounding image of the vehicle. In addition, the virtual viewpoint that becomes the starting point of the rotation display is the position corresponding to the viewpoint icon 43e.

[0044] like Figure 5 As shown in (a), the vehicle image contained in 3D image 47 facing backward (rear) becomes the starting point for the rotation display. With the movement of the virtual viewpoint, if the vehicle image rotates 45 degrees, it becomes as shown... Figure 5 The state shown in (b). If rotated further, the vehicle image becomes as shown. Figure 5 The state shown in (c). Up to this point, the 3D image 47 has rotated 90 degrees, and then the 3D image 47 rotates the remaining 270 degrees. That is, the orientation of the vehicle image contained in the 3D image 47 changes from the rear (back) position to the left side, front (front) and right side in that order, returning to the initial rear (back) position. In addition, the surrounding images located around the vehicle image also rotate along with the vehicle image. Thus, in the 3D view, the displayed image (3D image 47) is automatically rotated 360 degrees.

[0045] In this embodiment, the virtual viewpoint is moved and the displayed image is automatically switched at a timing point at least once before the vehicle starts moving and once it has finished parking. The automatic display control performed before the vehicle starts moving and once it has finished parking will be described below.

[0046] First, the control content automatically displayed before vehicle start-up will be explained. The controller 10 determines whether it is before vehicle start-up based on vehicle data acquired by the vehicle data acquisition unit 12. Before vehicle start-up is a state where the vehicle is capable of driving and has stopped (vehicle speed less than or equal to a predetermined threshold). The controller 10 detects the state of the vehicle's main switch based on the vehicle data. The vehicle's main switch is a switch that toggles whether the vehicle is in a drivable state. For vehicles with an engine, the ignition switch is equivalent to the main switch. The vehicle's main switch is also called the power switch or starter switch. When the controller 10 detects that the main switch is in the on state, it determines that it is before vehicle start-up.

[0047] Furthermore, the controller 10 can also determine that the vehicle is about to start when it detects the main switch switching from off to on. The controller 10 can also determine that the vehicle is about to start when the main switch is on and the gear position is not reverse. Positions other than reverse include, for example, parking, neutral, and all drive gears. Additionally, the controller 10 can also determine that the vehicle is about to start when the main switch is on and the vehicle speed is less than or equal to a predetermined speed threshold. Furthermore, the "vehicle starting" condition can be set as, for example, a time elapsed before the automatic switch switches from off to on (e.g., tens of seconds or minutes).

[0048] If it is determined that the vehicle has started moving, the image processing unit 13 performs a 360-degree automatic rotation display of the displayed image (3D image 47) in the 3D view. Figure 6 The display screen 41 shows the screen displayed. Figure 6 The example illustrates a temporary parking situation where the vehicle is temporarily stopped in order to park forward in the surrounding parking space. For example... Figure 6 As shown, a top-view image 46 is displayed on the left side of the display 41, and a three-dimensional image 47 is displayed on the right side. When the vehicle is in a position other than reverse before starting, the image processing unit 13 sets the rear of the vehicle as the start and end points of the virtual viewpoint. Figure 6 In the example, the position corresponding to viewpoint icon 43e becomes the start and end point of the virtual viewpoint. The start and end points of the virtual viewpoint are the starting and ending points when the 3D image 47 is rotated 360 degrees for display. The image processing unit 13 moves the position of the virtual viewpoint from the position corresponding to viewpoint icon 43e, in the order of viewpoint icons 43f-43h and 43a-43e, to the positions corresponding to each icon. That is, the movement of the virtual viewpoint is as follows: Figure 6 As indicated by the dashed arrow, in the top view image 46, the image rotates clockwise one full revolution from the viewpoint icon 43e. Furthermore, when rotating the display, the image processing unit 13 can also... Figure 6 The guide image, indicated by the dashed arrow representing the direction of rotation, is superimposed on the top-view image 46 and displayed on the top-view screen. In the three-dimensional view, the image processing unit 13 generates a three-dimensional image 47 by rotating it 360 degrees in accordance with the movement of the virtual viewpoint, and outputs it to the display 41. When the virtual viewpoint is at the position corresponding to the viewpoint icon 43e, the image processing unit 13 generates... Figure 6 As shown, the vehicle is displayed rearward on a three-dimensional image on display 41. Thus, before starting the vehicle, the user can automatically rotate the three-dimensional image 47 360 degrees to check the surroundings of the vehicle in a three-dimensional view.

[0049] Furthermore, when the 3D image 47 is automatically rotated 360 degrees, the image processing unit 13 can also display selection icons on the top-view screen in accordance with the movement of the virtual viewpoint. First, the starting point of the virtual viewpoint is the position corresponding to viewpoint icon 43e; therefore, the image processing unit 13 displays viewpoint icon 43e as a selection icon. If the 3D image 47 rotates 45 degrees, the virtual viewpoint moves from the position corresponding to viewpoint icon 43e to the position corresponding to viewpoint icon 43f; therefore, the image processing unit 13 sets viewpoint icon 43e to the normal icon display and displays viewpoint icon 43f as a selection icon. Moreover, the image processing unit 13 moves the selection icons in the order of viewpoint icons 43g-43h and 43a-43e in accordance with the 45-degree rotation of the 3D image 47. That is, when the vehicle is about to start moving and the gear shift position is not reverse, the image processing unit 13, in the top view, makes the selection icon start from the viewpoint icon 43e located behind the vehicle image 44. Furthermore, as... Figure 6 As shown, the image processing unit 13 rotates the display of the selection icon 360 degrees with the vehicle image 44 as the center. This allows the user to confirm the position of the virtual viewpoint in the top-down view when the 3D image 47 is displayed with automatic 360-degree rotation.

[0050] Next, the control content automatically displayed after parking is completed will be explained. The controller 10 determines whether parking has been completed based on vehicle data acquired by the vehicle data acquisition unit 12. Parking completion occurs when the vehicle cannot move and has stopped (vehicle speed less than or equal to a predetermined threshold). The controller 10 detects the state of the vehicle's main switch and / or gear shift position based on the vehicle data. The controller 10 determines that parking has begun when the main switch is in the off state.

[0051] Furthermore, the controller 10 can also determine that parking is complete when it detects the main switch changing from on to off. The controller 10 can also determine that parking is complete when it detects the gear shift position changing from a position other than parking to parking. Additionally, the controller 10 can obtain information indicating the on / off state of the electric parking brake from vehicle data, and determine that parking is complete when the electric parking brake becomes on. Furthermore, parking completion can be defined, for example, as the time elapsed before the time when the gear shift position changes from a position other than parking to parking, the time when the main switch changes from on to off, or the time when the electric parking brake changes from off to on, based on a predetermined time (e.g., tens of seconds or minutes).

[0052] Once parking is deemed complete, the image processing unit 13 performs a 360-degree automatic rotation display of the displayed image (3D image 47) in the 3D view. Figure 7 The display screen 41 shows the screen displayed. Figure 7 In the example shown, the gear shift position is reverse, and the vehicle is in a temporarily stopped or parked state. For example... Figure 7 As shown, a top-view image 46 is displayed on the left screen of the display 41, and a three-dimensional image 47 is displayed on the right screen. When the vehicle is parked and the gear is shifted to reverse, the image processing unit 13 sets the position in front of the vehicle as the start and end points of the virtual viewpoint. Figure 7 In the example, the position corresponding to viewpoint icon 43a becomes the start and end point of the virtual viewpoint. The start and end points of the virtual viewpoint are the starting and ending points when the 3D image 47 is rotated 360 degrees for display. The image processing unit 13 moves the position of the virtual viewpoint from the position corresponding to viewpoint icon 43a, in the order of viewpoint icons 43b to 43h, 43a, to the positions corresponding to each icon. That is, the movement of the virtual viewpoint is as follows: Figure 7 As indicated by the dashed arrow, in the top view image 46, the image rotates clockwise one full revolution from viewpoint icon 43a. Furthermore, the image processing unit 13 can also, during rotation display, [perform a function such as...]. Figure 7 The dotted arrow indicating the direction of rotation guides the image superimposed on the top-view image 46 and displayed on the top-view screen. In the three-dimensional view, the image processing unit 13 generates a three-dimensional image 47 that rotates 360 degrees in accordance with the movement of the virtual viewpoint and outputs it to the display 41. When the virtual viewpoint is at the position corresponding to the viewpoint icon 43a, the image processing unit 13... Figure 7 As shown, a 3D image of the vehicle facing forward is generated and displayed on the monitor 41. This allows the user to check the surroundings of the vehicle in a 3D view after parking by automatically rotating the 3D image 47 360 degrees.

[0053] Next, refer to Figure 8 The display control method implemented by controller 10 will be described. Figure 8 This is a flowchart of the display control method implemented by controller 10. During the process of displaying the image on the left side of display 41 in a top view and the image on the right side in a three-dimensional view, controller 10 repeatedly executes the following control flow.

[0054] In step S1, the image acquisition unit 11 acquires a camera image from the camera 20. The vehicle data acquisition unit 12 acquires vehicle data from the ATCU 30 and / or the vehicle speed sensor 50, etc. In step S2, the image processing unit 13 detects the current vehicle state based on the vehicle data. The vehicle state is detected by determining the timing of at least one of the following: before the vehicle starts and after parking is completed. For example, it is detected based on the state of the vehicle's main switch, gear shift position, vehicle speed, etc.

[0055] In step S3, the image processing unit 13 generates a top-view image of the vehicle from above, based on multiple camera images. In step S4, the image processing unit 13 generates a three-dimensional image of the vehicle from a virtual viewpoint, based on multiple camera images. In step S5, the controller 10 determines whether the vehicle is about to start moving. If it is determined that the vehicle is about to start moving, in step S6, the controller 10 sets the rear position of the vehicle as the start and end point of the virtual viewpoint. The control flow proceeds to step S9.

[0056] Returning to step S5, if it is determined that the vehicle is not starting, in step S7, the controller 10 determines whether parking has been completed. If parking has been completed, in step S8, the controller 10 sets the position in front of the vehicle as the start and end point of the virtual viewpoint. The control flow then proceeds to step S9.

[0057] In step S9, the controller 10 outputs the top view image and the three-dimensional image to the display 41. By moving the virtual viewpoint from the starting point to the ending point, the three-dimensional image is automatically switched, thereby performing a 360-degree automatic rotation display.

[0058] Returning to step S7, if the controller 10 determines in step S7 that the timing is not after parking is completed, in step S10, the controller 10 performs a normal display. In the normal display, the controller 10 outputs a top-view image and a top-view image to the display 41, for example, displaying them from a user-specified virtual viewpoint. After executing the control flow of steps S9 and S10, the control flow ends. The controller 10 restarts the control flow from step S1.

[0059] As described above, in this embodiment, the controller 10 acquires camera images from the camera 20, generates a display image based on multiple camera images, and outputs the display image to the display 41. Furthermore, the controller 10 automatically displays the image by moving the virtual viewpoint and switching the display image at a timing that occurs at least once before the vehicle starts moving and after parking. Therefore, the display 41 can be configured to display images corresponding to the vehicle's state, providing the user with an appropriate display image that matches the vehicle's status. As a result, highly operable display control of the display 41 is achieved.

[0060] In this embodiment, automatic display involves setting predetermined locations around the vehicle as the start and end points of a virtual viewpoint. The virtual viewpoint moves from the start to the end point, causing the displayed image to rotate 360 ​​degrees. When the vehicle is in a drivable state and the gear shift is not in reverse, the controller 10 determines that the vehicle is about to start and sets the rear of the vehicle as the start and end points of the virtual viewpoint. Conversely, when the vehicle's main power switch is off, the controller 10 determines that parking has been completed and sets the front of the vehicle as the start and end points of the virtual viewpoint. This allows for the timely provision of images to passengers whenever they wish to check the surroundings of the vehicle.

[0061] In this embodiment, the controller 10 outputs a top-view image 46, which includes viewpoint icons 43a-43h indicating the position of a virtual viewpoint, to the display 41 when viewing the vehicle from above. By selecting an icon, the controller 10 displays the viewpoint icon 43a-43h that corresponds to the virtual viewpoint of the currently displayed image on the display 41, from among the multiple viewpoint icons 43a-43h arranged around the vehicle image 44 included in the top-view image 46. Before the vehicle starts moving, the controller 10 selects an icon in the top-view image starting from the viewpoint icon located behind the vehicle image 44. After parking is completed, the controller 10 selects an icon in the top-view image starting from the viewpoint icons 43a-43h located in front of the vehicle image 44. Thus, it is possible to confirm from which position the three-dimensional image displayed in the three-dimensional view was viewed in the top-view image.

[0062] Furthermore, in this embodiment, the selection icon moves around the vehicle image 44 in accordance with the automatic display of the 3D image 47. This allows the movement of the virtual viewpoint to be confirmed in the top-down view.

[0063] Furthermore, in this embodiment, the controller 10 can automatically display the three-dimensional image 47 at a timed time either before the vehicle starts or after parking, at a timed timed only before the vehicle starts, or at a timed timed only after parking.

[0064] Furthermore, in this embodiment, the automatic display of the 3D image 47 does not necessarily have to be a 360-degree rotation display; for example, it can be a 90-degree, 135-degree, or 180-degree rotation display. For example, when the controller 10 times the vehicle before it starts moving, in order to display the surrounding area in front of the vehicle, it can move the position of the virtual viewpoint from the position corresponding to viewpoint icon 43g, in the order of viewpoint icons 43h, 43a to 43c, to the positions corresponding to each icon, performing a rotation display where the virtual viewpoint rotates 180 degrees in front of the vehicle. Similarly, when the controller 10 times the vehicle is parked, in order to display the surrounding area behind the vehicle, it can move the position of the virtual viewpoint from the position corresponding to viewpoint icon 43c, in the order of viewpoint icons 43d to 43g, to the positions corresponding to each icon, performing a rotation display where the virtual viewpoint rotates 180 degrees in front of the vehicle.

[0065] Furthermore, in this embodiment, the controller 10 does not need to perform... Figure 8 The complete control flow shown is optional; otherwise, it may not be followed. Figure 8 The control processes are executed in the order shown.

[0066] Explanation of the label

[0067] 1 Display Control System

[0068] 10 Controllers

[0069] 11 Image Acquisition Unit

[0070] 12 Vehicle Data Acquisition Department

[0071] 13 Image Processing Department

[0072] 19 Recording media

[0073] 20 cameras

[0074] 40. In-vehicle displays

[0075] Viewpoint icons 43a~43h

[0076] 44. Automatically display the Start icon

[0077] 45 Three-dimensional images

[0078] 46 Top view image

[0079] 50 vehicle speed sensor

[0080] 100 Display control device

Claims

1. A display control device comprising a controller that acquires camera images from a plurality of vehicle-mounted cameras capturing images of the area around a vehicle, generates a display image based on the plurality of said camera images when viewing the vehicle from a virtual viewpoint, and outputs the display image to a display screen, wherein, The controller, at a timed interval at least once before the vehicle starts moving and after parking, performs an automatic switch to display the image by moving the virtual viewpoint.

2. The display control device according to claim 1, wherein, The automatic display involves setting predetermined positions around the vehicle as the start and end points of the virtual viewpoint, moving the virtual viewpoint from the start point to the end point, and rotating the displayed image 360 ​​degrees. When the vehicle is in a drivable state and the gear shift position is not in reverse, the controller determines that the vehicle is about to start and sets the rear position of the vehicle as the start and end point of the virtual viewpoint. When the vehicle's main switch is turned off, it is determined that the parking process is complete, and the position in front of the vehicle is set as the start and end point of the virtual viewpoint.

3. The display control device according to claim 1 or 2, wherein, The controller outputs a top-down view image of the vehicle viewed from above to the display, the top-down view image including viewpoint icons indicating the location of the virtual viewpoint. By selecting an icon, the viewpoint icon corresponding to the virtual viewpoint of the currently displayed image on the monitor is displayed, from among the plurality of viewpoint icons arranged around the vehicle image contained in the top-view image. In the case of the vehicle before it starts moving, in the top view, the selection icon starts from the viewpoint icon located behind the vehicle image. Once parking is complete, in the top view, the selection icon starts from the viewpoint icon located in front of the vehicle image.

4. The display control device according to claim 3, wherein, The selection icon moves around the vehicle image in a manner that matches the automatic display.

5. A display control method, executed by a controller, for controlling the display of a monitor. In this display control method, The controller acquires camera images from multiple onboard cameras that capture images of the vehicle's surroundings. Based on multiple camera images, a display image is generated when viewing the vehicle from a virtual viewpoint. The displayed image is output to the monitor. At a timed interval, either before the vehicle starts moving or after it has finished parking, the virtual viewpoint is moved, and the displayed image is automatically switched.

6. A recording medium that records a control program executed by a controller, In this recording medium, The control program causes the controller to perform a process comprising the following steps: Images are obtained from multiple onboard cameras that take pictures around the vehicle; Based on multiple camera images, a display image is generated when viewing the vehicle from a virtual viewpoint; The displayed image is output to the monitor; as well as At a timed interval, either before the vehicle starts moving or after it has finished parking, the virtual viewpoint is moved, and the displayed image is automatically switched.

Images