Display control device, display control method, and recording medium

Abstract

A display control device has a controller (10) that acquires camera images from multiple onboard cameras that capture images of the vehicle's surroundings, generates a display image of the vehicle viewed from a virtual viewpoint based on the multiple camera images, and outputs the display image to a display (41). The controller (10) moves the position of the virtual viewpoint through user operation, detects the vehicle's speed, and restricts the movement of the virtual viewpoint by user operation based on the detected vehicle speed.

Description

Technical Field

[0001] This invention relates to a display control device, a display control method, and a recording medium. Background Technology

[0002] Conventionally, display control devices that control the display of images representing the surroundings of a vehicle are known (e.g., Patent Document 1). The display control device described in Patent Document 1 includes: an image generation unit that generates a three-dimensional image representing the surroundings of the vehicle based on images captured by multiple vehicle-mounted cameras that capture images of the surroundings of the vehicle, and outputs a display image displayed on a display device based on the three-dimensional image; and a viewpoint changing unit that, when a user slides in the three-dimensional image displayed on the display device, changes viewpoint parameters based on the position of the slide and either the amount or speed of the slide.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2023-156871 Summary of the Invention

[0004] The display control device described in Patent Document 1 has the following problem: because it can display images that do not correspond to the state of the vehicle, it is difficult for users to operate.

[0005] The problem this invention aims to solve is to provide a display control device, display control method, and recording medium that can improve operability.

[0006] The present invention solves the above-mentioned problems by generating a display image of the vehicle viewed from a virtual viewpoint based on camera images, outputting the display image to a display screen, and restricting the movement of the virtual viewpoint operated by the user according to the vehicle speed.

[0007] The effects of the invention

[0008] According to the present invention, operability can be improved. Attached Figure Description

[0009] Figure 1 This is a block diagram illustrating the structure of the display control system according to this embodiment.

[0010] Figure 2 It is a floor plan showing a portion of the dashboard.

[0011] Figure 3 It is a diagram representing the screen displayed on the monitor.

[0012] Figure 4 It is a diagram representing the screen displayed on the monitor.

[0013] Figure 5It is a diagram that represents a three-dimensional view of the display screen.

[0014] Figure 6 This is a flowchart of the display control method executed by the controller.

[0015] Figure 7 It is a diagram representing the screen displayed on the monitor.

[0016] Figure 8 It is a diagram representing the screen displayed on the monitor.

[0017] Figure 9 It is a diagram representing the screen displayed on the monitor.

[0018] Figure 10 It is a diagram representing the screen displayed on the monitor.

[0019] Figure 11 It is a diagram representing the screen displayed on the monitor. Detailed Implementation

[0020] Hereinafter, with reference to the accompanying drawings, a display control system including a display control device according to an embodiment of the present invention will be described. The display control system is a system that generates an image of the surroundings of a vehicle. Figure 1 This is a block diagram illustrating the structure of the display control system according to this embodiment. The display control system 1 includes a front camera 21, a right camera 22, a left camera 23, a rear camera 24, an ATCU 30, a display 41, a display 42, a vehicle speed sensor 50, and a display control device 100. The display control system 1 is installed in a vehicle. The front camera 21, right camera 22, left camera 23, rear camera 24, ATCU 30, vehicle display 40, vehicle speed sensor 50, and display control device 100 are connected via an in-vehicle communication network such as CAN or LIN.

[0021] A front camera 21, a right camera 22, a left camera 23, and a rear camera 24 are mounted on the vehicle to photograph the area around the vehicle. The front camera 21 photographs the front of the vehicle, the right camera 22 photographs the right side of the vehicle, the left camera 23 photographs the left side of the vehicle, and the rear camera 24 photographs the rear of the vehicle. The front camera 21 is mounted on the vehicle's grille, the right camera 22 is mounted on the right rearview mirror, the left camera 23 is mounted on the left rearview mirror, and the rear camera 24 is mounted on the rear door. Alternatively, the cameras can be mounted in other locations on the vehicle. Furthermore, in the following description, when referring to the front camera 21, right camera 22, left camera 23, and rear camera 24 collectively, they are referred to as camera 20.

[0022] The ATCU (Automatic Transmission Control Unit) 30 is a control device that controls the transmission. The ATCU 30 outputs control signals, including shift information, to the display control device 100. The shift information indicates the position of the gear shift lever, showing the current state of the gears such as forward, neutral, reverse, and park.

[0023] The in-vehicle display 40 is a display included in the IVI (in-vehicle infotainment) system, comprising display 41 and display 42. Display 41 and display 42 are located on the vehicle's dashboard. Display 41 and display 42 are touchscreen displays. Figure 2 This is a floor plan showing a portion of the dashboard. For example... Figure 2 As shown, display 41 is located in the center of the instrument panel. Display 42 is located in the instrument panel, positioned in front of the driver. Display 41 displays, for example, maps from the navigation system, menu screens (selection screens for various applications, etc.) or content images from the entertainment system. Display 42 displays, for example, instruments, a gear position display indicating the current gear position, icons indicating the seatbelt status, vehicle images, images of the vehicle's surroundings, predicted driving routes, driving paths, etc. In addition, display 41 and / or display 42 display images generated by the display control device 100.

[0024] Displays 41 and 42 have a width direction ( Figure 2 The horizontally elongated display screen (in the left-right direction on the paper). For example... Figure 2 As shown, displays 41 and 42 are arranged horizontally along the vehicle width direction, with their display screens close to each other. Alternatively, the vehicle may not need both displays 41 and 42; either display is sufficient. The vehicle speed sensor 50 detects the vehicle speed and outputs the detected value to the controller 10.

[0025] The display control device 100 includes a controller 10 and a recording medium 19. The controller 10 has functions such as acquiring various data including camera images and vehicle data, and image processing functions. As a functional module, the controller 10 includes an image acquisition unit 11, a vehicle data acquisition unit 12, and an image processing unit 13. The controller 10 stores programs for implementing various functions in the recording medium 19, and executes the processing flow represented by the program by having a processor execute the control program. Furthermore, the recording medium 19, which stores the control program, can also be stored as memory within the controller 10. Additionally, the recording medium 19 can be located inside or outside the controller 10. For example, the recording medium 19 can be stored on a server outside the vehicle, and the display control device 100 can access the server to store the control program data recorded in the recording medium 19 into memory within the controller 10. Furthermore, the display control device 100 can also execute the control program within the server via communication with the server. For example, the display control device 100 can send instructions for executing the processing steps included in the control program or camera images to the server. The server, containing a controller, executes a control program based on instructions from the display control device 100 to process data such as camera images. The server then sends the processed display image to the display control device 100. The display control device 100 sends the display image to the monitors 41 and 42 and displays the image on them.

[0026] The image acquisition unit 11 acquires camera images from the camera 20, etc. The vehicle data acquisition unit 12 acquires vehicle data from sensors mounted in the vehicle via the in-vehicle communication network. Vehicle data represents the current state of the vehicle, including, for example, vehicle speed, steering angle, current position (current coordinates), and gear shift. The vehicle data acquisition unit 12 acquires vehicle speed from the vehicle speed sensor 50 and steering angle from the steering angle sensor. The vehicle data acquisition unit 12 acquires current position information from the navigation system. Additionally, the vehicle data acquisition unit 12 acquires gear shift information from the ATCU 30.

[0027] The image processing unit 13 performs image processing on the camera images in order to display the current surrounding image and / or vehicle image on the displays 41 and 42. The image processing unit 13 generates display images and outputs them to the displays 41 and 42 in a manner that displays the images on the displays 41 and 42 according to a display mode specified by the user or system. Multiple display modes exist, including a front view showing the front of the vehicle, a rear view showing the rear of the vehicle, a top view looking down at the vehicle, a three-dimensional view of the vehicle or its surroundings from a virtual viewpoint, a perspective view, etc. In the front view, an image showing the exterior of the vehicle is displayed at a position further forward than the vehicle's grille. In the rear view, an image showing the exterior of the vehicle is displayed at a position further back than the vehicle's rear doors. In the top view (surround view), a top view image is displayed, showing the vehicle from a virtual viewpoint directly above it. In the top view, a two-dimensional image is displayed, where the vehicle image is represented by a plan view of the vehicle; if there are three-dimensional objects of equal height around the vehicle, the objects are displayed as if tilted. The 3D view displays an image of the object as viewed from a virtual viewpoint, using the area surrounding the object as a virtual viewpoint. The object is not limited to vehicles; it can also be obstacles around the vehicle. The virtual viewpoint is the position of the viewpoint when simulating the observation of the object from its outside. The virtual viewpoint can be moved. For example, when the image of a vehicle viewed from a virtual viewpoint is displayed on monitors 41 and 42, the 3D-displayed vehicle image can be rotated by moving the virtual viewpoint around the vehicle along a plane. Additionally, monitors 41 and 42 display the vehicle's perimeter, for example, through perspective views, front wide-angle views, and rear wide-angle views. In the perspective view, in addition to the current camera image (real image) showing the vehicle's perimeter, a virtual image showing the state under the vehicle's floor is also displayed. The front wide-angle view, for example, displays an image of the area in front of the vehicle that is the driver's blind spot, based on the image from the front camera 21 with a wide-angle lens, when the vehicle is making a left or right turn at a poorly visible intersection such as a T-junction.

[0028] The display mode can be appropriately changed according to the user's specifications or the vehicle's driving status via the vehicle system. For example, when the vehicle is in reverse gear to park, the display mode can automatically switch to a top view or a rear view. Alternatively, the display mode can be switched by the driver operating a switch or other means while the vehicle is in motion.

[0029] The image processing unit 13 can display images in multiple display modes on the display 41 through multiple screens. Furthermore, the image processing unit 13 can display images in different display modes on both the display 41 and the display 42, or it can display images in the same display mode on both displays 41 and 42. Additionally, the image processing unit 13 can display images in multiple display modes as a single screen on both displays 41 and 42, or it can display images in a single display mode. Furthermore, the following description focuses on the image processing performed by the image processing unit 13 when displaying an image on the display 41; however, the image processing performed on the display 41 can also be applied to the image display on the display 42 when displaying an image on the display 42.

[0030] The image processing unit 13 generates a display image corresponding to the display mode based on multiple camera images acquired by the image acquisition unit 11, and outputs the display image to the display 41. When displaying an image through a front view or a rear view, the image processing unit 13 generates a display image that displays the front or rear of the vehicle based on the camera images of the front camera 21 or the rear camera 24.

[0031] When displaying an image from a top-down view, the image processing unit 13 generates a top-down view image of the vehicle based on multiple camera images captured by the camera 20. The virtual viewpoint of the top-down view image is a predetermined position located outside the vehicle, extending vertically away from the roof of the vehicle. For example, the image processing unit 13 generates a viewpoint transformation image by performing viewpoint transformation processing on images captured by the front camera 21, right camera 22, left camera 23, and rear camera 24, allowing the vehicle to be viewed from the virtual viewpoint's front and rear areas as well as its left and right areas. In other words, the image processing unit 13 transforms camera images corresponding to those viewed from the camera's position into viewpoint transformation images viewed from the virtual viewpoint. The viewpoint transformation from the camera position to the virtual viewpoint is performed using viewpoint parameters, which are stored in the recording medium 19. The image processing unit 13 adapts each viewpoint-transformed image to the surrounding area of ​​the vehicle. The image processing unit 13 performs fusion processing on the superimposed portions of the images from the cameras 20. The fusion process is used to maintain continuity between two superimposed images, and methods known at the time of this application can be applied. Furthermore, the fused image becomes a surrounding image representing the area around the vehicle. Additionally, the image processing unit 13 composites the vehicle image located in the center of the screen with the surrounding image. The composite image becomes a top-view image, and the image processing unit 13 outputs the top-view image as a display image to the display 41.

[0032] When an image is displayed in a 3D view, the image processing unit 13 generates a display image of the vehicle viewed from a virtual viewpoint based on multiple camera images captured by the camera 20. First, the image processing unit 13 sets the virtual viewpoint according to user specifications or driving conditions. The position of the virtual viewpoint and the direction of the line of sight can be arbitrarily set based on user instructions. For example, the user can specify the position of the virtual viewpoint or the direction of the line of sight through touch operation on the display 41. Furthermore, the image processing unit 13 can set the virtual viewpoint according to driving conditions. For example, when moving forward, a virtual viewpoint can be set behind the vehicle to observe the image in front of it; when reversing, a virtual viewpoint can be set in front of the vehicle to observe the image behind it. The image processing unit 13 sets the position of the virtual viewpoint upon receiving shift information indicating that the gear shift position has entered reverse gear. Furthermore, the image processing unit 13 can also set the virtual viewpoint according to vehicle speed. For example, based on the vehicle speed input from the vehicle speed sensor 50, the virtual viewpoint can be set to: when the vehicle speed is greater than or equal to a specified value, the position of the virtual viewpoint is far away from the vehicle; when the vehicle speed is less than a specified value, the position of the virtual viewpoint is close to the vehicle.

[0033] The image processing unit 13 deforms at least a portion of a predefined reference surface coordinate system, which is used to project images from a camera, based on the position of the virtual viewpoint. The reference surface coordinate system is stored in the recording medium 19. For example, the reference surface coordinate system is shaped like a bowl surrounding the vehicle. The reference surface coordinate system has a center on the xy-plane parallel to the vehicle's mounting surface (driving surface), and is formed by a surface that has curvature from or near the center, having a component in the vehicle's height direction (z-direction).

[0034] Near the center of the reference surface coordinate system, a base surface parallel to the xy plane can be formed, and its shape can be any shape such as rectangle, triangle, ellipse, or circle. Furthermore, the curvature of the surfaces in the reference surface coordinate system does not have to be uniform. For example, the curvature of the surfaces near the base surface can be relatively large, while the curvature of the surfaces far from the base surface can be relatively small. In addition, the image processing unit 13 can pre-store multiple reference surface coordinate systems with different shapes according to the shape (size, shape) of the vehicle. When multiple reference surface coordinate systems are stored, the reference surface coordinate system used is stored in association with the vehicle speed, the vehicle's gear shift position, etc., in the recording medium 19, and the image processing unit 13 can select the reference surface coordinate system according to the vehicle speed, the vehicle's gear shift position, etc.

[0035] The image processing unit 13 can also read a pre-stored reference surface coordinate system and deform the shape of the read reference surface coordinate system. The image processing unit 13 can also deform the shape of the reference surface coordinate system according to the position of the virtual viewpoint. For example, if the virtual viewpoint is located outside the outer edge of the reference surface coordinate system, a portion of the reference surface coordinate system can be deformed so that the virtual viewpoint is located further inside the outer edge of the reference surface coordinate system. By deforming a portion of the reference surface coordinate system so that the virtual viewpoint is located further inside the outer edge of the reference surface coordinate system, and thus displaying obstacles of height on the screen, distortion of the obstacle's shape can be suppressed, and an image without image defects can be displayed on the display 41. Furthermore, other coordinate system deformation methods known at the time of application can be used for the deformation method of the reference surface coordinate system.

[0036] The image processing unit 13 projects multiple camera images acquired from the camera 20 onto a reference surface coordinate system. Furthermore, if the reference surface coordinate system is deformed, the image processing unit 13 only needs to project the camera images onto the deformed surface coordinate system (deformable coordinate system). The image processing unit 13 has an image transformation table that associates the coordinates of the pixels contained in the camera images with the coordinates of the coordinate system, so as to project the camera image data onto the reference surface coordinate system or the deformed coordinate system. The image transformation table is recorded on the recording medium 19.

[0037] Alternatively, a pre-prepared vehicle image can be superimposed on a reference surface coordinate system. The vehicle image can be pre-created based on the vehicle's appearance and stored in the recording medium 19. By superimposing the vehicle image on the reference surface coordinate system, not only can the image surrounding the vehicle be displayed in three dimensions, but the vehicle itself can also be displayed in three dimensions, which helps in understanding the relationship between the vehicle's position and orientation and the surrounding images. As described above, the image processing unit 13, referring to the image transformation table, establishes a relationship between the coordinates of the pixels contained in the camera image and the coordinates of the coordinate system, projects the camera image onto the reference surface coordinate system or the deformed coordinate system, and superimposes the vehicle image onto the projected three-dimensional image. This generates a display image (three-dimensional image) in a three-dimensional view of the vehicle and its surroundings when viewed from a virtual viewpoint.

[0038] The image processing unit 13 is not limited to generating display images from virtual viewpoints set according to user specifications or driving conditions, but can also generate display images from selectable virtual viewpoints. For example, when a display image is shown on the monitor 41 in a three-dimensional view, the user can freely move the position of the virtual viewpoint through touch operation on the monitor 41. When a display in a three-dimensional view is selected, the image processing unit 13 generates multiple three-dimensional view display images for multiple virtual viewpoints selectable by the user. Furthermore, for example, when the user moves the virtual viewpoint through touch operation on the monitor 41, a three-dimensional image of the vehicle is displayed on the monitor 41 rotated around the center point of a surface coordinate system. In addition, the image processing unit 13 is not limited to multiple virtual viewpoints selectable by the user, but can also set virtual viewpoints at any position around the vehicle and generate multiple three-dimensional view display images for each virtual viewpoint. That is, when the image processing unit 13 displays a three-dimensional image on the monitor 41 in a rotated display manner, it generates a continuous three-dimensional image so that smooth image display is possible when moving relative to the virtual viewpoint.

[0039] Furthermore, the image processing methods for the displayed images in various display modes such as front / rear view, top view, and three-dimensional view are not limited to the methods described above, and other image processing methods can also be used. In addition, the image processing methods for the displayed images in other display modes such as perspective view, front wide-angle view / rear wide-angle view, etc., can appropriately use methods known at the time of application.

[0040] Next, a specific example of the display control implemented by the display control device 100 and the display screen of the display 41 will be described together. Figure 3 The display screen 41 shows the screen displayed. Figure 3In the example, when the vehicle is parking on a straight road, the left side of the display 41 shows a top-down view image, and the right side of the display 41 shows a three-dimensional view image. When the controller 10 displays the top-down view image and the three-dimensional view image on the display 41, it generates a top-down view image 46 by overlaying viewpoint icons 43a-43h and an automatic display start icon 44 onto the top-down view image of the vehicle. The top-down view image 46 includes a top-down view of the vehicle from above, viewpoint icons 43a-h, and an automatic display start icon 44. Viewpoint icons 43a-h are icons indicating the position of a virtual viewpoint and / or the direction of the line of sight, and are selectable by the user. Multiple viewpoint icons 43a-43h are arranged around the vehicle image, centered on it. Viewpoint icon 43a is positioned above the vehicle image located at the center of the top-down view image, while viewpoint icons 43c, 43e, and 43g are positioned to the right, bottom, and left of the vehicle image, respectively. Additionally, viewpoint icons 43b, 43d, 43f, and 43h are positioned relative to the vehicle image in the upper right, lower right, lower left, and lower right directions, respectively. If the user selects viewpoint icons 43a to 43h, a display image (3D image) of the vehicle viewed from the selected viewpoint icon is displayed on the right side of the display 41.

[0041] like Figure 3 As shown, if the user touches the viewpoint icon 43a, the controller 10 causes the viewpoint icon 43a to be displayed as a selection icon. The controller 10 displays the selection icon, for example, by changing the color of the viewpoint icon 43a. Figure 3 In the example, the selection icon is represented by setting the outer perimeter of the viewpoint icon 43a to a thick line. The image processing unit 13 sets a virtual viewpoint corresponding to the viewpoint icon 43a. Thus, the virtual viewpoint is set according to the user's specification. Figure 3 In this example, the position when viewing the vehicle from the front is set as a virtual viewpoint. Furthermore, the image processing unit 13 generates a three-dimensional image of the vehicle and its surroundings when viewed from the front, based on multiple camera images. The image processing unit 13 outputs the three-dimensional image as a display image to the display 41. The three-dimensional image (display image of the three-dimensional view) 45 is displayed on the right side of the display 41. And, as... Figure 3 As shown, the display 41 displays a top-view image 46 and a three-dimensional image 45 arranged in a row.

[0042] The auto-display start icon 44 is an icon used in the 3D view to initiate automatic switching of the 3D image by moving the virtual viewpoint. The auto-display start icon 44 is displayed at the center of the vehicle image contained in the top-view image 46. If auto-display is executed, predetermined positions around the vehicle become the start and end points of the virtual viewpoint. The virtual viewpoint moves from the start to the end point, and the vehicle image displayed in the 3D view rotates 360 degrees along a plane parallel to the road surface. In other words, during auto-display, a center point is set on the vehicle image, and the vehicle image and the images surrounding the vehicle rotate 360 ​​degrees around this center point. For example, in... Figure 3 In the example, if the user touches the automatically displayed start icon 44, the selected viewpoint icon 43a becomes the start and end point of the virtual viewpoint. The virtual viewpoint moves sequentially from viewpoint icon 43b to viewpoint icon 43h, rotating clockwise around the vehicle image, starting from the position corresponding to viewpoint icon 43a, and returning to the position corresponding to the initial viewpoint icon 43a. Accompanying the rotation and movement of the virtual viewpoint, the 3D image 45 in the 3D view rotates. The orientation of the vehicle image contained in the 3D image 45 is as follows: Figure 3 Starting from the front (front view), the display changes in the order of right side, back (rear view), and left side, returning to the initial front (front view). That is, the 3D image becomes a 360-degree rotated display (hereinafter also referred to as "360-degree rotated display"). Thus, in the 3D view, the displayed image (3D image 45) is automatically rotated 360 degrees.

[0043] Figure 4 The display screen 41 shows the screen displayed. Figure 4 In the example, when the vehicle is parking on a T-shaped road, the left side of display 41 shows a top-view image, and the right side of display 41 shows a three-dimensional view image. The image processing methods for the top-view image 46 and the three-dimensional image 45 are similar to... Figure 3 The example is the same; the top-view image 46 also includes viewpoint icons 43a~h and the automatically displayed start icon 44. (Similar to...) Figure 3 The difference in the example is that viewpoint icon 43e is selected, displaying a 3D image 45 of the vehicle and its surroundings as viewed from behind in the 3D view. Furthermore, as... Figure 4 As shown, when the vehicle is parked on a T-shaped road, the controller 10 can set a virtual viewpoint corresponding to the viewpoint icon 43a and display a 3D image 45 viewed from the virtual viewpoint on the display 41. Therefore, when there is a blind spot in front of the vehicle that is a blind spot for the driver, the status of the blind spot area can be displayed on the display 41.

[0044] In addition, such as Figure 3 and Figure 4As shown, when the vehicle is parked, the user can select viewpoint icons 43a-43h and confirm the 3D image on the display 41 in the direction of the selected viewpoint icon 43a-43h. For example, if the user touches the position of viewpoint icon 43c on the screen of the display 41, the image processing unit 13 switches viewpoint icon 43c to a selection icon, so that the user can view the 3D image from the current virtual viewpoint (in the viewpoint icon 43a-43h). Figure 3 In the example, the virtual viewpoint (corresponding to viewpoint icon 43a) moves to the virtual viewpoint corresponding to viewpoint icon 43c. Furthermore, the image processing unit 13 generates a 3D image from the virtual viewpoint based on the camera image and outputs the image to the display 41, so that the 3D image is displayed through a 3D view. The display 41 shows a 3D image of the vehicle viewed from the right side.

[0045] Additionally, when the vehicle is parked, the controller 10 can perform display control, allowing the user to not only select a virtual viewpoint via touch operation of viewpoint icons 43a-43h, but also arbitrarily specify the position of the virtual viewpoint. The user touches any position on the 3D view display screen. Furthermore, while the finger is touching the screen, the user moves their finger in any direction (slide operation). During the period the finger is touching the screen, the position of the virtual viewpoint moves in accordance with the slide operation, and the 3D image rotates in accordance with the movement of the virtual viewpoint. The amount of movement of the virtual viewpoint is determined by the amount of slide (equivalent to the amount of finger movement while touching the screen); the larger the amount of slide, the larger the amount of movement. Moreover, the amount of movement of the virtual viewpoint is equivalent to the length of the arc of a circle centered on the vehicle image. For example, the user touches the 3D view display screen with their finger, moving their finger with a large amount of slide. In the case of a large amount of slide, the image processing unit 13 moves the virtual viewpoint around the center point of the vehicle image, rotating the 3D image for display. When the sliding amount is small, the image processing unit 13 rotates the 3D image by moving the virtual viewpoint along an arc centered on the center point of the vehicle image with a movement amount equivalent to the sliding amount. Furthermore, the rotation display achieved by the sliding operation is not limited to being performed through a sliding operation on the display screen of the 3D view, but can also be performed through a sliding operation on the display screen of the top view. For example, the image processing unit 13, in situations such as... Figure 3 or Figure 4 In the top-view image 46 shown, the image with viewpoint icons 43a-43h removed is displayed on the monitor 41. Furthermore, the user can perform a sliding operation at any position on the screen displaying the top-view image 46. Based on the sliding operation on the top-view display screen, the image processing unit 13 rotates and displays the three-dimensional image on the three-dimensional view display screen.

[0046] In this embodiment, when the vehicle is parked or stationary, no restrictions are imposed on the movement of the virtual viewpoint implemented by the sliding operation, and the virtual viewpoint can move freely according to the amount of sliding. On the other hand, when the vehicle is moving, restrictions are imposed on the movement of the virtual viewpoint implemented by the sliding operation (hereinafter also referred to as "sliding restriction"). The display control when the movement of the virtual viewpoint is restricted will be described below.

[0047] The image processing unit 13 determines whether the vehicle is moving based on the vehicle data acquired by the vehicle data acquisition unit 12. If the vehicle speed contained in the vehicle data exceeds a parking threshold, the image processing unit 13 determines that the vehicle is moving. The parking threshold is a preset speed threshold used to determine whether a vehicle is in a parking or stopped state (e.g., 0 km / h or a few km / h). Furthermore, the image processing unit 13 can also determine whether the vehicle is moving based on gear shift information contained in the vehicle data. For example, if the current gear position (gear) is a forward gear according to the gear shift information contained in the vehicle data, the image processing unit 13 can also determine that the vehicle is moving.

[0048] When the vehicle is in motion, the image processing unit 13 restricts the movement of the virtual viewpoint controlled by the user. The image processing unit 13 switches between a movement restriction mode, which restricts the movement of the virtual viewpoint, and a movement freedom mode, which allows the virtual viewpoint to move freely. The movement freedom mode does not restrict the movement of the virtual viewpoint. In movement freedom mode, the virtual viewpoint can move freely according to the amount of sliding, and in the 3D view, the 3D image can be rotated in a manner that matches the movement of the virtual viewpoint. On the other hand, in movement restriction mode, the movement of the virtual viewpoint is independent of the amount of sliding operation and is restricted to one unit per operation. One unit of movement of the virtual viewpoint is the smallest unit of movement. When the vehicle is in motion, the image processing unit 13 selects the movement restriction mode. When the vehicle is parked or stationary, the image processing unit 13 selects the movement freedom mode.

[0049] Reference Figure 5 The rotational display of the 3D image 45 under the condition of implementing sliding restrictions will be explained. Figure 5 This is a diagram used to illustrate the rotational display of a 3D image 45 in a 3D view. Figure 5 (a) Shows a 3D image 45 as viewed from a virtual viewpoint corresponding to viewpoint icon 43e. Figure 5 (b) Shows a three-dimensional image 45 as viewed from a virtual viewpoint corresponding to viewpoint icon 43f. Figure 5 (c) A three-dimensional image 45 is shown when viewed from a virtual viewpoint corresponding to viewpoint icon 43g. Furthermore, in Figure 5 The image shown here is only a 3D view displayed on the screen of monitor 41. The user touches monitor 41 and performs a sliding operation. The 3D image 45 rotates in the direction specified by the sliding operation. The amount of movement of the virtual viewpoint is independent of the amount of sliding; for one sliding operation, it moves by one unit. For example, in... Figure 5 On the display shown in (a), the user performs a sliding operation to rotate the vehicle image counterclockwise. The image processing unit 13 determines the operation direction (sliding direction) based on the user's sliding operation and moves the virtual viewpoint by one unit. Along with the one unit movement of the virtual viewpoint, the 3D image 45 rotates counterclockwise by 45 degrees. Figure 5 In the example, one unit of movement of the virtual viewpoint corresponds to a rotation angle of 45 degrees in the 3D image, and is equivalent to a movement of one viewpoint in viewpoint icons 43a~43f. Furthermore, one viewpoint represents movement from the current virtual viewpoint to an adjacent virtual viewpoint. Figure 5 The 3D image shown in (a) is rotated by a single sliding operation, becoming Figure 5 (b) shows a three-dimensional image. Furthermore, if in such a case... Figure 5 (b) On the display screen shown, when the user performs the same sliding operation, the image processing unit 13 rotates the 3D image 45 counterclockwise by 45 degrees. The display screen of the monitor 41 changes from... Figure 5 (b) The display screen changes to Figure 5 (c) is the display screen. That is, in the movement restriction mode, the virtual viewpoint moves only one unit in the direction of the swipe operation with one swipe.

[0050] If the vehicle speed is less than or equal to the parking threshold, and the vehicle is parking or in a parking position, the image processing unit 13 releases the movement restriction mode and selects the free movement mode. In the free movement mode, the image processing unit 13 determines the operation direction (sliding direction) based on the sliding operation performed by the user, and allows the virtual viewpoint to move freely according to the sliding amount. That is, in the free movement mode, while the finger touches the display screen of the display 41, the virtual viewpoint moves in match with the movement of the finger's contact point. Furthermore, the 3D image rotates in match with the free movement of the virtual viewpoint.

[0051] When the vehicle is in motion and its speed is greater than or equal to the viewpoint restriction threshold, the virtual viewpoints selectable by the user are restricted based on the vehicle's direction of travel. The following describes the display control when restrictions are imposed on selectable virtual viewpoints (hereinafter also referred to as "virtual viewpoint selection restrictions").

[0052] The image processing unit 13 determines whether the current vehicle speed is greater than or equal to a viewpoint restriction threshold based on the vehicle speed obtained by the vehicle data acquisition unit 12. The viewpoint restriction threshold is a preset threshold for vehicle speed (e.g., 10~20 km / h), which is used to determine whether to apply virtual viewpoint selection restriction.

[0053] When the current vehicle speed is greater than or equal to the viewpoint limitation threshold, the image processing unit 13 sets selectable and non-selectable virtual viewpoints from a plurality of virtual viewpoints located around the vehicle, based on the vehicle's direction of travel. The plurality of virtual viewpoints located around the vehicle are the positions of viewpoints when simulating an observation of the vehicle from the outside. The plurality of virtual viewpoints correspond to viewpoint icons 43a-43h. For example, when the vehicle speed is greater than or equal to the viewpoint limitation threshold and the vehicle is moving forward, the image processing unit 13 limits the selectable virtual viewpoints to viewpoints from behind the vehicle. Furthermore, when the vehicle speed is greater than or equal to the viewpoint limitation threshold and the vehicle is reversing, the image processing unit 13 limits the selectable virtual viewpoints to viewpoints from in front of the vehicle. In other words, when the vehicle speed is greater than or equal to the viewpoint limitation threshold, virtual viewpoint selection is restricted in such a way that the vehicle image contained in the 3D image faces the direction of travel.

[0054] For example, when the vehicle is moving forward, in situations such as Figure 3 When the 3D image 45 shown is displayed in a 3D view, the vehicle image is oriented directly towards the driver, and the state of the moving vehicle does not match the display of the 3D view. Furthermore, in situations such as... Figure 3 In the displayed 3D view, it is difficult to see the direction of travel. In this embodiment, when the vehicle is moving forward, only three viewpoints from the rear can be selected as virtual viewpoints. Therefore, in the 3D view, when a 3D image 45 is displayed showing the view from the restricted virtual viewpoints, the vehicle image becomes an image indicating forward movement, making the direction of travel easier to observe. Furthermore, when the vehicle is reversing, only three viewpoints from the front can be selected as virtual viewpoints.

[0055] Next, the display of the top-view image when virtual viewpoint selection restrictions are applied will be explained. In the top-view image, viewpoint icons 43a-43h and the automatic display start icon 44 are superimposed on the top-view image of the vehicle. (See attached image.) Figure 3 and Figure 4As shown, when the vehicle is parked, viewpoint icons 43a-43h and the automatic display start icon 44 are selectable. On the other hand, when the vehicle speed is greater than or equal to the viewpoint limit threshold and the vehicle is moving forward, the image processing unit 13 sets viewpoint icons 43a-43c, 43g, 43h and the automatic display start icon 44 to a state where they are not selectable. To indicate that they are not selectable, the image processing unit 13 can, for example, set the icons to a gray display or a gray shadow, or it can remove viewpoint icons 43a-43c, 43g, 43h. Additionally, the image processing unit 13 sets viewpoint icons 43d-43f to a selectable state.

[0056] In addition, when the vehicle speed is greater than or equal to the viewpoint limit threshold and the vehicle is reversing, the image processing unit 13 sets the viewpoint icons 43c~43g and the automatic display start icon 44 to an unselectable state, and sets the viewpoint icons 43a, 43b and 43h to a selectable state.

[0057] Next, refer to Figure 6 The display control method implemented by controller 10 will be described. Figure 6 This is a flowchart of the display control method implemented by controller 10. During the process of displaying the image on the left side of display 41 in a top view and the image on the right side in a three-dimensional view, controller 10 repeatedly executes the following control flow.

[0058] In step S1, the image acquisition unit 11 acquires a camera image from the camera 20. The vehicle data acquisition unit 12 acquires vehicle data from the ATCU 30 and / or the vehicle speed sensor 50. In step S2, the image processing unit 13 detects the current vehicle state based on the vehicle data. Furthermore, in this process, the vehicle state is detected based on the vehicle speed. In step S3, the image processing unit 13 determines whether the current vehicle speed is less than or equal to a parking threshold, or the current vehicle speed is higher than the parking threshold but lower than the viewpoint limitation threshold, or the current vehicle speed is greater than or equal to the viewpoint limitation threshold.

[0059] If the current vehicle speed is less than or equal to the parking threshold, the vehicle is determined to be in a parking state, and the image processing unit 13 cancels the movement restriction or virtual viewpoint selection restriction implemented by the sliding limit (step S4). In step S5, the image processing unit 13 generates a top-view image of the vehicle viewed from above based on multiple camera images. The top-view image is an image superimposed with the viewpoint icons 43a-43h and the automatic display start icon 44 on the top-view image of the vehicle and its surroundings. Furthermore, the display of the viewpoint icons 43a-43h and the automatic display start icon 44 is set to a selectable state. In step S6, the image processing unit 13 generates a three-dimensional image of the vehicle viewed from a virtual viewpoint based on multiple camera images. Furthermore, the virtual viewpoint is generated by repeatedly executing... Figure 6 During the control flow, specified actions are performed through user-implemented icon touch operations, etc.

[0060] Then, in step S7, the image processing unit 13 outputs the image, including the top view image and the three-dimensional image, to the display 41 as a display image. Figure 7 The display screen shown on the display 41 is shown according to the control flow of step S7. For example... Figure 7 As shown, a top-down view image is displayed on the left side of the monitor 41, with viewpoint icons 43a-43h and the automatic display start icon 44 set to selectable states. Additionally, in Figure 7 In the example, the user selects a viewpoint icon 43h, and the screen on the right side of the display 41 displays a 3D image (a virtual image viewed from the left front of the vehicle) 45, which is viewed from the virtual viewpoint corresponding to the viewpoint icon 43h. Furthermore, when the display 41 is displaying... Figure 7 When the screen is displayed, if the user operates, for example, selects the viewpoint icon 43g, the image processing unit 13 moves the position of the virtual viewpoint to the position corresponding to the viewpoint icon 43g, and outputs the three-dimensional image observed from the moved virtual viewpoint to the display 41.

[0061] Returning to step S3, if the current vehicle speed is higher than the parking threshold but lower than the viewpoint restriction threshold, the controller 10 determines that the vehicle is in motion and executes the control flow of steps S8-S11. In step S8, the image processing unit 13 performs a sliding restriction, limiting the movement of the virtual viewpoint implemented by the user. The virtual viewpoint selection restriction is canceled. In step S9, the image processing unit 13 generates a top-down view image based on multiple camera images. Viewpoint icons 43a-43h are set to selectable, and the automatic display start icon 44 is set to unselectable. Furthermore, although the virtual viewpoint selection restriction is canceled, the automatic display start icon 44 is set to unselectable. In step S10, the image processing unit 13 generates a three-dimensional image of the vehicle viewed from the virtual viewpoint based on multiple camera images. Furthermore, the virtual viewpoint is repeatedly executed... Figure 6 The control flow is specified through user touch operations on icons, etc.

[0062] Then, in step S11, the image processing unit 13 outputs the image, including the top view image and the three-dimensional image, to the display 41 as a display image. Figure 8 This shows the display screen shown on the display 41 as a result of the control flow in step S11. For example... Figure 8 As shown, a top-down view image is displayed on the left side of the monitor 41, with viewpoint icons 43a-43h set to selectable and the auto-start icon 44 set to unselectable. Additionally, in Figure 7 In the example, the user selects viewpoint icon 43h, and the right side of the display 41 shows a 3D image viewed from the virtual viewpoint corresponding to viewpoint icon 43h (a virtual image viewed from the left front of the vehicle). Figure 8 As shown in the example, automatic display that causes the 3D image to rotate 360 ​​degrees can also be disabled when the sliding limit is applied. The vehicle image contained in the 3D image 45 becomes the image viewed from a virtual viewpoint specified by the user, and the orientation of the vehicle image may not correspond to the vehicle's direction of travel.

[0063] Returning to step S3, if the current vehicle speed is greater than or equal to the viewpoint limit threshold, the controller 10 determines that the vehicle is in motion (moving at a speed higher than slow-moving) and executes the control flow of steps S12 to S15. In step S12, the image processing unit 13 executes sliding limit and virtual viewpoint selection limit. In step S13, the image processing unit 13 generates a top-view image based on multiple camera images. According to the vehicle's direction of travel, one of the viewpoint icons 43a-43h is set to selectable, and the other viewpoint icons 43a-43h are set to unselectable. For example, when the vehicle is moving forward, viewpoint icons 43d-43f are set to selectable, and other viewpoint icons 43a-43c, 43g, and 43h are set to unselectable. In addition, the automatic display start icon 44 is set to unselectable. In step S14, the image processing unit 13 generates a three-dimensional image of the vehicle viewed from a virtual viewpoint based on multiple camera images.

[0064] Then, in step S14, the image processing unit 13 outputs the image, including the top view image and the three-dimensional image, to the display 41 as a display image. Figure 9 The display screen shown on the display 41 is shown according to the control flow of step S14. For example... Figure 9 As shown, a top-down view image is displayed on the left side of the monitor 41. Viewpoint icons 43d-43f are selectable, while viewpoint icons 43a-43c, 44g, 44h, and the auto-display start icon 44 are unselectable. The orientation of the vehicle image included in the 3D image 45 corresponds to the vehicle's direction of travel. That is, the virtual viewpoint is set to any one of the three viewpoints located behind the vehicle (the virtual viewpoints corresponding to viewpoint icons 43d-43f). Figure 9 In the example, the virtual viewpoint of the 3D image 45 is set at the center of the vehicle behind the vehicle.

[0065] Then, after executing the control flow of steps S7, S11, and S15, the control flow ends. Controller 10 restarts the control flow from step S1.

[0066] Furthermore, when repeatedly executed Figure 6During the control flow, if the current vehicle speed is greater than or equal to the viewpoint limitation threshold and a new virtual viewpoint limitation is applied, the image processing unit 13 can automatically switch the virtual viewpoint to the direction of the vehicle's travel when executing the control flow in step S14. For example, when the vehicle is moving forward, the image processing unit 13 sets the virtual viewpoint to the center of the rear of the vehicle. When the vehicle changes from a slow-moving state to a state with a speed higher than slow-moving, the image processing unit 13 automatically switches the virtual viewpoint to a viewpoint that reflects the vehicle's travel direction in the three-dimensional view. For example, when the vehicle is moving forward slowly, and the speed is higher than the parking threshold but lower than the viewpoint limitation threshold, the display 41 displays as shown in the image. Figure 8 The image shown. From Figure 8 When the vehicle speed increases and becomes greater than or equal to the viewpoint limitation threshold, the image processing unit 13 moves the virtual viewpoint to the position corresponding to the viewpoint icon 43e, generates a 3D image with the vehicle image in the 3D view aligned with the vehicle's direction of travel, and outputs the 3D image to the display 41. The display screen on the display 41 then... Figure 8 The displayed screen automatically switches to Figure 9 The screen shown is shown.

[0067] Furthermore, if the vehicle decelerates from a state where it is moving forward and its speed is greater than or equal to the viewpoint limitation threshold, and the current speed becomes less than the viewpoint limitation threshold, the image processing unit 13 sets the viewpoint icons 43a-43h, which were previously set to be unselectable, to be selectable. For example, when the display 41 shows viewpoint icons 43a-43h... Figure 9 In the state shown in the image, when the vehicle speed becomes less than the viewpoint limitation threshold, the image processing unit 13 executes the control flow of steps S8 to S11, thereby achieving the following... Figure 10 The top-view image 46 shown is displayed on the monitor 41 in such a way that viewpoint icons 43a~43h are selectable. Figure 9 In the top-down view, viewpoint icons 43a~43c, 43g, and 43h are unselectable, but... Figure 10 In the top view, viewpoint icons 43a~43c, 43g, and 43h are switched to selectable states.

[0068] Furthermore, as mentioned above, refer to Figures 7-10 The control flow for each step is explained using the example of a vehicle moving forward. However, the control flow for steps S1 to S15 can also be applied to the display control of a display 41 that is moving backward. For example, when the vehicle is moving backward, the controller 10 repeatedly executes... Figure 6 During the control process, if the current vehicle speed becomes greater than or equal to the viewpoint limitation threshold, the image processing unit 13 makes such a control as follows: Figure 11The image shown is displayed on monitor 41, generating a top-view image 46 and a 3D image 45. Of the multiple viewpoint icons 43a-43h superimposed on the top-view image 46, one viewpoint icon 43a, 43b, or 43h that allows selection of the area in front of the vehicle as a virtual viewpoint becomes selectable, while the other viewpoint icons 43c-43g become unselectable. Furthermore, the orientation of the vehicle image included in the 3D image 45 corresponds to the vehicle's direction of travel. When the vehicle is reversing, the virtual viewpoint is set to any one of the three viewpoints located behind the vehicle (the virtual viewpoints corresponding to viewpoint icons 43a, 43b-43h). Figure 11 In the example, the virtual viewpoint of the 3D image 45 is set at the center of the vehicle in front of it.

[0069] Furthermore, when the vehicle is reversing, and the speed changes from a slow-moving state to a state with a speed higher than the slow-moving state, the image processing unit 13 can automatically switch the virtual viewpoint to a viewpoint that reflects the vehicle's direction of travel in the three-dimensional view. For example, when the vehicle is slowly reversing and the speed is higher than the parking threshold but lower than the viewpoint restriction threshold, the image processing unit 13 sets viewpoint icons 43a-43h to selectable, generates a top-view image, and generates a three-dimensional image viewed from a user-specified virtual viewpoint. When the vehicle is reversing and the speed becomes greater than or equal to the viewpoint restriction threshold, the image processing unit 13 moves the virtual viewpoint to the position corresponding to viewpoint icon 43a, generates a three-dimensional image with the vehicle image in the three-dimensional view facing the direction of the vehicle's travel, and outputs the three-dimensional image to the display 41. Thus, the orientation of the vehicle image included in the three-dimensional image is automatically switched.

[0070] In this embodiment, the controller 10 executes different control flows depending on whether the vehicle speed is less than or equal to the parking threshold, higher than the parking threshold but lower than the viewpoint limitation threshold, or higher than or equal to the viewpoint limitation threshold. That is, the display control content executed by the controller 10 differs depending on the vehicle speed condition. The control content for situations where the vehicle speed changes to meet other speed conditions will be described below. Furthermore, in the following description, the vehicle state when the vehicle speed is less than or equal to the parking threshold is defined as "parking state," the vehicle state when the vehicle speed is higher than the parking threshold but lower than the viewpoint limitation threshold is defined as "slowing down state," and the vehicle state when the vehicle speed is higher than or equal to the viewpoint limitation threshold is defined as "normal driving state."

[0071] When the vehicle is moving forward, and the transition from "slow-moving state" to "normal driving state" occurs, the controller 10 performs display control as follows: When driving in "slow-moving state," if the virtual viewpoint is set to any one of the three viewpoints from behind the vehicle (the virtual viewpoints corresponding to viewpoint icons 43d-43f), and the vehicle speed becomes greater than or equal to the viewpoint limit threshold, the image processing unit 13 maintains the position of the virtual viewpoint. Conversely, when driving in "slow-moving state," if the virtual viewpoint is set to a virtual viewpoint other than the three viewpoints from behind the vehicle (the virtual viewpoints corresponding to viewpoint icons 43d-43f), and the vehicle speed becomes greater than or equal to the viewpoint limit threshold, the image processing unit 13 automatically switches the position of the virtual viewpoint to one of the three viewpoints from behind the vehicle. Furthermore, when the vehicle is moving backward, and the transition from "slow-moving state" to "normal driving state" occurs, the same control content can be set simply by replacing "the three viewpoints from behind the vehicle" with "the three viewpoints from the front of the vehicle." That is, the image processing unit 13 automatically switches the virtual viewpoint to a viewpoint that reflects the vehicle's direction of travel in the three-dimensional view.

[0072] When the virtual viewpoint is set to any one of the three viewpoints (corresponding to viewpoint icons 43d-43f) from behind the vehicle under "normal driving conditions," and the vehicle speed falls below the viewpoint restriction threshold, the image processing unit 13 maintains the position of the virtual viewpoint. Additionally, the image processing unit 13 switches the viewpoint icons 43a-43h from "unselectable" to "selectable."

[0073] When the vehicle is in a "parked" state, if the vehicle speed exceeds the parking threshold while displaying a 3D image via 360-degree rotation, the controller 10 maintains the 360-degree rotation display until a user touch operation is received. However, if the vehicle speed becomes greater than or equal to the viewpoint limit threshold while displaying the 360-degree rotation display, the controller 10 stops the 360-degree rotation display and automatically switches the virtual viewpoint to a viewpoint reflecting the vehicle's direction of travel in the 3D view. Furthermore, when the vehicle is moving forward, the virtual viewpoint switching target is any virtual viewpoint from three viewpoints behind the vehicle, or the center position of the vehicle behind the vehicle. Conversely, when the vehicle is reversing, the virtual viewpoint switching target is any virtual viewpoint from three viewpoints in front of the vehicle, or the center position of the vehicle in front of the vehicle.

[0074] When the vehicle is in a "parked" state, if the vehicle speed exceeds the parking threshold when the virtual viewpoint is moved by the user's swipe operation, the controller 10 applies a swipe limit.

[0075] Furthermore, when the vehicle is in a "parked" state, and the shift position changes to something other than parking when the 3D image is displayed via a 360-degree rotating display, the controller 10 can also stop the 360-degree rotating display and automatically switch the virtual viewpoint to a viewpoint that can reflect the vehicle's direction of travel in the 3D view.

[0076] As described above, in this embodiment, the controller 10 acquires camera images from the camera 20, generates a display image based on multiple camera images, showing the vehicle from a virtual viewpoint, and outputs the display image to the display 41. Furthermore, the controller 10 moves the virtual viewpoint via user operation, detects the vehicle's speed, and restricts the movement of the virtual viewpoint based on the detected speed. Therefore, the display 41 can be configured to correspond to the vehicle's state, providing the user with an appropriate display image that matches the vehicle's status. As a result, highly operable display control of the display 41 is achieved.

[0077] However, in a display control system such as display 41, where the image of the vehicle viewed from a virtual viewpoint is displayed on the display, the user can freely move the virtual viewpoint to confirm the vehicle and its surroundings through various displayed images. However, for example, if the user can freely manipulate the position of the virtual viewpoint while the vehicle is traveling at high speed, the user might focus their attention on the screen or touch operation of display 41 rather than driving. In this embodiment, the movement of the virtual viewpoint operated by the user is restricted based on the detected vehicle speed, thus preventing the user's attention from being drawn to the operation of display 41 rather than driving.

[0078] Furthermore, in this embodiment, the controller 10 selects at least one virtual viewpoint from a plurality of virtual viewpoints located around the vehicle, generating a display image such that a three-dimensional image of the vehicle viewed from the selected virtual viewpoint is displayed on the display 41. The selectable virtual viewpoints for displaying the three-dimensional image are limited based on the detected vehicle speed. This allows virtual viewpoints unsuitable for the vehicle's driving conditions to be excluded from the selection. Therefore, for example, when the vehicle is traveling at high speed, virtual viewpoints where the direction of travel is difficult to observe can be excluded from the selection, preventing the display of images that are difficult to observe the direction of travel while the vehicle is in motion. As a result, highly operable display control of the display 41 can be achieved.

[0079] Furthermore, in this embodiment, when the vehicle is moving forward, the controller 10 restricts the selectable virtual viewpoint to a viewpoint from the rear of the vehicle, and when the vehicle is reversing, it restricts the selectable virtual viewpoint to a viewpoint from the front of the vehicle. This allows virtual viewpoints that are difficult to observe the direction of travel to be excluded from the selection, preventing the display of images that are difficult to observe the direction of travel while the vehicle is in motion. As a result, highly operable display control of the display 41 can be achieved.

[0080] Furthermore, in this embodiment, the controller 10 selects a movement restriction mode that limits the movement of the virtual viewpoint based on the detected vehicle speed. When the movement restriction mode is selected, the movement of the virtual viewpoint is restricted to one unit per operation, regardless of the amount of sliding. Conversely, when the movement restriction mode is deactivated, the virtual viewpoint moves freely according to the amount of sliding. This enables highly operable display control of the display 41.

[0081] Furthermore, in this embodiment, when the vehicle is moving forward or backward, and when the vehicle speed changes from a slow-moving state to a state with a speed higher than slow-moving, the image processing unit 13 automatically switches the virtual viewpoint to a viewpoint that can reflect the vehicle's direction of travel in the three-dimensional view. However, when switching the virtual viewpoint, the height of the virtual viewpoint can be fixed before and after the switch. Therefore, when the virtual viewpoint is switched, since the height of the three-dimensional image is fixed, it is possible to prevent the image from shifting in the height direction when switching images.

[0082] Furthermore, as a variation of this embodiment, when the vehicle changes from a parked or stationary state (parked state) to a driving state, the image processing unit 13 can automatically switch the virtual viewpoint so that the three-dimensional image in the three-dimensional view reflects the vehicle's direction of travel. That is, the image processing unit 13 can also automatically switch the virtual viewpoint to a viewpoint that can reflect the vehicle's direction of travel in the three-dimensional view.

[0083] Furthermore, in this embodiment, the controller 10 does not need to perform... Figure 6 The complete control flow shown can also be omitted. Figure 6 The control processes are executed in the order shown.

[0084] Explanation of the label

[0085] 1 Display Control System

[0086] 10 Controllers

[0087] 11 Image Acquisition Unit

[0088] 12 Vehicle Data Acquisition Department

[0089] 13 Image Processing Department

[0090] 19 Recording media

[0091] 20 cameras

[0092] 40. In-vehicle displays

[0093] Viewpoint icons 43a~43h

[0094] 44. Automatically display the Start icon

[0095] 45 Three-dimensional images

[0096] 46 Top view image

[0097] 50 vehicle speed sensor

[0098] 100 Display control device

Claims

1. A display control device comprising a controller that acquires camera images from a plurality of vehicle-mounted cameras capturing images of the area around a vehicle, generates a display image based on the plurality of said camera images when viewing the vehicle from a virtual viewpoint, and outputs said display image to a display screen, wherein, The controller moves the position of the virtual viewpoint through user input. Detect the vehicle's speed. Based on the detected vehicle speed, the movement of the virtual viewpoint operated by the user is restricted.

2. The display control device according to claim 1, wherein, The controller selects at least one virtual viewpoint from among a plurality of virtual viewpoints located around the vehicle. The display image is generated in such a way that a three-dimensional image of the vehicle viewed from the selected virtual viewpoint is displayed on the display. Based on the detected vehicle speed, the selectable virtual viewpoints used to display the 3D image are restricted.

3. The display control device according to claim 1 or 2, wherein, When the vehicle is moving forward, the controller restricts the selectable virtual viewpoint to a viewpoint from behind the vehicle. When the vehicle is reversing, the selectable virtual viewpoint is limited to a viewpoint from the front of the vehicle.

4. The display control device according to any one of claims 1 to 3, wherein, The position of the virtual viewpoint moves according to the sliding operation on the display. The controller selects a movement restriction mode that limits the movement of the virtual viewpoint based on the detected vehicle speed. When the movement restriction mode is selected, the movement of the virtual viewpoint is restricted to one unit per operation, regardless of the amount of sliding in the swipe operation. When the movement restriction mode is removed, the virtual viewpoint can move freely according to the amount of sliding.

5. The display control device according to any one of claims 1 to 4, wherein, When the vehicle changes from a parked or stationary state to a driving state, the controller automatically switches the virtual viewpoint so that the displayed image reflects the direction of the vehicle's travel.

6. A display control method, executed by a controller, for controlling the display of a monitor. In this display control method, The controller acquires camera images from multiple onboard cameras that capture images of the vehicle's surroundings. Based on multiple camera images, a display image is generated when viewing the vehicle from a virtual viewpoint. The displayed image is output to the monitor. Detect the vehicle's speed. The virtual viewpoint can be moved by user interaction. Based on the detected vehicle speed, the movement of the virtual viewpoint operated by the user is restricted.

7. A recording medium that records a control program executed by a controller. In this recording medium, The control program causes the controller to perform a process comprising the following steps: Images are obtained from multiple onboard cameras that take pictures around the vehicle; Based on multiple camera images, a display image is generated when viewing the vehicle from a virtual viewpoint; The displayed image is output to the monitor; Detect the vehicle's speed; The virtual viewpoint can be moved by user interaction; and Based on the detected vehicle speed, the movement of the virtual viewpoint operated by the user is restricted.

Images