View display device and view display method

Abstract

To provide a view display device capable of easily calling a desired view.SOLUTION: A screen generation unit generates a screen including a display region for displaying a view from a first viewpoint generated by a view generation unit, in response to detecting a specific input operation on a touch screen. When a slide operation on the touch screen is detected, the view generation unit modifies camera coordinates in the first viewpoint to camera coordinates in a second viewpoint set in advance for a slide direction of the slide operation, to generate a view from the second viewpoint.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention generally relates to a technique for calling up a view from a predetermined viewpoint that is generated based on an image signal output from an imaging unit that captures an image of the surroundings of a vehicle. [Background technology]

[0002] There is known a technology for displaying on a monitor a composite image (view), such as a stereoscopic image or a bird's-eye view of the vehicle's surroundings, which is created by combining images captured by multiple cameras attached to the vehicle (see Patent Document 1). With this view, the vehicle and its surrounding images can be displayed in three dimensions by using three-dimensional coordinates as the projection surface. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2018 / 0315244 Summary of the Invention [Problem to be solved by the invention]

[0004] When calling up a conventional view, if the menu has multiple levels, it is necessary to repeatedly touch, swipe, and perform other operations to call up the view. Furthermore, in order to check the vehicle's surroundings (surrounding environment) in the view, it is necessary to adjust the viewpoint of the view after calling up the view, which requires the driver to keep their eyes on the screen for this adjustment. In addition to the many steps required to use the view, the need to keep their eyes on the screen makes it a hassle to use the view, reducing opportunities to use the view when starting to drive, parking, driving, etc.

[0005] The present invention has been made in consideration of the above points, and aims to propose a view display device or the like that can easily call up a desired view. [Means for solving the problem]

[0006] In order to solve the above problem, the present invention provides a view display device that is mounted on a vehicle and displays a view related to the vehicle on a touch screen, the view display device comprising: a view generation unit that generates a view from a predetermined viewpoint based on an image signal output from an imaging unit that images the surroundings of the vehicle; and a screen generation unit that generates a screen including a display area for displaying the view generated by the view generation unit, wherein camera coordinates indicating the position of a viewpoint for generating a view for each of a plurality of sliding directions in a sliding operation on the touch screen are set in advance, and the screen generation unit generates a screen including a display area for displaying a view from a first viewpoint generated by the view generation unit when a specific input operation on the touch screen is detected, and when a sliding operation on the touch screen is detected, the view generation unit changes the camera coordinates of the first viewpoint to camera coordinates of a second viewpoint that are set in advance for the sliding direction of the sliding operation, and generates the view from the second viewpoint. I'll I did.

[0007] In the above configuration, when a slide operation is performed in one of a plurality of predetermined slide directions, a view from a viewpoint preset for that slide direction is displayed. According to the above configuration, for example, when a user wants to check the surrounding environment that interests them, the user can easily call up a view that is close to the desired view from among the predetermined views. [Effects of the Invention]

[0008] According to the present invention, a highly convenient view display device can be realized. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of an overall configuration of a driving assistance system. [Figure 2] FIG. 1 is a diagram illustrating an example of a hardware configuration of a driving assistance system. [Figure 3] FIG. 2 is a diagram illustrating an example of a functional configuration of a view display device. [Figure 4] FIG. 2 is a diagram illustrating an example of a space model. [Figure 5] FIG. 10 is a diagram illustrating an example of a layout of a display area. [Figure 6] FIG. 10 illustrates an example of a view calling process. [Figure 7] FIG. 10 is a diagram illustrating an example of a screen transition. [Figure 8] FIG. 10 is a diagram illustrating an example of view switching. DETAILED DESCRIPTION OF THE INVENTION

[0010] (I) Embodiment An embodiment of the present invention will be described in detail below. However, the present invention is not limited to the embodiment. The view display device of this embodiment synthesizes images of the vehicle surroundings taken by multiple cameras installed outside the vehicle into a view from a point selected by the user and displays the synthesized image on a touch screen. For example, this view display device executes the following steps A) to C).

[0011] A) A step of displaying a view including a vehicle from a predetermined viewpoint using a specific touch operation as a trigger. B) detecting a user's sliding action on the touch screen; C) A step of changing the viewpoint to a view in which the vehicle portion and the surroundings in the direction corresponding to the slide operation are enlarged and displayed.

[0012] The above step C) can be carried out, for example, as follows: C1) When the above-mentioned slide operation in the upward direction of the screen is detected, change the viewpoint to a view in which the view in front of the vehicle is enlarged and displayed; C2) When the above-mentioned sliding operation in the downward direction of the screen is detected, change the viewpoint to a view in which the rear of the vehicle is enlarged and displayed; C3) When the above-mentioned slide operation to the right of the screen is detected, change the viewpoint to a view in which the right side of the vehicle is enlarged and displayed; and C4) When the above-mentioned sliding operation to the left of the screen is detected, the viewpoint is changed to a view in which the left side of the vehicle is displayed in an enlarged manner.

[0013] This view display device is characterized in that after the display in step C) above is performed, if a slide operation by the user on the touch screen is further detected, the display is sequentially changed to a view from a predetermined viewpoint. The display in step C) above is, for example, a perspective view from a viewpoint other than directly above the vehicle, and if a slide operation by the user on the touch screen is further detected, the view display device sequentially changes the display to a view from a predetermined viewpoint. Bird's-eye view When a further sliding operation by the user on the touch screen is detected, the view display device changes the viewpoint to one in which various parts are further enlarged than in step C).

[0014] In this way, this view display device is characterized by using a specific touch operation as a trigger to move coordinates by a slide operation, change the layout in conjunction with the coordinate movement, change the camera coordinates on the view in conjunction with the coordinate movement, and change the coordinates of the camera target in conjunction with the coordinate movement. With the above configuration, when a user wants to quickly check whether they will come into contact with an obstacle, they can call up the desired view from specified patterns with very few steps, which contributes to safe driving.

[0015] Next, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and some omissions and simplifications have been made as appropriate for clarity of explanation. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0016] In the following description, the same elements in the drawings are given the same numbers, and the description will be omitted as appropriate. When describing elements of the same type without distinguishing between them, the common part (the part excluding the branch number) of the reference sign including the branch number may be used, and when describing elements of the same type while distinguishing between them, the reference sign including the branch number may be used. For example, when describing cameras without distinguishing between them, they may be written as "camera 102," and when describing individual cameras while distinguishing between them, they may be written as "front camera 102-1."

[0017] The designations "first," "second," "third," etc. in this specification are used to identify components and do not necessarily limit the number or order. Furthermore, numbers used to identify components are used in different contexts, and numbers used in one context do not necessarily indicate the same configuration in another context. Furthermore, this does not prevent a component identified by a certain number from also serving the function of a component identified by another number.

[0018] 1 shows an example of the overall configuration of a driving assistance system 100. The driving assistance system 100 is mounted on a vehicle 101 and assists the vehicle 101 in driving.

[0019] Cameras 102 (an example of an imaging unit) equipped with wide-angle lenses, fisheye lenses, or the like that can observe a wide range are provided on the front, rear, left, and right sides of the vehicle 101. These cameras 102 can observe (photograph) the entire area around the vehicle 101, including the road surface.

[0020] For example, the driving assistance system 100 includes a front camera 102-1, a rear camera 102-2, a left side camera 102-3, and a right side camera 102-4.

[0021] The front camera 102-1 is a camera 102 that is provided facing the front of the vehicle 101, and is provided on a front part of the vehicle 101, such as the front bumper or front grille. The rear camera 102-2 is a camera 102 that is provided facing the rear of the vehicle 101, and is provided on a rear part of the vehicle 101, such as the rear bumper or rear garnish. The left side camera 102-3 is a camera 102 that is provided facing the left side of the vehicle 101, and is provided on a left door mirror or other part of the left side of the vehicle 101. The right side camera 102-4 is a camera 102 that is provided facing the right side of the vehicle 101, and is provided on a right door mirror or other part of the right side of the vehicle 101.

[0022] The driving assistance system 100 generates a composite image from a predetermined viewpoint based on images of the surroundings of the vehicle 101 captured by the camera 102, and presents the generated composite image to a user such as a driver. The composite image may be a perspective view image (stereoscopic image), looking down eye view image The perspective view image is an image displayed in perspective from a viewpoint other than directly above the vehicle 101, and includes a background image of the area around the vehicle 101 viewed from diagonally above or from the side, and an icon representing the vehicle 101. Bird's-eye view is an image taken from a viewpoint directly above the vehicle 101, and includes a background image of the area around the vehicle 101 viewed from above and an image of the top surface of the vehicle 101. Hereinafter, a perspective view image or a video thereof will be referred to as a "perspective view." Bird's-eye view Or the video " Bird's-eye view " Perspective view and Bird's-eye view When there is no need to distinguish between "view" and "views," the term is used.

[0023] 2 shows an example of a hardware configuration of the driving assistance system 100. The driving assistance system 100 includes an image processing device 210, a navigation device 220, a wheel speed sensor 230, and a steering angle sensor 240.

[0024] The image processing device 210 is, for example, a camera ECU (Electronic Control Unit), and is mainly configured with a microcontroller including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The image processing device 210 (an example of a view generation unit) controls the camera 102 that captures images of the surroundings of the vehicle 101, and generates a view from a predetermined viewpoint (for example, a virtual viewpoint 411 described later) based on an image signal output from the camera 102. The image processing device 210 (an example of a screen generation unit) also generates a screen including a display area for displaying the generated view. The functions of the image processing device 210 will be described later with reference to FIG. 3.

[0025] The navigation device 220 is an example of a touch screen that displays views, screens, etc. generated by the image processing device 210 and accepts input operations from the user. More specifically, the navigation device 220 includes a monitor 221 that displays images and a touch panel 222 on which the user performs input operations. The monitor 221 has the touch panel 222 superimposed on the display surface on which images, etc. are displayed. The touch panel 222 generates virtual camera information in response to the user's input operations. Note that the touch sensor of the touch panel 222 may be any sensor, such as a pressure sensor, electrostatic sensor, or optical sensor, as long as it can acquire the coordinates of a touch on the touch panel 222 (touch coordinates). The virtual camera information is information related to a virtual camera virtually installed in a predetermined spatial model onto which a view is projected. Note that the spatial model will be described later with reference to FIG. 4.

[0026] Note that, in addition to the touch panel 222, the operation unit that can be used for input operations may include various operation buttons, rotary dials, joysticks, etc. provided around the monitor 221. Additionally, the navigation device 220 may store map data or the like for route guidance, and provide route guidance to a destination point set by the user based on this map data and current position data of the vehicle 101 detected by a GPS device or the like (not shown).

[0027] Wheel speed sensor 230 is a sensor that detects the wheel speed of vehicle 101. Detected information (wheel speed) detected by wheel speed sensor 230 is input to vehicle control ECU 250. Steering angle sensor 240 detects the steering angle of vehicle 101. Steering angle sensor 240 defines the steering angle when vehicle 101 is traveling straight as a neutral position (0 degrees), and detects the rotation angle from the neutral position as the steering angle. Detected information (steering angle) detected by steering angle sensor 240 is input to vehicle control ECU 250.

[0028] Furthermore, the driving assistance system 100 includes a vehicle control ECU 250, a steering control unit 260, a throttle control unit 270, and a brake control unit 280.

[0029] The vehicle control ECU 250 is mainly configured with a microcontroller including a CPU, ROM, RAM, etc. The vehicle control ECU 250 controls the drive of a power steering actuator 261, a throttle actuator 271, and a brake actuator 281 based on information input from the image processing device 210, the navigation device 220, the wheel speed sensor 230, and the steering angle sensor 240, and executes various processes to assist the driving of the vehicle 101.

[0030] The steering control unit 260 drives a power steering actuator 261 based on vehicle control information determined in the vehicle control ECU 250 to control the steering angle of the vehicle 101. The throttle control unit 270 drives a throttle actuator 271 based on the vehicle control information determined in the vehicle control ECU 250 to control the throttle of the vehicle 101. The brake control unit 280 drives a brake actuator 281 based on the vehicle control information determined in the vehicle control ECU 250 to control the brakes of the vehicle 101.

[0031] The image processing device 210, the navigation device 220, the wheel speed sensor 230, and the steering angle sensor 240 are connected to the vehicle control ECU 250 via an in-vehicle LAN (Local Area Network) 290-1 for communicating sensor information. The steering control unit 260, the throttle control unit 270, and the brake control unit 280 are connected to the vehicle control ECU 250 via an in-vehicle LAN 290-2 for communicating vehicle information.

[0032] In this embodiment, the view display device 200 related to the display of a view will be described by taking as an example a case where the view display device 200 is configured to include an image processing device 210 and a navigation device 220. That is, the view display device 200 includes an image processing device 210, a touch panel 222, and a monitor 221 to which the touch panel 222 is attached, and is mounted on a vehicle 101. Note that the view display device 200 may or may not include a camera 102. Furthermore, the view display device 200 may include the image processing device 210, and may be configured such that the monitor 221 and the touch panel 222 are communicatively connected.

[0033] FIG. 3 shows an example of the functional configuration of the view display device 200.

[0034] The image processing device 210 includes a control unit 310 and a storage unit 320. The control unit 310 controls the entire image processing device 210. The control unit 310 may be a programmable logic device such as a CPU, FPGA, or GPU (Graphics Processing Unit), or a computing element such as an ASIC. In this case, in order to perform 3D graphic image processing at high speed, the control unit 310 preferably includes a display control unit such as a GPU specialized for image processing or a computing element capable of high-speed calculations (for example, an FPGA).

[0035] The storage unit 320 stores setting information 321 such as coordinate conversion information used in coordinate conversion processing, image processing information used in image processing, parameters, etc. Furthermore, the storage unit 320 may store a control program (image processing program). The control program is executed when the image processing device 210 is started, thereby realizing the functions of the image processing device 210 (such as the acquisition unit 311, drawing control unit 312, processing unit 313, and composition unit 314).

[0036] The acquisition unit 311 acquires virtual camera information from the input unit 331, which is a function of the touch panel 222. The acquisition unit 311 transmits the acquired virtual camera information to the rendering control unit 312. The rendering control unit 312 acquires coordinate data of all vertices of the projection plane of the spatial model, or coordinate data of the vertices of the projection plane that constitutes the projection destination (projection area) of the view, from the storage unit 320, and transmits this data together with GPU setting values ​​to the processing unit 313. The GPU setting values ​​include information on matrices used for coordinate transformation (rotation matrix, enlargement matrix, reduction matrix, translation matrix) and rendering conditions (light settings such as light intensity and direction, transmittance settings, and object texture).

[0037] The processing unit 313 performs coordinate transformation processing on the coordinate data input from the rendering control unit 312. The processing unit 313 combines the received matrices to calculate and use a model transformation matrix, a view transformation matrix, and a projection transformation matrix. The model transformation matrix is ​​a transformation matrix that is responsible for moving, rotating, and scaling the spatial model itself. The view transformation matrix is ​​a matrix that transforms coordinates so that the position of the virtual camera is the origin. The projection transformation matrix is ​​a transformation matrix that creates a sense of perspective. The processing unit 313 performs coordinate transformation by multiplying these matrices by the coordinate data of each vertex. Note that the coordinate data and coordinate transformation information (including the transformation formula) are associated by a table or the like.

[0038] The synthesis unit 314 acquires the coordinate data converted by the processing unit 313 and an image signal (captured image) from the camera 102, and synthesizes a view to be displayed on the monitor 221 based on these. When synthesizing the views, the synthesis unit 314 can use a known technique such as texture mapping. The synthesis unit 314 transmits a display control signal to the navigation device 220 to display a screen in which a display area for displaying the synthesized view has been set in accordance with an input operation (touch operation, touch coordinates, slide direction of slide operation, etc.) on the touch panel 222. The output unit 332 of the navigation device 220 displays a screen (view) on the monitor 221 based on the display control signal from the image processing device 210.

[0039] FIG. 4 shows an example of a space model (space model 400) for generating a perspective view.

[0040] The space model 400 includes a hemispherical (bowl-shaped) projection surface 410 that extends hemispherically around a vehicle model 401 representing the vehicle 101. On the projection surface 410, the center of the hemisphere is the zenith, the vertical direction passing through the zenith is the Y axis, one axis perpendicular to the Y axis is the X axis, and an axis perpendicular to the X axis and the Y axis is the Z axis.

[0041] In the space model 400, a virtual camera that is virtually installed can be set at any position in the three-dimensional space of the projection surface 410. By operating the touch panel 222, the user can set a desired virtual camera position and a desired line-of-sight direction (display direction) when a perspective view is displayed on the monitor 221. For example, by operating the touch panel 222 in the up, down, left, or right directions (touch, slide, drag, swipe, flick, etc.), or by operating a rotary dial or the like, the position of the virtual camera can be specified and a desired line-of-sight direction can be selected. The touch panel 222 detects the user's operation, generates virtual camera information, and transmits it to the acquisition unit 311.

[0042] The virtual camera information includes the position (x, y, z) of the virtual camera on the space model 400, the line of sight direction of the virtual camera, the viewing angle of the virtual camera, etc. The line of sight direction is set as a direction vector calculated from the position and angle (roll angle, pitch angle, yaw angle) of the camera 102, or a vector determined from the position of the virtual camera and the point of gaze of the virtual camera. The angle and point of gaze are determined from the operation direction and amount of operation of the touch panel 222. The point of gaze of the virtual camera is a coordinate indicating the location that the user wants to look at.

[0043] Here, it is required to be able to instantly check the surrounding environment of interest with a simple operation when starting to drive, parking, driving, etc. The view display device 200 is provided with preset virtual viewpoints 411, and the surrounding environment of interest can be checked by switching to the desired virtual viewpoint 411 with a single touch operation.

[0044] The virtual viewpoint 411 is information that can set an image captured by the virtual camera, and includes information on the position (x, y, z) of the virtual camera and the line-of-sight direction of the virtual camera. Note that the viewing angle of the virtual camera is predetermined by the camera 102 to be used, and therefore may be separately defined, or may be included in the virtual viewpoint 411.

[0045] In addition, Bird's-eye viewThe difference with respect to (1) is that the spatial model is a plate-shaped projection surface, but since it is basically the same as the perspective view, its explanation will be omitted.

[0046] 5 shows an example of the layout of a view display area 510 when a slide operation is performed upward on the touch panel 222 after a predetermined screen (screen 500) is displayed. Note that the following description takes as an example a case where the view display area 510 is included in a screen different from the screen 500 (another screen). In this case, as shown in FIG. 5, a first range 501 to a third range 503 and a first subdivided range 521 to a third subdivided range 523 are associated with the touch panel 222.

[0047] When touch coordinates (current coordinates) in a slide operation on the upper surface are within first range 501, display area 510-1 is provided over the entire surface of another screen that is smaller than screen 500-1.

[0048] When the last displayed view is a perspective view, the display area 510-1 displays a perspective view from a first virtual viewpoint 411 of a predetermined perspective view, for example, a virtual viewpoint 411-7 whose camera coordinates are in the first camera coordinates of the perspective view. Bird's-eye view When this is the case, the display area 510-1 displays a predetermined Bird's-eye view The first virtual viewpoint, for example, the camera coordinates are Bird's-eye view From a virtual viewpoint (not shown) in the first camera coordinates Bird's-eye view is displayed.

[0049] When the touch coordinates in the slide operation on the upper surface are within second range 502, display area 510-2 is provided, which is deformed to have a vertical length shorter in the upward direction than display area 510-1.

[0050] When the last displayed view is a perspective view, a perspective view from a predetermined second virtual viewpoint 411 of the perspective view is displayed in the display area 510-2. The second virtual viewpoint 411 is, for example, a virtual viewpoint 411-8 at the second camera coordinates of the perspective view (camera coordinates where the angle between the vehicle 101 and the virtual camera is maintained but the shooting range is widened) that are farther from the vehicle model 401 than the first camera coordinates of the perspective view. Note that the second camera coordinates may be camera coordinates where the angle between the vehicle 101 and the virtual camera is widened but the shooting range is maintained. When the last displayed view is Bird's-eye view When this is the case, the display area 510-2 displays a predetermined Bird's-eye view from a second virtual viewpoint Bird's-eye view The second virtual viewpoint is, for example, a virtual camera whose coordinates are: Bird's-eye view is farther from the vehicle model 401 than the first camera coordinate of Bird's-eye view The virtual viewpoint is in the second camera coordinate system.

[0051] When the touch coordinates in the slide operation on the upper surface are within third range 503, display area 510-3 is provided, which is deformed to have a vertical length shorter in the upward direction than display area 510-2.

[0052] In the display area 510-3, views from different virtual viewpoints are displayed depending on the position of the touch coordinates. When the touch coordinates are within the first subdivision range 521 and the last displayed view is a perspective view, a perspective view from a third virtual viewpoint 411 of a predetermined perspective view is displayed in the display area 510-3. The third virtual viewpoint 411 is, for example, a virtual viewpoint 411-9 whose camera coordinates are in the third camera coordinates of the perspective view that are farther from the vehicle model 401 than the second camera coordinates of the perspective view. When the touch coordinates are within the first subdivision range 521 and the last displayed view is a perspective view, a perspective view from a third virtual viewpoint 411 of a predetermined perspective view is displayed in the display area 510-3. Bird's-eye view When this is the case, the display area 510-3 displays a predetermined Bird's-eye view from a third virtual perspective Bird's-eye view The third virtual viewpoint is, for example, a virtual camera whose coordinates are: Bird's-eye viewis farther from the vehicle model 401 than the second camera coordinate of Bird's-eye view The virtual viewpoint is located at the third camera coordinate.

[0053] When the touch coordinates are within the second sub-range 522, the camera coordinates are displayed in the display area 510-3. Bird's-eye view from the virtual viewpoint at the fourth camera coordinate Bird's-eye view When the touch coordinates are within the third sub-range 523, the camera coordinates are displayed in the display area 510-3 as follows: Bird's-eye view is closer to the vehicle model 401 than the fourth camera coordinate of Bird's-eye view from the virtual viewpoint at the fifth camera coordinate Bird's-eye view is displayed. The virtual viewpoint of the fifth camera coordinate is a virtual viewpoint at a camera coordinate where the angle between the vehicle 101 and the virtual camera is maintained, but the shooting range is narrowed. Note that the fifth camera coordinate may be a camera coordinate where the angle between the vehicle 101 and the virtual camera is narrowed, but the shooting range is maintained.

[0054] 5, first range 501 to third range 503 have been described as an example, but the number of such ranges is not limited to three and may be two, or four or more. Also, first subdivision range 521 to third subdivision range 523 have been described as an example, but the number of such subdivision ranges is not limited to three and may be two, or four or more.

[0055] 5 has described the layout of display area 510 when a slide operation is performed in an upward direction on touch panel 222. By replacing the above-described directions with other directions (downward, right, left, diagonally upper right, diagonally lower right, diagonally upper left, diagonally lower left, etc.), the layout of display area 510 when a slide operation is performed in other directions will be obtained, and therefore description thereof will be omitted.

[0056] For example, the multiple sliding directions are defined as an upper direction associated with a sliding operation to the upper side of the touch panel 222, a lower direction associated with a sliding operation to the lower side of the touch panel 222, a right direction associated with a sliding operation to the right side of the touch panel 222, and a left direction associated with a sliding operation to the left side of the touch panel 222. In this case, camera coordinates indicating the position of a virtual viewpoint for generating a view of the upper direction are provided, where the front of the vehicle 101 is displayed as a target. Furthermore, camera coordinates indicating the position of a virtual viewpoint for generating a view of the lower direction are provided, where the rear of the vehicle 101 is displayed as a target. Furthermore, camera coordinates indicating the position of a virtual viewpoint for generating a view of the right direction are provided, where the right side of the vehicle 101 is displayed as a target. Furthermore, camera coordinates indicating the position of a virtual viewpoint for generating a view of the left direction are provided, where the left side of the vehicle 101 is displayed as a target. According to the above configuration, for example, when a user wants to check a view of the surrounding environment that concerns them, they can easily call up a view that is closest to their intended purpose from among the forward view, rearward view, right view, and left view.

[0057] In this way, in the view display device 200, camera coordinates indicating the position of a virtual viewpoint for generating a view of each slide direction are provided in advance for each of a plurality of slide directions in a slide operation on the touch panel 222. Furthermore, a plurality of camera coordinates indicating the position of a virtual viewpoint for generating a view of each slide direction are provided for each slide direction.

[0058] 6 shows an example of the view call process. The view call process is started when an operation to display a predetermined screen is accepted in order to use a predetermined function in the driving assistance system 100. The predetermined screen may be any screen, such as a map screen, a road guidance screen, an audio screen, or a menu screen.

[0059] In S601, the navigation device 220 displays a predetermined screen on the monitor 221.

[0060] In S602, the image processing device 210 determines whether or not a first operation has been detected on the touch panel 222. If the image processing device 210 determines that the first operation has been detected, it shifts the process to S603, and if it determines that the first operation has not been detected, it shifts the process to S602.

[0061] The first operation is a specific input operation on the touch panel 222. The first operation is an operation of touching the touch panel 222 with multiple fingers, an operation of pressing the touch panel 222, an operation of pressing the touch panel 222 for a long time, or the like.

[0062] In S603, the image processing device 210 generates a display control signal for displaying a screen (another screen) different from the predetermined screen including a display area for displaying a view of the entire vehicle 101, and transmits the generated display control signal to the navigation device 220. The view (perspective view or Bird's-eye view ) is displayed. Note that an icon representing the vehicle 101 may be displayed instead of the view.

[0063] In S604, the image processing device 210 determines whether the touch coordinates are within the first movement range. If the image processing device 210 determines that the touch coordinates are within the first movement range, it proceeds to S605, and if the image processing device 210 determines that the touch coordinates are not within the first movement range, it proceeds to S606.

[0064] The first movement range is, for example, a range in which the size (layout) of the display area of ​​the view is changed depending on the position of the touch coordinates, and when the slide operation is performed upward, for example, it is the first range 501, the second range 502, or the first subdivided range 521.

[0065] In S605, the image processing device 210 sets the layout of the display area according to the slide direction of the slide operation and the position of the touch coordinates, sets the camera coordinates according to the slide direction of the slide operation and the position of the touch coordinates, and generates a view in which the target of the virtual camera is the entire vehicle 101. The image processing device 210 generates a display control signal for displaying the generated view in the set layout, and transmits the signal to the navigation device 220.

[0066] In S606, the image processing device 210 determines whether the touch coordinates are within the second movement range. If the image processing device 210 determines that the touch coordinates are within the second movement range, it proceeds to S607, and if the image processing device 210 determines that the touch coordinates are not within the second movement range, it proceeds to S608.

[0067] The second moving range is set as the target of the virtual camera to the part of the vehicle 101 in the direction of the slide. Bird's-eye view For example, if the slide operation is performed in the upward direction, the range is the second subdivided range 522.

[0068] In S607, the image processing device 210 captures a target corresponding to a slide operation in a predetermined direction from directly above. Bird's-eye view Set the camera coordinates of the target Bird's-eye view The target corresponding to the upward sliding operation is a front part of the vehicle 101. The target corresponding to the downward sliding operation is a rear part of the vehicle 101. The target corresponding to the rightward sliding operation is a right part of the vehicle 101. The target corresponding to the leftward sliding operation is a left part of the vehicle 101.

[0069] In this way, when the touch coordinates of the slide operation on the touch panel 222 are within the first movement range, the image processing device 210 generates, for example, a perspective view capturing the vehicle 101 from an oblique angle. Also, when the touch coordinates are within the second movement range that is farther in the slide direction from the edge of the touch panel 222 in the opposite direction to the slide direction of the slide operation than the first movement range, the image processing device 210 generates a perspective view capturing the vehicle 101 from directly above. Bird's-eye view According to the above configuration, even if the previously activated view was a perspective view, by proceeding with the slide operation, Bird's-eye view You can switch to the Bird's-eye view When you want to check, Bird's-eye view can be called with minimal effort.

[0070] In S608, the image processing device 210 determines whether the touch coordinates are within the third movement range. If the image processing device 210 determines that the touch coordinates are within the third movement range, it proceeds to S609, and if the image processing device 210 determines that the touch coordinates are not within the third movement range, it proceeds to S604.

[0071] The third movement range is based on the position of the touch coordinates. Bird's-eye view For example, when the slide operation is performed in the upward direction, the third subdivision range 523 is selected.

[0072] In S609, the image processing device 210 captures a target corresponding to a slide operation in a predetermined direction from directly above, and outputs a touch image according to the position of the touch coordinates. Bird's-eye view The camera coordinates were set and the target was enlarged. Bird's-eye view Generate.

[0073] In this way, for example, when the touch coordinates of the slide operation on the touch panel 222 are within the second movement range or the third movement range, the image processing device 210 sequentially changes the camera coordinates of the virtual viewpoint to camera coordinates where the parts of the vehicle 101 in the slide direction are enlarged and displayed as the touch coordinates move away from the end of the touch panel 222 in the direction opposite to the slide direction of the slide operation. According to the above configuration, for example, the view in a predetermined slide direction is sequentially enlarged and displayed, so that the user can quickly determine whether the view is close to what he or she intended.

[0074] In S610, the image processing device 210 determines whether or not a second operation has been detected on the touch panel 222, etc. If the image processing device 210 determines that the second operation has been detected, it proceeds to S611, and if it determines that the second operation has not been detected, it proceeds to S604.

[0075] The second operation is an operation for confirming the first operation, such as an operation of removing a finger from the touch panel 222 or an operation of turning the rotary dial to switch ON / OFF.

[0076] In S611, the image processing device 210 fixes the layout of the display area (confirms the view) and ends the view calling process.

[0077] The view call process is not limited to the above-described process. For example, the processes of S604 and S605 do not have to be provided.

[0078] In this way, the image processing device 210 generates a separate screen including a display area for displaying a view from a first viewpoint when a first operation on the touch panel 222 is detected. Furthermore, when a slide operation on the touch panel 222 is detected, the image processing device 210 changes the camera coordinates of the first viewpoint to the camera coordinates of a second viewpoint predefined for the slide direction of the slide operation, and generates a view from the second viewpoint. With the above configuration, for example, when a user slides to a slide direction that the user wants to check from among multiple slide directions, a view from a virtual viewpoint corresponding to the slide direction is displayed. With the above configuration, for example, when a user wants to check a surrounding environment that interests the user, the user can easily call up a view that is close to the desired view from among the specified views. For example, if the user is concerned about the left side while driving, the user can use the view generated in S607 to check whether a motorcycle is passing by on the left side. Also, for example, if the user wants to check the situation strictly when driving at low speed or starting, the user can use the view generated in S609 to check whether there are any children in front or whether there are any animals or bicycles behind. In this way, the view can be called up depending on the level of confirmation, providing appropriate support for driving.

[0079] FIG. 7 shows an example of a screen transition when a view is called.

[0080] A screen 710 is an example of a predetermined screen, and in this example, a map screen is shown. When a touch operation (first operation) is performed on the screen 710 with two fingers 701, the screen transitions from the screen 710 to a screen 720.

[0081] On the screen 720, a view (a perspective view in this example) showing the entire vehicle 101 is displayed on another screen 721. By displaying the screen 721 smaller than the screen 720, an affordance is provided that the screen 721 can be attracted to and moved by two fingers 701.

[0082] For example, when an operation of releasing two fingers 701 from touch panel 222 (second operation) is detected, the screen transitions from screen 721 to screen 730. Also, for example, when an upward sliding operation is performed and the touch coordinates are within a first movement range, the screen transitions from screen 721 to screen 740. When a leftward sliding operation is performed and the touch coordinates are within the first movement range, the screen transitions from screen 721 to screen 750. When a rightward sliding operation is performed and the touch coordinates are within the first movement range, the screen transitions from screen 721 to screen 760. When a downward sliding operation is performed and the touch coordinates are within the first movement range, the screen transitions from screen 721 to screen 770.

[0083] On the screen 730, a view showing the entire vehicle 101 is displayed across the entire screen 730. In this example, the view is a perspective view, so the user can specify the position of the virtual camera and select a desired viewing direction.

[0084] Screen 740 is a screen in which the size of display area 510 is gradually reduced upward as the sliding operation on the upper surface progresses. In this example, screen 740 itself is reduced in size. When the sliding operation on the upper surface reaches first subdivision range 521, display area 510-3 of the layout corresponding to the sliding operation on the upper surface is set, and when the sliding operation on the upper surface reaches second subdivision range 522, the screen transitions from screen 740 to screen 741.

[0085] In this way, when the touch coordinates of the slide operation on the touch panel 222 are within the first movement range on the upper surface, the image processing device 210 generates a separate screen including the display area 510 reduced on the upper surface according to the touch coordinates. According to the above configuration, the display area 510 is reduced in response to the slide operation on the upper surface, so that the user can easily guess that a view of the front of the vehicle 101 is being displayed, for example. Note that the above content is similar for other surfaces, and therefore a description thereof will be omitted.

[0086] Screen 741 shows the vehicle 101 with the frontmost part of the virtual camera as the target. Bird's-eye view When the slide operation on the upper surface reaches the third subdivision range 523, the screen transitions from screen 741 to screen 742. Screen 742 targets the frontmost part of the vehicle 101. Bird's-eye view is displayed in a display area 510-3 at a second size larger than the first size.

[0087] In this way, when the touch coordinates of the slide operation on the touch panel 222 are within a first movement range on the upper surface, the image processing device 210 generates a view targeting the entire vehicle 101. Furthermore, when the touch coordinates are within a second movement range that is farther upward from the edge of the touch panel 222 on the lower surface opposite the upper surface of the slide operation than the first movement range, the image processing device 210 generates a view targeting the forward-most part of the vehicle 101 that corresponds to the upper surface. According to the above configuration, for example, after a view of the entire vehicle 101 is displayed, a view of a part of the vehicle 101 is displayed, so that a view close to the target can be easily called up. The above content is similar for other surfaces, and therefore a description thereof will be omitted.

[0088] Screen 750 is a screen in which the size of the display area is gradually reduced to the left as the sliding operation to the left progresses. In this example, screen 750 itself is reduced in size. When the sliding operation to the left reaches a first subdivided range, a display area (left display area) of a layout corresponding to the sliding operation to the left is set, and when the sliding operation to the left reaches a second subdivided range, the screen transitions from screen 750 to screen 751.

[0089] Screen 751 shows the leftmost part of the vehicle 101 as the target of the virtual camera. Bird's-eye view In the left display area, the first size is displayed. When the leftward sliding operation reaches the third subdivision range, the screen transitions from screen 751 to screen 752. Screen 752 targets the leftmost part of the vehicle 101. Bird's-eye view This is a screen that displays the above in the left display area at a second size that is larger than the first size.

[0090] Screen 760 is a screen in which the size of the display area is gradually reduced to the right as the sliding operation to the right progresses. In this example, screen 760 itself is reduced in size. When the sliding operation to the right reaches a first subdivided range, a display area (right display area) of a layout corresponding to the sliding operation to the right is set, and when the sliding operation to the right reaches a second subdivided range, the screen transitions from screen 760 to screen 761.

[0091] Screen 761 shows the rightmost part of the vehicle 101 as the target of the virtual camera. Bird's-eye view In the right display area, the screen 761 is displayed at a first size. When the rightward sliding operation reaches the third subdivision range, the screen transitions from the screen 761 to the screen 762. The screen 762 targets the rightmost part of the vehicle 101. Bird's-eye view This is a screen that displays the above in a second size that is larger than the first size in the right display area.

[0092] Screen 770 is a screen in which the size of the display area is gradually reduced downward as the sliding operation on the lower surface progresses. In this example, screen 770 itself is reduced in size. When the sliding operation on the lower surface reaches a first subdivision range, a display area (lower display area) of a layout corresponding to the sliding operation on the lower surface is set, and when the sliding operation on the lower surface reaches a second subdivision range, the screen transitions from screen 770 to screen 771.

[0093] Screen 771 shows the rearmost part of the vehicle 101 as the target of the virtual camera. Bird's-eye view When the slide operation on the lower surface reaches the third subdivision range, the screen transitions from screen 771 to screen 772. Screen 772 targets the rearmost part of the vehicle 101. Bird's-eye view This is a screen that displays the above in the lower display area at a second size that is larger than the first size.

[0094] In this way, when the touch coordinates of a slide operation on the touch panel 222 are within a predetermined movement range in the slide direction of the slide operation, the image processing device 210 sequentially changes the camera coordinates of the first viewpoint to one of the camera coordinates related to the slide direction as the touch coordinates move in the slide direction. According to the above configuration, for example, the view of the desired slide direction is sequentially changed, so that the user can easily call up a view that is closer to the intended view.

[0095] FIG. 8 is a diagram showing an example of view switching when a slide operation is performed to the right.

[0096] The view 800 shows a perspective view that is displayed in the right display area of ​​the screen 760 when the slide operation to the right reaches the first subdivision range. When the slide operation to the right reaches the second subdivision range, the view 800 passes through the view 801 and the view 802 and becomes the view 810 ( Bird's-eye view ) and is displayed on the screen 761. Then, when the slide operation to the right reaches the third subdivision range, the view 810 changes to a view 820 (an enlarged view of the right side of the vehicle 101) via a view 811 and a view 812. Bird's-eye view ) and is displayed on the screen 762.

[0097] According to this embodiment, the multiple operations required in the past, such as displaying a screen with a hard key (menu button), selecting a menu to call up a view with a soft key, and adjusting the view to the desired viewpoint, are no longer necessary, making it possible to easily call up the desired view.

[0098] (II) Supplementary Note The above-described embodiment includes, for example, the following content.

[0099] In the above embodiment, the present invention is described as being applied to a view display device, but the present invention is not limited to this and can be widely applied to various other systems, devices, methods, and programs.

[0100] In the above-described embodiment, some or all of the programs may be installed from a program source into a device such as a computer that realizes the view display device 200. The program source may be, for example, a program distribution server connected via a network or a computer-readable recording medium (e.g., a non-transitory recording medium). In the above description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0101] Furthermore, the screens shown and described in the above-described embodiment are merely examples, and any design may be used as long as the information received is the same.

[0102] Furthermore, the screens shown and described in the above-described embodiment are merely examples, and any design may be used as long as the information presented is the same.

[0103] In addition, in the above description, information such as programs, tables, files, etc. that realize each function can be stored in a storage device such as a memory, a hard disk, or an SSD (Solid State Drive), or in a recording medium such as an IC card, an SD card, or a DVD.

[0104] Furthermore, the above-described configurations may be modified, rearranged, combined, or omitted as appropriate within the scope of the present invention. [Explanation of symbols]

[0105] 200...view display device, 210...image processing device, 221...monitor, 222...touch panel.

Claims

1. A view display device that displays a view related to a vehicle on a touch screen, a view generating unit that generates a view from a predetermined viewpoint based on an image signal output from an imaging unit that captures an image of the surroundings of the vehicle; a screen generation unit that generates a screen including a display area for displaying the view generated by the view generation unit, The screen generation unit When an operation of touching the touch screen with a plurality of fingers on which an arbitrary screen is displayed is detected, a screen including a display area displaying a perspective view of the vehicle as seen from an oblique angle is displayed; when it is detected that the plurality of fingers have been slid relative to the display area while still touching the touch screen, an overhead image of the vehicle viewed from above is displayed in the display area, and as touch coordinates move away from an edge of the touch screen in a direction opposite to the sliding direction of the sliding operation while the plurality of fingers are still touching the touch screen, a vehicle portion of the vehicle represented in the overhead image that corresponds to the sliding direction is enlarged and displayed; When it is detected that the operation of separating the multiple fingers has been performed, the layout of the display area at the time when the operation of separating the multiple fingers has been performed is fixed. View display device.

2. When multiple sliding directions for a sliding operation on the touchscreen are defined, such as an upper direction corresponding to a sliding operation to the upper side of the touchscreen, a lower direction corresponding to a sliding operation to the lower side of the touchscreen, a right direction corresponding to a sliding operation to the right side of the touchscreen, and a left direction corresponding to a sliding operation to the left side of the touchscreen, camera coordinates indicating the position of the viewpoint for generating a view of the upper direction are provided where a front part of the vehicle is displayed as a target, camera coordinates indicating the position of the viewpoint for generating a view of the lower direction are provided where a rear part of the vehicle is displayed as a target, camera coordinates indicating the position of the viewpoint for generating a view of the right direction are provided where a right part of the vehicle is displayed as a target, and camera coordinates indicating the position of the viewpoint for generating a view of the left direction are provided where a left part of the vehicle is displayed as a target. The view display device according to claim 1 .

3. A view display method executed by a view display device that generates a view from a predetermined viewpoint based on an image signal output from an imaging unit that images the surroundings of a vehicle, and displays a screen including a display area for displaying the generated view on a touch screen, When an operation of touching the touch screen with a plurality of fingers on which an arbitrary screen is displayed is detected, a screen including a display area displaying a perspective view of the vehicle as seen from an oblique angle is displayed; when it is detected that the plurality of fingers have been slid relative to the display area while still touching the touch screen, an overhead image of the vehicle viewed from above is displayed in the display area, and as touch coordinates move away from an edge of the touch screen in a direction opposite to the sliding direction of the sliding operation while the plurality of fingers are still touching the touch screen, a vehicle portion of the vehicle represented in the overhead image that corresponds to the sliding direction is enlarged and displayed; When it is detected that the operation of separating the multiple fingers has been performed, the layout of the display area at the time when the operation of separating the multiple fingers has been performed is fixed. View display method.

Images