Display Control Device

Abstract

To provide a display control device capable of improving usability.SOLUTION: The display control device includes: an image generation unit that generates a 3D image showing the surroundings of a vehicle based on images picked up by multiple in-vehicle cameras for picking up the surroundings of the vehicle and outputs images to be displayed on a display unit based on the images; an instruction determination unit that determines the instruction of a user according to the position operated by the user on the image displayed on the display device; and a viewpoint changing unit that changes viewpoint parameters related to the 3D image generation based on the instruction of the user determined by the instruction determination unit. The instruction determination unit sets multiple regions corresponding to different viewpoint parameters on a display image and determines the instruction of the user based on which region of multiple regions the position operated by the user belongs to. The viewpoint-changing unit changes the viewpoint parameter according to the instructions of the user. The instruction determination unit sets multiple regions in the 3D image after the change.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a display control device that controls the display of an image showing the surroundings of a vehicle. [Background technology]

[0002] BACKGROUND ART Conventionally, a technology is known for generating and displaying a three-dimensional image showing a vehicle and its surroundings as seen from a virtual viewpoint based on images output from multiple on-board cameras that capture the surroundings of the vehicle.

[0003] For example, Patent Document 1 discloses an image display device that includes a display control means for displaying on a screen a plurality of buttons and a composite image (three-dimensional image) associated with a plurality of reference virtual viewpoints that are the same height but different positions of the virtual viewpoint, and a detection means for detecting a user operation for changing the position of the virtual viewpoint of the composite image displayed on the screen, wherein the generation means generates a composite image viewed from a reference virtual viewpoint selected by operating the plurality of buttons, and changes the position of the virtual viewpoint of the composite image based on the user operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-076062 Summary of the Invention [Problem to be solved by the invention]

[0005] However, with this method of changing the viewpoint position of a composite image according to the button operated by the user, there was a problem that the relationship between the direction of viewpoint movement and the button was not intuitively connected, resulting in a poor usability.

[0006] An object of one aspect of the present disclosure is to provide a display control device with improved usability. [Means for solving the problem]

[0007] A display control device according to one embodiment of the present disclosure includes an image generation unit that generates a three-dimensional image showing the surroundings of a vehicle based on images captured by a plurality of onboard cameras that capture the surroundings of the vehicle, and outputs a display image to be displayed on a display device based on the three-dimensional image; an instruction determination unit that determines a user's instruction based on a position at which the user operates the display image displayed on the display device; and a viewpoint change unit that changes a viewpoint parameter related to the generation of the three-dimensional image based on the user's instruction determined by the instruction determination unit, wherein the instruction determination unit sets a plurality of areas in the display image corresponding to different viewpoint parameters, the instruction determination unit determines the user's instruction based on which of the plurality of areas the position operated by the user belongs to, the viewpoint change unit changes the viewpoint parameter in accordance with the user's instruction, and the instruction determination unit sets the multiple areas in the changed three-dimensional image. [Effects of the Invention]

[0010] According to the present disclosure, the usability can be further improved. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a schematic diagram of a vehicle according to an embodiment of the present disclosure viewed from directly above; [Figure 2] FIG. 1 is a block diagram illustrating a configuration of a display system and a display control device according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a schematic diagram illustrating a hardware configuration of a computer included in a display control device according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating a first example of a three-dimensional image according to an embodiment of the present disclosure. [Figure 5] FIG. 10 is a schematic diagram illustrating a second example of a three-dimensional image according to an embodiment of the present disclosure. [Figure 6] FIG. 10 is a schematic diagram illustrating a third example of a three-dimensional image according to an embodiment of the present disclosure. [Figure 7] Schematic diagram showing an example of a plurality of regions according to an embodiment of the present disclosure. [Figure 8] FIG. 10 is a schematic diagram illustrating another example of a plurality of regions according to an embodiment of the present disclosure. [Figure 9] 1 is a flowchart illustrating the flow of operations of a display control device according to an embodiment of the present disclosure. [Figure 10] FIG. 10 is a schematic diagram showing an example of division into a plurality of regions according to a first modification of the present disclosure; [Figure 11] FIG. 10 is a schematic diagram showing another example of division of a plurality of regions according to the first modification of the present disclosure; [Figure 12] Schematic diagram showing an insensitive region according to Modification 2 of the present disclosure. [Figure 13] FIG. 10 is a schematic diagram showing an image of a first effect process according to a third modification of the present disclosure. [Figure 14] FIG. 10 is a schematic diagram showing an image of a second effect process according to a third modification of the present disclosure; [Figure 15] FIG. 10 is a schematic diagram illustrating an image of a third effect process according to a third modification of the present disclosure. [Figure 16] FIG. 10 is a schematic diagram showing an example of viewpoint change according to Modification 4 of the present disclosure. [Figure 17] Schematic diagram showing the line of sight and operation direction on a three-dimensional image according to Modification 5 of the present disclosure. [Figure 18] FIG. 13 is a schematic diagram showing an operation direction on a three-dimensional image according to a sixth modification of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that common components in the drawings are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

[0013] First, a vehicle V of this embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of the vehicle V as seen from directly above. Note that, in this embodiment, the vehicle V will be described as a passenger car, but the vehicle type is not limited to passenger cars.

[0014] Vehicle V has multiple on-board cameras that capture images of the area around vehicle V. Specifically, as shown in FIG. 1, vehicle V has a front camera 11 that captures images in front of vehicle V (including the road surface in front), a rear camera 12 that captures images behind vehicle V (including the road surface behind), a left camera 13 that captures images to the left of vehicle V (including the road surface on the left), and a right camera 14 that captures images to the right of vehicle V (including the road surface on the right). Each of these cameras is mounted with a depression angle to capture images of the road surface. Furthermore, the viewing angle of each camera is 190 degrees or more, making it possible to capture images of the entire periphery of vehicle V with the four cameras.

[0015] In this embodiment, the number of on-board cameras is four, but the number of on-board cameras is not limited to this. Furthermore, the mounting positions of the on-board cameras are not limited to those shown in Fig. 1. For example, a side rear monitoring camera with a viewing angle of about 45 degrees may be added, and the display image may be synthesized from images captured by a total of six on-board cameras.

[0016] As shown in FIG. 1, a vehicle V includes a touch panel 20 and a display control device 100.

[0017] The touch panel 20 is an input / output device that is provided, for example, inside the vehicle cabin and that accepts various operations by a user (for example, an occupant of the vehicle V) and displays various images. The touch panel 20 can be said to be an operation accepting device and a display device.

[0018] The display control device 100 is a computer that generates a three-dimensional image (described in detail later) based on an image captured by the above-described vehicle-mounted camera and displays the image on the touch panel 20. The display control device 100 is realized by, for example, an ECU (Electronic Control Unit). Although not shown in the drawings, the display control device 100 is electrically connected to both the above-described vehicle-mounted camera and the touch panel 20. Details of the display control device 100 will be described later using FIG. 2 and subsequent figures.

[0019] The vehicle V has been described above.

[0020] Next, the configurations of the display system 1 and the display control device 100 of this embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the display system 1 and the display control device 100 of this embodiment.

[0021] 2, the display system 1 includes an imaging unit 10, a touch panel 20, and a display control device 100. The display system 1 may also be called a "vehicle surroundings monitoring device."

[0022] The imaging unit 10 corresponds to the above-mentioned vehicle-mounted cameras (that is, the front camera 11, rear camera 12, left camera 13, and right camera 14 shown in FIG. 1).

[0023] As shown in Fig. 2, the display control device 100 has an image acquisition unit 110, an image generation unit 120, an instruction determination unit 130, and a viewpoint change unit 140. Note that Fig. 2 does not limit the physical configuration of the vehicle periphery monitoring device, the number of parts, or the functional inclusion relationship. For example, there may be multiple touch panels 20, and the instruction determination unit 130 may be incorporated as one function of the viewpoint change unit 140.

[0024] 3, the display control device 100 includes, as hardware, for example, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502 that stores computer programs, and a RAM (Random Access Memory) 503. The CPU 501, ROM 502, and RAM 503 are connected via a bus 504.

[0025] Each function of the display control device 100 described in this specification is realized by the CPU 501 executing a computer program read from the ROM 502. Furthermore, this computer program may be recorded on a predetermined recording medium or provided to a user or the like via a network.

[0026] The image acquisition unit 110 acquires captured images from the imaging unit 10 (specifically, a front image captured by the front camera 11, a rear image captured by the rear camera 12, a left image captured by the left camera 13, and a right image captured by the right camera 14), and performs image processing (e.g., distortion correction, etc.) on the captured images to improve image quality.

[0027] The image generation unit 120 generates a three-dimensional image based on the captured image that has been subjected to the image processing, and outputs a display image based on the three-dimensional image. The touch panel 20 displays the display image, and the user can monitor the surroundings of the vehicle by looking at the display image.

[0028] The display image is, for example, an image obtained by superimposing a vehicle image (hereinafter simply referred to as the vehicle image) that three-dimensionally represents the vehicle V on an image that three-dimensionally represents the surroundings of the vehicle V that is generated from a captured image, and is a composite image that shows the vehicle V and its surroundings as viewed diagonally from a virtual viewpoint (hereinafter simply referred to as the viewpoint). This image that three-dimensionally represents the surroundings of the vehicle V that is generated based on the captured image may be called a three-dimensional image, or an image to which the vehicle image is added may also be called a three-dimensional image. Furthermore, since the three-dimensional image generated based on the captured image occupies a major portion of the display image, it may be said that the display control device 100 outputs a three-dimensional image as the display image. Portions of the image of the surroundings of the vehicle or the vehicle image that are closer to the viewpoint are larger on the display image, and portions that are farther from the viewpoint are smaller on the display image, so the appearance of the three-dimensional image varies depending on the position of the viewpoint.

[0029] The viewpoint in this embodiment (and the modified examples described below) is, for example, a viewpoint that is located around the vehicle V and is slightly higher than the vehicle V. Therefore, the viewpoint can be expressed as, for example, "upper right and forward," but since "upper" is common to all viewpoints, the notation "upper" will be omitted below.

[0030] Examples of three-dimensional images will now be described with reference to Fig. 4 to Fig. 8. Fig. 4 to Fig. 6 are schematic diagrams showing examples of three-dimensional images, respectively. Fig. 7 and Fig. 8 are schematic diagrams showing examples of division into a plurality of regions set in a three-dimensional image.

[0031] The three-dimensional image in Fig. 4 is an image showing an image of vehicle V looking down from a viewpoint on the right front of vehicle V. The three-dimensional image in Fig. 5 is an image showing an image of vehicle V looking down from a viewpoint directly behind vehicle V. The three-dimensional image in Fig. 6 is an image showing an image of vehicle V looking down from a viewpoint on the left rear of vehicle V.

[0032] The vehicle image A shown in FIGS. 4 to 6 is not an image based on a captured image, but an image synthesized from a three-dimensional model (e.g., a polygon model) of the vehicle V. The process of synthesizing the two-dimensional vehicle image A from the three-dimensional model of the vehicle V does not need to be performed in real time, and may be performed in advance outside the display control device 100. For example, a large number of vehicle images A synthesized on a computer outside the vehicle, each with a different viewpoint, may be stored in advance in the image generation unit 120, and one of the large number of vehicle images A may be selected according to the viewpoint selected by the viewpoint changing unit 140. Meanwhile, in FIGS. 4 to 6, a real-time image of the surroundings of the vehicle V (e.g., an image showing buildings, other vehicles, people, etc. present around the vehicle V at the time of capture) is displayed around the vehicle image A based on the captured image described above.

[0033] As shown in FIGS. 4 to 6, a plurality of regions (1) to (9) are set in the three-dimensional image. Region (9) is the region where the vehicle image A is displayed. Regions (1) to (8) around (9) display images of the surroundings of the vehicle V. Regions (1) to (8) correspond to different viewpoints.

[0034] An example of the setting of regions (1) to (9) is shown in Figure 7. Figure 7 is a schematic diagram of regions (1) to (9) viewed from directly above. As shown in Figure 7, the boundary lines (hereinafter simply referred to as boundary lines) separating regions (1) to (8) are set radially from the center of region (9). The angle between adjacent boundary lines is, for example, 45 degrees.

[0035] When a three-dimensional image is actually displayed on the touch panel 20, the positions and areas of the regions (1) to (9) set in this manner change for each three-dimensional image (which may also be called a viewpoint), as shown in Figures 4 to 6. This allows the user to feel a sense of perspective.

[0036] 4 to 6 are not displayed on the touch panel 20. On the other hand, the boundary line may or may not be displayed on the touch panel 20. For example, the boundary line may be displayed on the touch panel 20 only for a predetermined time (e.g., several seconds) from the start of display of the 3D image, or the boundary line may be displayed only when a touch is made, or the boundary line may be displayed only when the touched position is close to the boundary line. If the boundary line is not displayed, it does not interfere with the view of the vehicle surroundings image, and if it is displayed when a touch is made or when the finger is over the boundary line, it will be possible to touch a position that will be reliably determined the next time a touch is made.

[0037] Furthermore, although three 3D images corresponding to three viewpoints respectively have been described as an example here, it is assumed that 3D images corresponding to other viewpoints are also generated.

[0038] Also, although the case where the number of regions is nine has been described as an example, the number is not limited to this.

[0039] In addition, although the description has been given here of an example in which the boundary lines are set radially as shown in Fig. 7, the present invention is not limited to this. For example, the boundary lines may be configured with horizontal and vertical lines as shown in Fig. 8. Even in this case, when a three-dimensional image is actually displayed on touch panel 20, the positions and areas of regions (1) to (9) change for each three-dimensional image (viewpoint).

[0040] In addition, although the three-dimensional image here has been described as including a vehicle image A, the present invention is not limited to this. For example, the three-dimensional image may be only an image generated based on a captured image, or the three-dimensional image may be an image that includes another image (e.g., an image showing an arrow) indicating the direction of the vehicle V in place of the vehicle image A. The three-dimensional image may also be referred to as an output image.

[0041] An example of a three-dimensional image has been described above. Now, we will return to the description of FIG.

[0042] When a user performs an operation to change the viewpoint (hereinafter referred to as a viewpoint change operation) while a predetermined 3D image is displayed on the touch panel 20, the instruction determination unit 130 determines the position of the designated viewpoint. When generating a 3D image, in addition to the viewpoint, it is necessary to identify a line-of-sight direction, which indicates the direction of viewing from the viewpoint. The viewpoint and line-of-sight direction are collectively referred to as viewpoint parameters. Furthermore, since line-of-sight means direction, the line-of-sight direction may also be simply referred to as the line-of-sight. When generating a 3D image, an image of the vehicle's surroundings, centered on the vehicle V, is output as a display image, so it is preferable to always direct the line-of-sight toward the vehicle V. Based on this premise, the line-of-sight is uniquely determined once the viewpoint is determined, so the viewpoint parameter only requires information about the viewpoint. Furthermore, based on the additional premise that the viewpoint is located on a concentric circle centered on the vehicle V, the viewpoint is uniquely determined once the line-of-sight is determined, so the viewpoint parameter only requires information about the line-of-sight. Therefore, the instruction determination unit 130 may be said to determine a designated viewpoint parameter, and this viewpoint parameter may be either the viewpoint or the line-of-sight.

[0043] In this embodiment, the user can issue an instruction to change the viewpoint by touching a desired position on the three-dimensional image being displayed on the touch panel 20 with a finger or the like (an example of a viewpoint change operation). For example, when the three-dimensional image of FIG. 4 is displayed on the touch panel 20 (when the viewpoint is to the right front of the vehicle V), if the user wants to change the viewpoint to directly behind the vehicle V, the user touches area (5) on the three-dimensional image of FIG. 4. Then, based on a detection signal (a signal indicating the touched position) from the touch panel 20, the instruction determination unit 130 determines that the touched position belongs to area (5), and determines that the instructed viewpoint is directly behind the vehicle V.

[0044] The viewpoint changing unit 140 changes the viewpoint of the three-dimensional image to the viewpoint determined by the instruction determining unit 130, and displays a three-dimensional image corresponding to the changed viewpoint on the touch panel 20. At this time, the viewpoint changing unit 140 also changes a plurality of regions (specifically, positions and areas) in the three-dimensional image according to the changed viewpoint.

[0045] For example, if the 3D image being displayed is the 3D image of Fig. 4 and the viewpoint determined by the instruction determination unit 130 is directly behind the vehicle V, the viewpoint changing unit 140 changes the viewpoint from the front right of the vehicle V to directly behind it, and outputs the 3D image of Fig. 5 seen from that viewpoint. As a result, the 3D image of Fig. 5 is displayed on the touch panel 20. At this time, the viewpoint changing unit 140 also changes the images (1) to (8) of areas (1) to (8) shown in Fig. 4 seen from the viewpoint determined by the instruction determination unit 130. In other words, the positions and areas of areas (1) to (8) in Fig. 4 are changed to the positions and areas of areas (1) to (8) shown in Fig. 5.

[0046] In the present embodiment, for the sake of clarity, the functions of the display control device 100 are realized by four components, namely, the image acquisition unit 110, the image generation unit 120, the instruction determination unit 130, and the viewpoint change unit 140. However, the present invention is not limited to this. For example, the image generation unit 120 may also have the function of the image acquisition unit 110, and the viewpoint change unit 140 may also have the function of the instruction determination unit 130 (the same applies to each modified example described below).

[0047] The configurations of the display system 1 and the display control device 100 according to the present embodiment have been described above.

[0048] Next, the operation of the display control device 100 will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of the operation of the display control device 100.

[0049] 9 is started, for example, when a user performs an operation to instruct display of a three-dimensional image while no three-dimensional image is displayed on the touch panel 20. At that time, the image acquisition unit 110 acquires a captured image from the imaging unit 10 and performs predetermined image processing.

[0050] First, the image generating unit 120 determines a first viewpoint (step S1).

[0051] This first viewpoint may be a preset viewpoint, or may be the viewpoint of the three-dimensional image that was previously displayed.

[0052] Next, image generation unit 120 generates a three-dimensional image corresponding to the first viewpoint based on the captured image processed by image acquisition unit 110, and outputs the three-dimensional image to touch panel 20 (step S2). As a result, the three-dimensional image corresponding to the first viewpoint is displayed on touch panel 20, and the user can visually recognize it.

[0053] Next, the instruction determination unit 130 determines whether or not a viewpoint change operation has been performed by the user on the displayed three-dimensional image based on the presence or absence of a detection signal from the touch panel 20 (step S3). Specifically, the instruction determination unit 130 determines whether or not a position has been specified on the displayed three-dimensional image.

[0054] If the viewpoint change operation has not been performed (step S3: NO), the flow ends. Note that if the viewpoint change operation has not been performed, step S3 may be repeated until the viewpoint change operation is performed.

[0055] On the other hand, if a viewpoint change operation has been performed (step S3: YES), the instruction determination unit 130 determines the area to which the specified position belongs (step S4).

[0056] Then, instruction determination unit 130 determines the user's instruction based on the area determined in step S5, and viewpoint change unit 140 determines a second viewpoint in accordance with the user's instruction and changes the viewpoint position from the first viewpoint to the second viewpoint (step S5). Note that the second viewpoint is assumed to be different from the first viewpoint.

[0057] Next, image generation unit 120 outputs the three-dimensional image corresponding to the second viewpoint to touch panel 20 (step S6). As a result, the three-dimensional image corresponding to the second viewpoint is displayed on touch panel 20, and the user can visually recognize it.

[0058] In step S6, the instruction determination unit 130 sets the regions of the three-dimensional image corresponding to the second viewpoint to be different from the regions of the three-dimensional image corresponding to the first viewpoint. Specifically, the positions and areas of the regions are changed (for example, changed from the illustration in FIG. 4 to the illustration in FIG. 5).

[0059] The above is a series of steps, but after step S6, steps S3 to S6 may be repeated until, for example, the user issues an instruction to end the display of the three-dimensional image.

[0060] Furthermore, the first viewpoint determined in step S1 is not limited to the viewpoint looking diagonally downward at the vehicle V described above, but may be, for example, a viewpoint looking directly above the vehicle V. In this case, the image displayed in step S2 described below will not be a three-dimensional image such as those shown in FIGS. 4 to 6, but will be an image looking down on the vehicle V from directly above (for example, an image such as that shown in FIG. 7).

[0061] The operation of the display control device 100 has been described above.

[0062] As described above in detail, the display control device 100 of this embodiment includes an image generation unit 120 that generates a 3D image showing the surroundings of the vehicle V based on images captured by multiple on-board cameras (e.g., front camera 11, rear camera 12, left camera 13, and right camera 14) that capture the surroundings of the vehicle V, and displays the image on a display device (e.g., touch panel 20, the same applies below), and a viewpoint change unit 140 that, when a position on the 3D image displayed on the display device is specified by a user operation, changes viewpoint parameters based on the position and outputs a 3D image corresponding to the changed viewpoint parameters to the display device. The 3D image displayed on the display device has multiple regions (e.g., regions (1) to (8)) corresponding to different viewpoints set in it, and the viewpoint change unit 140 changes the viewpoint of the 3D image based on which of the multiple regions the position on the 3D image specified by the user belongs to, and changes the multiple regions in the 3D image according to the changed viewpoint.

[0063] Therefore, the user can intuitively instruct a change of viewpoint by specifying (specifically, touching) a desired position on the three-dimensional image, thereby further improving usability.

[0064] Furthermore, for example, while a conventional technique has been known in which an overhead image of a vehicle V viewed from directly above and a 3D image are displayed side by side and a viewpoint change operation is accepted in the overhead image, in the present embodiment, it is possible to accept a viewpoint change operation in the 3D image, thereby improving visibility. The display control device of the above embodiment may be said to include an image generation unit that generates a 3D image showing the surroundings of the vehicle based on images captured by a plurality of on-board cameras that capture the surroundings of the vehicle and outputs a display image to be displayed on a display device based on the 3D image, an instruction determination unit that determines a user instruction according to a position at which the user has operated the display image displayed on the display device, and a viewpoint change unit that changes a viewpoint parameter related to generation of the 3D image based on the user instruction determined by the instruction determination unit, wherein the instruction determination unit sets a plurality of regions corresponding to different viewpoint parameters in the display image, the instruction determination unit determines the user instruction based on which of the plurality of regions the position at which the user operation belongs, the viewpoint change unit changes the viewpoint parameter in accordance with the user instruction, and the instruction determination unit sets a plurality of regions in the changed 3D image.

[0065] The present disclosure is not limited to the above-described embodiment, and various modifications are possible without departing from the spirit of the present disclosure. Modifications will be described below.

[0066] [Variation 1] The plurality of regions in the three-dimensional image may be set so that the area of ​​each of them is equal to or greater than a predetermined threshold.

[0067] A specific example of this will be described below with reference to Figs. 10 and 11. Fig. 10 is a schematic diagram showing a first division example when the viewpoint is directly behind the vehicle V. Fig. 11 is a schematic diagram showing a second division example when the viewpoint is directly behind the vehicle V. Note that Figs. 10 and 11 show the states in which each region is viewed from directly above. Also, vehicle image A is not shown in Figs. 10 and 11.

[0068] In the three-dimensional image shown in FIG. 5, regions (1) to (9) are set as shown in FIG. 10. In this setting, the area of ​​region (5) close to the viewpoint is large, while the area of ​​region (1) far from the viewpoint is small. This makes it difficult for the user to touch region (1) correctly. For example, the user may touch region (2) instead of region (1) when intending to touch region (1).

[0069] Therefore, for example, regions (2) and (8) adjacent to region (1) may be integrated into region (1), widening region (1) as shown in FIG. 11 , or the position of the boundary line may be adjusted. For example, if there is a region whose area is less than a predetermined threshold, the viewpoint change unit 140 (or the image generation unit 120) may integrate that region with adjacent regions to reduce the total number of regions so that the area is equal to or greater than the threshold, or may reduce the total number of regions in advance and set boundary lines so that the area of ​​each region is equal to or greater than the threshold. When the viewpoint is in region (5), operations to move the viewpoint left or right are frequently performed by 45 degrees left or right, 90 degrees left or right, or 180 degrees left or right, but operations to move the viewpoint left or right 135 degrees left or right are rarely performed. Therefore, there is no practical problem even if regions (2) and (8) that move left or right 135 degrees left or right are eliminated.

[0070] This makes it easier for the user to touch when performing a viewpoint change operation, prevents erroneous touching, and improves usability.

[0071] [Variation 2] In the plurality of regions in the three-dimensional image, the boundary between adjacent regions may be set as an insensitive region that does not accept user operations. For example, the boundary between adjacent regions in the plurality of regions is set as an insensitive region that does not accept user operations, and when the user operates the insensitive region, the instruction determination unit does not determine the user's instruction.

[0072] A specific example of this will be described below with reference to Fig. 12. Fig. 12 is a schematic diagram showing an example in which an insensitive area is set in the example of dividing the area shown in Fig. 8. Fig. 12 shows each area as viewed from directly above. Also, vehicle image A is not shown in Fig. 12.

[0073] 12, an insensitive area B is set at each boundary line. Each insensitive area B is wider than each boundary line shown in FIG.

[0074] When the user touches the insensitive area B, the viewpoint changing unit 140 prevents the viewpoint from being changed, the three-dimensional image from being changed based on the change, and the multiple areas from being changed.

[0075] For example, if the user touches insensitive area B adjacent to area (5) in an attempt to touch a position within area (5) while the three-dimensional image of FIG. 4 is being displayed, the viewpoint changing unit 140 prevents the user from changing the viewpoint to directly behind the vehicle V. Furthermore, the viewpoint changing unit 140 maintains the display of the three-dimensional image of FIG. 4 without switching to the display of the three-dimensional image of FIG. 5. Furthermore, since the display of the three-dimensional image of FIG. 4 is maintained, the viewpoint changing unit 140 maintains the positions and areas of areas (1) to (9) in the three-dimensional image of FIG. 4 without changing them to the positions and areas shown in FIG. 5.

[0076] This prevents the user from touching an area adjacent to a desired area when they think they have touched that area, resulting in an undesired change of viewpoint.

[0077] In addition, the insensitive area B may be displayed temporarily so that it can be seen by the user when the 3D image is displayed. Alternatively, it may be displayed only when the user touches it. For example, if the user does not operate the insensitive area, the boundary line and the insensitive area may not be displayed, and if the user operates the insensitive area, the boundary line or the insensitive area may be displayed. In this case, to improve the visibility of the insensitive area B, the insensitive area B may be displayed with a brightness different from that of other areas to make it stand out. This brightness is preferably at a level that produces an afterimage effect so that the user can recognize the position of the insensitive area B even after the display of the insensitive area B disappears.

[0078] In addition, insensitive area B may be temporarily displayed only when the user touches insensitive area B. In this case, too, it is preferable to display insensitive area B at a brightness different from that of other areas to improve visibility. If the insensitive area is not displayed normally, the user's view of the vehicle surroundings image is not obstructed, and if the insensitive area is displayed when the user touches or when the insensitive area is touched, the user will be able to touch a position that will be correctly determined the next time they touch.

[0079] [Variation 3] A dead time period may be set in which a viewpoint change operation is not accepted until a predetermined time has elapsed since the viewpoint change operation was performed. If a user operation is performed during this dead time period, the instruction determination unit does not determine that the user's instruction is true, and therefore the viewpoint is not changed.

[0080] In this case, even if a viewpoint change operation is performed during the dead time period, the viewpoint change unit 140 changes the viewpoint, changes the 3D image based on that viewpoint, and changes the multiple regions based on the viewpoint change operation performed before the dead time period. In other words, the viewpoint change unit 140 invalidates the viewpoint change operation performed during the dead time period.

[0081] As a result, even if the user unintentionally performs consecutive viewpoint change operations, the subsequent viewpoint change operations are invalidated, thereby preventing the viewpoint from being changed to an unintended one.

[0082] Furthermore, during the dead time period, effect processing may be performed on the displayed 3D image. A specific example of this will be described below with reference to FIGS.

[0083] First, the first effect processing will be described with reference to Fig. 13. Fig. 13 is a schematic diagram showing an image of the first effect processing. In the figure, an arrow pointing from left to right indicates elapsed time. Also, in the figure, an arrow pointing from bottom to top indicates the timing at which a viewpoint change operation (specifically, a touch operation) is performed by the user. Also, in the figure, a double-headed arrow pointing left and right indicates a dead time period.

[0084] As shown in FIG. 13, if a touch operation is performed while a pre-viewpoint change image (a three-dimensional image before the viewpoint is changed) is displayed on touch panel 20, a dead time period starts from the start of the operation. During this dead time period, viewpoint change unit 140 controls touch panel 20 to fade out the three-dimensional image before the viewpoint change, and, once this is complete, to fade in the next image to be displayed after the viewpoint change (a three-dimensional image after the viewpoint has been changed). As a result, during the dead time period, the image after the viewpoint change is displayed so that fading in begins simultaneously with the completion of fading out the pre-viewpoint change image. For example, during the dead time period, either one or both of the effects of fading out the three-dimensional image before the viewpoint parameter change or fading in the three-dimensional image after the viewpoint parameter change may be performed.

[0085] Next, the second effect processing will be described with reference to Fig. 14. Fig. 14 is a schematic diagram showing an image of the second effect processing. The arrows in Fig. 14 are the same as those in Fig. 13.

[0086] As shown in FIG. 14, if a touch operation is performed while the image before the viewpoint change is displayed on the touch panel 20, a dead time period starts from the start of the operation. During this dead time period, the viewpoint change unit 140 increases the transmittance of the 3D image before the viewpoint change, while decreasing the transmittance of the image after the viewpoint change that will be displayed next. For example, it may be said that the blend ratio (mix ratio) of the two images is continuously changed. As a result, during the dead time period, the image before the viewpoint change gradually disappears while the image after the viewpoint change gradually appears. In other words, this display control device continuously changes the blend ratio between the 3D image before the viewpoint parameter change and the 3D image after the viewpoint parameter change during the dead time period.

[0087] Next, the third effect processing will be described with reference to Fig. 15. Fig. 15 is a schematic diagram showing an image of the third effect processing. The arrows in Fig. 15 are the same as those in Fig. 13.

[0088] As shown in Fig. 15, if a touch operation is performed while the pre-viewpoint change image is displayed on the touch panel 20, a dead time period starts from the start of the operation. During this dead time period, the viewpoint change unit 140 controls the touch panel 20 to reduce the brightness of the 3D image before the viewpoint change and increase the brightness of the image after the viewpoint change that is displayed next. As a result, during the dead time period, the pre-viewpoint change image disappears and the post-viewpoint change image appears. As shown in Fig. 15, if the brightness is changed significantly at the beginning and end of the dead time period, it is easier to recognize the start and end of the effect than if the brightness were changed continuously.

[0089] It should be noted that instead of brightness, transparency may be changed. For example, during the dead time period, viewpoint change unit 140 controls touch panel 20 to increase the transparency of the 3D image before the viewpoint change and decrease the transparency of the image after the viewpoint change that is subsequently displayed. As a result, during the dead time period, the image before the viewpoint change disappears and the image after the viewpoint change appears.

[0090] The first to third effect processes have been described above. Since the device does not accept user operations during the dead time period, the user may feel that the device is not responsive. However, by performing any of the first to third effect processes described above, the device can appear to be responding to user operations, which prevents the user from feeling that the device is not responsive and discourages the user from wanting to continue touching the device. Furthermore, the third effect process is more effective than the first and second effect processes because it has a greater visual impact.

[0091] [Variation 4] The viewpoint change unit 140 may continuously move the viewpoint from the viewpoint before the viewpoint change to the viewpoint after the viewpoint change during the period from when the three-dimensional image before the viewpoint change is switched to the three-dimensional image after the viewpoint change, and may output a corresponding three-dimensional image in which the viewpoint continuously moves during the transition period from when the output of the three-dimensional image before the viewpoint change is completed to when the three-dimensional image after the viewpoint change is output, and display it on the touch panel 20.

[0092] A specific example of this will be described below with reference to Fig. 16. Fig. 16 is a schematic diagram showing an example of viewpoint change. The curved arrow in the figure indicates a counterclockwise direction.

[0093] In Fig. 16, a to f indicate viewpoints. Here, an example will be described in which viewpoint a is changed to viewpoint f. In this case, in the embodiment, after a 3D image corresponding to viewpoint a is displayed, a 3D image corresponding to viewpoint f is then displayed. However, in this modification, after a 3D image corresponding to viewpoint a is displayed, 3D images corresponding to viewpoints b, c, d, and e are displayed in a transitional manner, and finally a 3D image corresponding to viewpoint f is displayed.

[0094] Viewpoints b and c are positions rotated 5 and 10 degrees counterclockwise from viewpoint a, respectively. Therefore, the 3D image corresponding to viewpoint b is an image obtained by rotating the 3D image corresponding to viewpoint a by 5 degrees counterclockwise, and the 3D image corresponding to viewpoint c is an image obtained by rotating the 3D image corresponding to viewpoint a by 10 degrees counterclockwise.

[0095] Viewpoints d and e are positions rotated 5 and 10 degrees clockwise, respectively, from viewpoint f. Therefore, the 3D image corresponding to viewpoint d is an image obtained by rotating the 3D image corresponding to viewpoint f 5 degrees clockwise, and the 3D image corresponding to viewpoint d is an image obtained by rotating the 3D image corresponding to viewpoint f 10 degrees clockwise.

[0096] That is, in this modification, three-dimensional images corresponding to the viewpoints a, b, c, d, e, and f are displayed sequentially, so that the user feels as if the viewpoint position is changing smoothly, which gives a good visual impression. Furthermore, since users tend to refrain from performing operations during the above-described transition display, even if the time when the transition display is performed is set during a dead time period when user operations are not accepted, the user will not be dissatisfied.

[0097] Furthermore, in this modification, the 3D images corresponding to positions from viewpoint c to viewpoint d are intentionally not displayed. The reason for this is that if 3D images corresponding to all positions shifted counterclockwise by 5 degrees from viewpoint a to viewpoint f were to be continuously displayed, the user might become annoyed. By intentionally not continuously displaying the images in detail, as in this modification, the user's annoyance can be avoided.

[0098] In this modification, four 3D images corresponding to viewpoints b, c, d, and e are displayed between the display of a 3D image corresponding to viewpoint a and the display of a 3D image corresponding to viewpoint f. However, the present invention is not limited to this. For example, only two 3D images corresponding to viewpoints b and c may be displayed, or only two 3D images corresponding to viewpoints d and e may be displayed. Alternatively, one or two sections in which the viewpoint position is continuously changed may be set between the direction of viewpoint c and the direction of viewpoint d. In other words, when changing viewpoint parameters in response to a user instruction, the viewpoint change unit of the display control device of modification 4 continuously changes the viewpoint within one or more predetermined ranges on a line connecting the viewpoint before the viewpoint parameter change and the viewpoint specified by the user. This line connecting the viewpoint before the viewpoint parameter change and the viewpoint specified by the user may be a curve or a straight line.

[0099] [Variation 5] In the embodiment, an example has been given in which the user's viewpoint change operation is the specification of a position on a three-dimensional image (for example, an operation of touching a desired position on a three-dimensional image), but the viewpoint change operation is not limited to this and may, for example, be the specification of a direction on a three-dimensional image.

[0100] A specific example of this will be described below with reference to Fig. 17. Fig. 17 is a schematic diagram showing an example of a line of sight and an operation direction on a three-dimensional image.

[0101] As shown in FIG. 17, regions (1) to (9) are set in the 3D image, similarly to FIG. 7. Furthermore, different line-of-sight directions (see solid arrows) are assigned to regions (1) to (8). These different line-of-sight directions serve as a basis for determining whether a user has changed their viewpoint, and may therefore be called reference directions. These reference directions are set by the instruction determination unit corresponding to the 3D image, but setting regions (1) to (9) is not essential. When the viewpoint is changed, the reference direction may be set according to the changed viewpoint. Just as the size of the regions changes when the viewpoint is changed in the example of FIG. 7, when the viewpoint is changed, the reference direction also changes, and the angular differences between the multiple reference directions become unequal.

[0102] The user performs an operation (an example of a viewpoint change operation) to specify a desired direction on such a three-dimensional image. For example, if the user wants to change the viewpoint to the left rear of vehicle V, the user swipes the finger diagonally upward to the right on the displayed three-dimensional image (see the dotted arrow in the figure). Note that, as an example, FIG. 17 illustrates a case where the swiped area is from area (5) through area (4) to area (3), but the swiping is not limited to this and may be any area on the image.

[0103] When the above-mentioned swipe is performed, the instruction determination unit 130 determines that the swipe direction is diagonally upward to the right based on the detection signal from the touch panel 20, and identifies an area (6) to which the line of sight in the direction closest to that direction is assigned. Then, based on the identified area (6), the instruction determination unit 130 determines that the position of the viewpoint to be changed is the left rear of the vehicle V. The subsequent processing by the viewpoint change unit 140 is the same as in the embodiment.

[0104] According to this modification, the user can intuitively instruct a change in viewpoint by specifying (specifically, swiping) a desired direction on the three-dimensional image, thereby further improving usability. The display control device of this modification includes an image generation unit that generates a three-dimensional image showing the surroundings of the vehicle based on images captured by a plurality of on-board cameras that capture the surroundings of the vehicle and outputs a display image to be displayed on a display device based on the three-dimensional image, an instruction determination unit that determines a user instruction according to a direction specified by a user operation on the display image displayed on the display device, and a viewpoint change unit that changes a viewpoint parameter related to generation of the three-dimensional image according to the user instruction determined by the instruction determination unit, in other words, a display control device in which the instruction determination unit sets a plurality of reference directions corresponding to different viewpoint parameters and determines which of the plurality of reference directions is closest to the direction specified by the user operation, the viewpoint change unit changes the viewpoint parameter based on the result determined by the instruction determination unit, and the instruction determination unit sets the plurality of reference directions in the three-dimensional image according to the changed viewpoint.

[0105] [Variation 6] In variant example 5, we have explained an example in which the direction is determined as a user's viewpoint change operation, assuming that multiple areas or multiple reference directions (line of sight) are set in the three-dimensional image, but this assumption does not have to be true.

[0106] A specific example of this will be described with reference to Fig. 18. Fig. 18 is a schematic diagram showing operation directions on a three-dimensional image. The viewpoint of the three-dimensional image shown in Fig. 18 is the front right of the vehicle V. For example, the initial value of the viewpoint of the three-dimensional image is directly above the vehicle, and when the user touches an area in the front right of the vehicle V, the viewpoint moves to the viewpoint shown in Fig. 18, and thereafter the viewpoint can be moved by swiping.

[0107] For example, if a user wishes to change the viewpoint to the left front of vehicle V while the three-dimensional image of Fig. 18 is being displayed, the user swipes the finger from left to right on the upper half of the three-dimensional image (the area above the dashed line in the figure). The dotted arrow C in the figure indicates the direction of the swipe. Furthermore, L1 indicates the amount of swipe operation (in other words, the amount of finger movement).

[0108] In this case, the instruction determination unit 130 determines, based on the detection signal from the touch panel 20, that an instruction has been given to move the viewpoint clockwise with the image of the vehicle image A rotating clockwise (see the curved dotted arrow). When the vehicle image A rotates clockwise, the front left of the vehicle image A is positioned in front, so the instruction determination unit 130 determines that the position of the viewpoint to be changed to is the front left of the vehicle V. The subsequent processing by the viewpoint change unit 140 is the same as in the embodiment.

[0109] If the user wishes to change the viewpoint to the left front of the vehicle V, the user may swipe the finger from right to left on the lower half of the three-dimensional image (the area below the dashed line in the figure). The dotted arrow D in the figure indicates the direction of the swipe. Furthermore, L2 indicates the amount of swipe operation (in other words, the amount of finger movement). In the fifth modification, the direction specified by the user's operation is determined by comparison with a plurality of reference directions, but in the sixth modification, the amount of operation may be compared with a plurality of thresholds to determine the amount of change in the viewpoint or line of sight.

[0110] In this case, for example, the instruction determination unit 130 determines, based on the detection signal of the touch panel 20, that an instruction has been given to move the viewpoint 45 degrees clockwise with the image of the vehicle image A rotating clockwise (see the curved dotted arrow), and determines that the position of the viewpoint to be changed to is the front left of the vehicle V.

[0111] According to this modification, the user can intuitively instruct a change of viewpoint by specifying (specifically, swiping) a desired direction on the three-dimensional image, which can further improve usability. Also, this modification does not require the premise as in modification 5, and can be implemented more easily.

[0112] In the above description, the case where the swipe is in the left-right direction has been described as an example, but the swipe may be in the up-down direction. For example, when a swipe is performed from top to bottom in the left half region of the three-dimensional image or a swipe is performed from bottom to top in the right half region of the three-dimensional image, the instruction determination unit 130 may determine that an instruction to move the viewpoint counterclockwise has been given with the image of the vehicle image A rotating counterclockwise. Then, the instruction determination unit 130 may determine the position of the viewpoint based on the rotation direction of the vehicle image A.

[0113] Furthermore, in addition to the swipe direction, either the amount of swiping (see, for example, L1 and L2) or the speed of the swipe may be taken into consideration. Specifically, the greater the amount of swiping (for example, the longer L1 or L2), the greater the amount of rotation of the vehicle image A, or the faster the speed of the swipe, the greater the amount of rotation of the vehicle image A. For example, when the viewpoint position changes from directly above to diagonally above, the image of vehicle image A is displayed larger in the horizontal direction at the bottom of the display image, whereas the image of vehicle image A is displayed compressed in the vertical direction. Therefore, when the bottom of the display image is swiped horizontally, the amount of rotation of the vehicle image A corresponding to the amount of swiping may be smaller than when the left and right sides of the image of vehicle image A are swiped vertically.

[0114] The display control device of variant example 6 can also be described as a display control device having an image generation unit that generates a three-dimensional image showing the surroundings of the vehicle based on images captured by multiple onboard cameras that capture the surroundings of the vehicle, and outputs a display image to be displayed on a display device based on this three-dimensional image, and a viewpoint change unit that, when a user swipes on the three-dimensional image displayed on the display device, changes the viewpoint parameters based on the position of the swipe and either the amount or speed of the swipe.

[0115] The above-described modified examples have been described. The modified examples described above may be combined as appropriate within the scope of the spirit of the invention. [Industrial Applicability]

[0116] The display control device of the present disclosure is useful in general for technologies for displaying three-dimensional images showing the surroundings of a vehicle. [Explanation of symbols]

[0117] 1 Display System 10. Imaging unit 11. Front camera 12 Rear camera 13 Left camera 14 Right camera 20 Touch Panel 100 Display control device 110 Image acquisition unit 120 Image generation unit 130 Instruction judgment section 140 Viewpoint change section V vehicle

Claims

1. an image generation unit that generates a three-dimensional image showing the surroundings of the vehicle based on images captured by a plurality of on-board cameras that capture the surroundings of the vehicle, and outputs a display image to be displayed on a display device based on the three-dimensional image; an instruction determination unit that determines an instruction from the user in accordance with a position at which the user operates the display image displayed on the display device; a viewpoint changing unit that changes viewpoint parameters related to generation of the three-dimensional image based on the user's instruction determined by the instruction determining unit, the instruction determination unit sets a plurality of areas corresponding to different viewpoint parameters in the display image; the instruction determination unit determines the instruction of the user based on which of the plurality of areas the position operated by the user belongs to; the viewpoint changing unit changes the viewpoint parameter in response to an instruction from the user; the instruction determination unit sets the plurality of regions in the changed three-dimensional image. Display control device.

2. A display control device according to claim 1, The plurality of regions are: The areas of the respective areas are set to be equal to or greater than a predetermined threshold value. Display control device.

3. A display control device according to claim 1, a boundary line between adjacent areas among the plurality of areas is set as an insensitive area that does not accept an operation by the user; The instruction determination unit When the user operates the insensitive area, the instruction of the user is not determined. Display control device.

4. A display control device according to claim 3, When the user does not operate the insensitive area, the boundary line and the insensitive area are not displayed, and when the user operates the insensitive area, the boundary line or the insensitive area is displayed. Display control device.

5. A display control device according to claim 1, the instruction determination unit determines an instruction from the user in accordance with a direction specified by a user operation on the display image displayed on the display device; the instruction determination unit sets a plurality of reference directions corresponding to different viewpoint parameters; the instruction determination unit determines which of the plurality of reference directions the direction specified by the user's operation is closest to; the viewpoint changing unit changes the viewpoint parameter based on the result of the determination by the instruction determining unit; the instruction determination unit sets the plurality of reference directions in the three-dimensional image according to a changed viewpoint. Display control device.

6. A display control device according to claim 5, The plurality of reference directions are: The angle difference is set to be equal to or greater than a predetermined threshold value. Display control device.

7. A display control device according to claim 1, the viewpoint changing unit further changes a viewpoint parameter based on a position of the swipe and either an operation amount or an operation speed of the swipe when the user swipes the three-dimensional image displayed on the display device. Display control device.

8. A display control device according to claim 1, when changing the viewpoint parameter in response to an instruction from the user, the viewpoint change unit continuously changes the viewpoint within one or more predetermined ranges on a line connecting a viewpoint before the change of the viewpoint parameter and a viewpoint specified by the user; Display control device.

9. A display control device according to any one of claims 1 to 8, a dead time period is set during which no operation by the user is accepted until a predetermined time has elapsed since the viewpoint parameter was changed, If the user performs an operation during the dead time period, the instruction from the user is not determined. Display control device.

10. A display control device according to claim 9, During the dead time period, either or both of the three-dimensional image before the change of the viewpoint parameter is faded out and the three-dimensional image after the change of the viewpoint parameter is faded in are performed. Display control device.

11. The display control device according to claim 9, continuously changing a blend ratio between the three-dimensional image before the viewpoint parameter is changed and the three-dimensional image after the viewpoint parameter is changed during the dead time period; Display control device.

12. The display control device according to claim 9, During the dead time period, the brightness of the three-dimensional image before the viewpoint parameter is changed is reduced, or the brightness of the three-dimensional image after the viewpoint parameter is changed is increased. Display control device.

13. The display control device according to claim 9, In the dead time period, the transparency of the three-dimensional image before the viewpoint parameter is changed is increased, and the transparency of the three-dimensional image after the viewpoint parameter is changed is decreased. Display control device.

Images