Head-up display device

Abstract

An object of the present invention is to realize warping control whereby more natural visibility can be ensured for at least one of a depth image (oblique image) in which three-dimensional vision is assumed, and a direct-facing image (standing image) which is preferably displayed so as to face an operator and in which three-dimensional vision is not assumed in principle. When performing warping processing on a depth image AW displayed on an inclined virtual image display surface PS1a, a control unit 190 (or 195) for performing viewpoint position tracking / warping control performs warping control with the outline of an image area of the depth image as a trapezoid in which at least one set of opposite sides are parallel to each other, and, when performing warping processing on a direct-facing image SP, performs warping control with the outline of the image area of the direct-facing image as a rectangle, including a long rectangle or a square.

Description

[Technical Field]

[0001] The present invention relates to a head-up display (HUD) device that projects image display light onto a projection target such as a vehicle windshield or combiner, and displays a virtual image in front of a driver, etc. [Background technology]

[0002] In a HUD device, an image correction process (hereinafter referred to as a warping process) is known that distorts a projected image in advance so that the image has characteristics opposite to the distortion of a virtual image caused by the optical system, the curved surface shape of a windshield, etc. The warping process in a HUD device is described in, for example, Patent Document 1.

[0003] Furthermore, performing warping processing based on the driver's viewpoint position (viewpoint tracking warping) is described in, for example, Patent Document 2. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-87619 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-199385 Summary of the Invention [Problem to be solved by the invention]

[0005] The inventors have considered implementing viewpoint-tracking warping control, which updates warping parameters according to the viewpoint position of a driver (which can be broadly interpreted as a pilot, crew member, etc.), and have recognized the following new problems.

[0006] The HUD devices described in Patent Documents 1 and 2 are designed to have only a virtual image display surface that is erected perpendicular to the road surface. In this case, warping control involves making the image area rectangular (rectangle or square) and performing image correction that imparts distortion with characteristics opposite to those caused by optical components (optical systems in a broad sense, including windshields, etc.).

[0007] Meanwhile, in recent years, a HUD device (inclined surface HUD) has been proposed in which the virtual image display surface is inclined in the depth direction (forward direction of the vehicle).

[0008] The inventor proposes further improving the inclined surface HUD by incorporating an inclined surface and an elevation (including a pseudo elevation) into a single virtual image display surface, and, for example, displaying a depth image (virtual image) such as a long arrow extending forward on the road surface on a distant inclined surface, and displaying non-superimposed content (for example, a virtual image consisting of numbers, letters, etc. that are always displayed) that is not superimposed on the background, such as a vehicle speed display, on a nearby elevation surface.

[0009] In this case, for example, warping processing is required to create a sense of depth (three-dimensionality) that gives the visual impression of being superimposed on the road surface. For example, warping processing that takes parallax into account is required.

[0010] Furthermore, when driving a vehicle, the driver's viewpoint may move (shift) along the width (left-right) of the vehicle. In this case, visibility changes due to motion parallax, and if this causes adverse visual effects, it is preferable to devise a warping process to address this issue as well.

[0011] "Motion parallax" refers to the parallax that occurs when the observer's viewpoint (or the object being observed) moves. Even if the direction of the line of sight is changed by the same amount, the closer an object is, the greater its position in the field of view changes, while the farther away an object is, the less its position changes. This difference in the amount of change in position is said to make it easier to perceive perspective.

[0012] Patent Documents 1 and 2 do not consider the relationship between parallax and warping processing, nor do they consider how to deal with a viewpoint shift that occurs during warping processing.

[0013] One of the objects of the present invention is to realize warping control that can ensure more natural visibility, for example, for depth images (tilted images) that assume three-dimensional vision, or frontal images (standing images) that do not assume three-dimensional vision.

[0014] Other objects of the present invention will become apparent to those skilled in the art by referring to the following exemplary aspects and best modes, as well as the accompanying drawings. [Means for solving the problem]

[0015] In order to facilitate an understanding of the outline of the present invention, the following examples are given of embodiments according to the present invention.

[0016] In a first aspect, a head-up display device is a head-up display (HUD) device that is mounted on a vehicle and projects an image onto a projection member provided in the vehicle, thereby allowing a driver to visually recognize a virtual image of the image, an image generation unit that generates the image; a display unit that displays the image; an optical system including an optical member that reflects display light of the image and projects the image onto the projection target; a control unit that updates warping parameters according to the driver's viewpoint position in the eyebox and performs viewpoint position tracking warping control to correct the image displayed on the display unit using the warping parameters; and The control unit When performing warping processing on the depth image, warping control is performed by treating the contour of the image area of ​​the depth image, when correctly viewed as a virtual image, as a trapezoid with at least one pair of opposite sides parallel to each other; When warping is performed on a frontal image, warping control is performed by regarding the contour of the image area of ​​the frontal image as a rectangle or a rectangle including a square when the contour is correctly viewed as a virtual image.

[0017] In the first mode, the contour (outline) of the image area to be warped is a trapezoid for a depth image (e.g., a tilted image) and a rectangle (rectangle or square) for a frontal image (e.g., a standing image). A depth image is an image with depth that is displayed on a virtual image display surface on an inclined surface, for example. It may be simply referred to as a depth image or a tilted image. A frontal image is an image that is preferably displayed facing the driver. For example, it is an image displayed on a standing surface that stands upright relative to the road surface (including not only a surface that stands upright perpendicular to the road surface, but also a "pseudo-standing surface" that is at least partially tilted but can be treated as a standing surface as a whole). In the following description, it may be simply referred to as a frontal image or a standing image.

[0018] Furthermore, the viewpoint position tracking warping process can be realized, for example, by updating warping parameters (e.g., function values ​​for filters that determine polynomials, multipliers, constants, etc. for warping image correction using a digital filter) according to the viewpoint position in the eyebox, and then using the warping parameters to perform coordinate transformation on multiple coordinate points set in the image area of ​​the image to be warped, thereby providing in advance distortion with opposite characteristics to the distortion caused by optical components.

[0019] Conventionally, assuming that an image (virtual image) is displayed on a vertical surface perpendicular to the road surface, the contour of the virtual image display (in other words, the contour when correctly perceived by the viewer as a virtual image, which is assumed to be visible for convenience of explanation) corresponding to the contour (which may be the entire region or a partial region) on the display to be subjected to image correction (which may be visible and has a rectangular shape for convenience of explanation) has been uniformly rectangular (rectangular or square). However, in this embodiment, a contour (external shape) suitable for each of a depth image (e.g., an oblique image) and a frontal image (e.g., a standing image) is adopted. Note that the above-mentioned "contour when correctly perceived by the viewer as a virtual image" can also be referred to as "the contour of the 'correct image (virtual image)' or 'correct image (virtual image)' that the HUD device intends to display." If the contour of the image (virtual image) is a trapezoid, the HUD device warps the image of the contour (rectangle) of the image region, assuming the trapezoid to be the correct shape, and displays it.

[0020] For example, when a virtual image display surface of an inclined surface located in front of a vehicle is viewed from a viewpoint located in the center of the eyebox (in other words, when the driver looks from directly in front), its outline (assumed to be visible here. Also, while the outline of the entire virtual image display surface is used here, it may also be the outline of the image display area for a certain image) appears to be a trapezoid (including a parallelogram) because it is perceived with a sense of perspective, with the width narrowing as it gets further away. When the viewpoint moves (shifts) in the width direction (left and right direction) of the vehicle, unless the position of the virtual image being displayed changes, the relative position between the viewpoint and the virtual image changes, and the trapezoid is distorted and deformed to the left or right due to the influence of motion parallax. Taking these points into consideration, the outline of the image area of ​​an image (such as an image of a long arrow or an image of a center line) displayed on the virtual image display surface of an inclined surface is also basically a trapezoid.

[0021] As a result, compared to conventional cases in which the contours perceived when a virtual image is displayed (contours when correctly perceived as a virtual image) corresponding to the contours of an image area are rectangular (rectangle or square), the virtual image after warping processing (after distortions caused by a windshield, etc. are removed) has a more natural sense of perspective. A more natural three-dimensional effect is obtained, improving visibility. Furthermore, in the case of an image (virtual image) superimposed on a background such as a road surface, the change in appearance of the image (virtual image) as the viewpoint moves is perceived as the same as the road surface in the background. This is advantageous in making the driver perceive the image (virtual image) as being superimposed on the road surface (matching the road surface), and in this respect, it also leads to improved visibility of the road surface superimposed HUD.

[0022] Furthermore, for frontal images (images) displayed on elevations (including pseudo-elevations), the outline (shape) of the image area is rectangular (rectangular or square), as in the past. Vehicle speed displays and the like displayed on elevations are often always displayed in a fixed position, and it is important that numbers, symbols, letters, etc. can be read accurately, and a sense of perspective is not particularly necessary. Therefore, there is no advantage to using a trapezoid, as in the case of tilted images, and a rectangular shape is used here, as in the past.

[0023] In this embodiment, the contours of the image area of ​​the depth image (tilt image) and the contours of the image area of ​​the front image (standing image) are set separately and warped, so that different warping methods are essentially performed for each image. As a result, for the depth image (tilt image), a display with improved visibility that gives a natural sense of perspective is realized, and for the front image (standing image), a display that is easy to see and has improved recognizability (such as a display suitable for accurately grasping information) is realized.

[0024] In a second aspect dependent on the first aspect, The control unit When the viewpoint moves along the width direction of the vehicle while the warping process is being performed on the depth image, a warping control is performed to deform the trapezoid so as to reflect the deformation of the trapezoid caused by the movement; When performing warping processing on the front-view image, if the viewpoint moves along the width direction of the vehicle, warping control may be performed to deform the rectangle so as to cancel out the deformation of the rectangle caused by the movement.

[0025] In the second aspect, we consider the case where the viewpoint moves (shifts) left and right, and in this case, we apply different types of image region deformation to each of the depth image (tilted image) and the front image (standing image) taking into account the effect of parallax. As explained in the first aspect, for the depth image (tilted image), warping is performed with the image region as a trapezoid. For example, when the viewpoint is located at the center of the eyebox, the trapezoid appears to be an isosceles trapezoid (a trapezoid whose upper and lower bases are parallel and whose interior angles at both ends of the upper base are equal, and whose interior angles at both ends of the lower base are also equal). Now, we consider the case where the viewpoint shifts in the width direction (left and right direction) of the vehicle.

[0026] If the position of the displayed virtual image is fixed (more precisely, its position in a coordinate system set in real space is fixed) (for example, the virtual image of a center line is meaningless unless it is superimposed on the center of the road surface, so its position cannot be moved), then the relative positional relationship between the virtual image and the viewpoint will change if the viewpoint moves.

[0027] Even if the distortion caused by optical components is removed and an image of the same shape (a trapezoid in this case) arrives at the viewpoint before and after moving the viewpoint, the angle of incidence of light entering the human eye changes when the viewpoint moves. Therefore, the image reflected by the eye is distorted, and the human brain interprets the image based on this distorted image. Therefore, if the above isosceles trapezoid was seen before moving the viewpoint, a distorted trapezoid will be seen after moving the viewpoint.

[0028] Considering the influence of the above-mentioned image distortion on a depth image (tilt image), since the depth image (tilt image) is an image generated to create a three-dimensional effect due to parallax, the above-mentioned image distortion (the image appears distorted due to the influence of motion parallax) can naturally be expected from the beginning. As mentioned above, the motion parallax creates a natural three-dimensional effect (perspective), and therefore the above-mentioned distortion assists the natural visual perception of the depth image.

[0029] Therefore, when the viewpoint shifts to the left while viewpoint-tracking warping is being performed during image generation, the isosceles trapezoid is corrected to tilt to the left to make the perspective more natural, and then the corrected image area is corrected in advance to impart distortion with the opposite characteristics to the distortion caused by the optical components (conventional warping).This makes it possible to display a depth image with a more natural perspective (three-dimensional effect).

[0030] On the other hand, for a frontal image (standing image), the motion parallax that accompanies viewpoint movement has a negative effect on vision. To avoid reducing the recognizability of the display, it is necessary for a vehicle speed display, etc. to always appear as a standing image without any distortion.

[0031] Here, let's assume that a rectangular image (virtual image) is visible from a viewpoint located at the center of the eyebox, and the viewpoint moves left or right. In this case, even if the position of the virtual image in a coordinate system set in real space is fixed (for example, it can be assumed that a vehicle speed display is always displayed near the lower right edge of the windshield), and the light of the rectangular image arrives with distortion due to the optical system removed, the relative positional relationship between the virtual image and the viewpoint changes, and the angle of incidence of the light changes. Therefore, the rectangle that was visible before the viewpoint shifts is deformed so that it tilts to the left if the viewpoint shifts to the left, and tilts to the right if the viewpoint shifts to the right. As described above, such deformation reduces the recognizability of the display.

[0032] Therefore, in the second aspect, for example, when warping a frontal image (standing image), a motion parallax correction is performed to deform the image (the outline of the image area: rectangular) in a direction opposite to the actual deformation due to the influence of motion parallax so that deformation (deformation of a rectangle, etc.) due to such motion parallax is suppressed (preferably canceled out), and a distortion correction is performed to impart to the deformed image area a distortion with characteristics opposite to the distortion caused by the optical system. The "warping" of the present invention does not simply mean correcting distortion, but also includes a correction process that optimizes the appearance of a virtual image taking into account parallax and motion parallax, and particularly has the meaning of "correction that controls deformation of an image (virtual image) due to motion parallax." In a specific example, as described above, motion parallax correction and distortion correction are performed as a set.

[0033] As a result, when the viewpoint shifts, the light of the image that has been deformed in the opposite way to the deformation caused by motion parallax enters the human eye at that viewpoint. At this time, the deformation caused by motion parallax is canceled out by the inverse deformation that was previously applied, and when the human brain judges the image, it appears to be a rectangular (rectangular or square) image. In other words, the image remains rectangular or other such shape before and after the viewpoint shifts, and there is no change, so there is no decrease in recognition. Visibility is improved because an accurate, frontal image is always obtained.

[0034] In a third aspect dependent from the second aspect, When the control unit deforms the rectangle so as to cancel out the deformation of the rectangle in the warping control for the front-view image, The bottom side of the rectangle may be fixed and the top side may be moved to distort the rectangle. Or, The top edge may be fixed and the bottom edge may be moved to create distortion.

[0035] In the third aspect, a method of pre-deforming the image area of ​​a front-view image is adopted in which either the bottom side of a rectangle (rectangle or square) is fixed and the top side is moved to deform it, or the inverse method is adopted.

[0036] Either method can impart a deformation (distortion) to an image region that cancels out the actual deformation caused by motion parallax, and in this respect, both methods have the same effect.

[0037] However, from the driver's perspective, the top edge of the rectangle displayed in front of him is far away and the bottom edge is close, and when motion parallax is taken into account, the further away the rectangle is, the smaller the perceived change in position. Therefore, when the rectangle is transformed by moving the top edge, it is thought that the amount of movement (actual amount of transformation) can be reduced compared to when the rectangle is transformed by moving the bottom edge, and in this respect there is a difference.

[0038] In a fourth aspect dependent from the first aspect, The control unit When the viewpoint moves along the width direction of the vehicle while performing warping processing on the front-view image, The position of the front-facing image may be changed so that the virtual image of the front-facing image is fixed at a predetermined position in a viewpoint coordinate system based on the viewpoint.

[0039] In the second and third embodiments described above, the problem of distortion of the front-facing image due to a shift in viewpoint position is addressed by preliminarily providing a distortion (inverse rectangular deformation) that is opposite to the distortion (rectangular deformation) caused by motion parallax. In the fourth embodiment, however, this problem is addressed by shifting the display position of the front-facing image (virtual image) in accordance with the movement of the viewpoint.

[0040] As described above, the problem of distortion of the frontal image due to a shift in viewpoint position can arise because the display position of the frontal image (virtual image) is fixed in a coordinate system set in real space.

[0041] In this aspect, the above problem is addressed by eliminating the assumption that the display position in a coordinate system set in real space is fixed.

[0042] In other words, in this embodiment, in a viewpoint coordinate system (a coordinate system with a person's viewpoint at the center and axes set along the front-to-back, left-to-right, and up-and-down directions of the vehicle, and when the person's viewpoint (including face, etc.) moves, the coordinate system also moves in accordance with that movement), control is performed to appropriately move the position of the image (rectangular image area) on the display surface of the display unit (or the position on the image generation surface or image generation space of the image generation unit) in response to the movement of the viewpoint so that the virtual image of the front-facing image is always in the same position.

[0043] This means that in a coordinate system set in real space, the position of the virtual image moves appropriately depending on the viewpoint deviation. In this case, for example, the relative positional relationship between the virtual image of the frontal image and the person's viewpoint is always constant, so there is no need to consider motion parallax and there is no change in appearance. Therefore, this method can also solve the above-mentioned problem of image deformation due to motion parallax, which reduces recognizability.

[0044] In a fifth aspect dependent on any one of the first to fourth aspects, The control unit When displaying a mixture of the depth image and the front-facing image, image processing may be performed separately on the depth image and the front-facing image, and then the processed images may be combined to form a single image.

[0045] In the fifth aspect, when a single image contains a mixture of depth images and frontal images, a method is adopted in which each image (which can also be called a region of each image) is individually subjected to, for example, image correction specific to each image, and then the images are combined to generate a single image.

[0046] This method can be realized, for example, by image rendering. By treating the depth image (or the image region of the depth image) and the front-facing image (or the image region of the front-facing image) separately, even when different image corrections need to be applied to each image, no particular problems arise and processing can be simplified.

[0047] In a sixth aspect dependent on any one of the first to fifth aspects, The control unit an image of the depth image as viewed from a predetermined first position in the left-right direction of the eyebox; Among the regions, a region surrounded by a trapezoid in which at least one pair of opposite sides is parallel to each other is defined as a first contour. When you do, When viewed from the left side of the first position, the deformation of the trapezoid caused by the movement The upper side of the first contour is located to the right of the lower side, When viewed from the right side of the first position, the deformation of the trapezoid caused by the movement and the warping control is performed so that the top side of the first contour is positioned to the left relative to the bottom side. and The positive side as viewed from the first position in the left-right direction of the eye box versus Image area of ​​the image When the area surrounded by a rectangle including a rectangle or a square is defined as the second contour, When viewed from the left and right sides of the position, the position of the upper side of the second contour is relative to the position of the lower side. The warping control is performed so that the image does not change relative to the original image.

[0048] When the virtual image display surface is disposed above the road surface (in other words, when the virtual image display surface is disposed closer to the driver than the road surface), the motion parallax of the depth image (tilt image) differs from the motion parallax of the background, such as the road surface, on which the depth image (tilt image) is superimposed. Specifically, it is expected that the motion parallax of the depth image (tilt image) will be larger than the motion parallax of the background, such as the road surface, and the superimposition (matching) of the depth image and the background, such as the road surface, will be reduced.

[0049] In a sixth aspect, when the entire (or a part of) area surrounded by a trapezoid of the image area of ​​the depth image viewed from a first position, which is the center of the eyebox in the horizontal direction, is defined as the first contour, warping is performed to reduce distortion of the first contour caused by motion parallax so that the first contour is visible to the driver. When viewed from the left of the first position, motion parallax distorts the position of the top edge of the first contour so that it is shifted to the left relative to the position of the bottom edge. This method performs correction to reduce the shift between the top and bottom edges of the first contour. In other words, both the correction to the first contour and the correction to the second contour are corrections in the direction of reducing distortion caused by motion parallax, but the correction to the first contour is smaller than the correction to the second contour. This brings the motion parallax of the depth image closer to the motion parallax with the background, such as the road surface, thereby realizing a display with improved visibility and a natural sense of perspective.

[0050] In a seventh aspect dependent from the second aspect, When the control unit performs warping control on the depth image, if the viewpoint moves along a width direction of the vehicle, the control unit performs warping control to deform the trapezoid so as to reflect deformation of the trapezoid caused by the movement, the display image is a road surface superimposed image displayed so as to be superimposed on the road surface, When the virtual image of the road surface superimposed image is displayed on an inclined surface that is inclined relative to the road surface and the virtual image appears to be floating above the road surface, warping control involving motion parallax correction may be performed to reduce the degree of deformation of the trapezoid compared to when the virtual image does not appear to be floating above the road surface.

[0051] In the seventh aspect, for example, when the display image is a road surface superimposed image (e.g., an image of an arrow shape) that is displayed so as to be superimposed on the road surface, and the virtual image of the road surface superimposed image is displayed on an inclined surface that is inclined with respect to the road surface, and the virtual image appears to be floating above the road surface, motion parallax correction is performed to reduce the degree of deformation of the arrow (in other words, the trapezoid as the outline of the display area of ​​the arrow image) compared to when the virtual image does not appear to be floating above the road surface (when the degree of superimposition on the road surface is relatively high).

[0052] The reason for this is that the position of the virtual image of the actual arrow (the virtual image of the road surface superimposed image) is closer to the driver (viewer) than the position of the road surface on which the arrow appears to be superimposed (the visual landing position of the arrow). In this case, motion parallax occurs for the virtual image of the arrow that appears in the foreground, and motion parallax also occurs for the road surface that appears behind it. Since the road surface that appears in the background is displayed on the far side, the motion parallax is smaller, and since the virtual image of the arrow that appears in the foreground is displayed on the near side, the motion parallax is perceived to be greater.

[0053] Because the human eye perceives this relative difference, the motion parallax of the virtual arrow image is perceived as greater than when the virtual arrow image is in close contact with the road surface, for example. Therefore, the deformation (tilt) of the virtual arrow image is emphasized. In this case, motion parallax correction is performed to reduce the deformation (tilt), in other words, to suppress to some extent the trapezoidal deformation that defines the outline of the display area of ​​the arrow image (however, the deformation remains without being canceled out). This results in a display with appropriate motion parallax.

[0054] Those skilled in the art will easily understand that the exemplified embodiments according to the present invention can be further modified without departing from the spirit of the present invention. [Brief explanation of the drawings]

[0055] [Figure 1] FIG. 1(A) is a diagram illustrating an overview of warping control (conventional warping control) in a conventional HUD device that displays a virtual image on a virtual image display surface that is erected relative to the road surface, and explains the manner in which the virtual image (and virtual image display surface) displayed through warping control is distorted. FIG. 1(B) is a diagram illustrating an example of a virtual image that the driver sees through the windshield. [Figure 2] FIG. 2A is a diagram for explaining an overview of viewpoint position tracking warping control, and FIG. 2B is a diagram showing an example of the configuration of an eyebox whose interior is divided into a plurality of partial regions. [Figure 3]Figure 3(A) is a diagram showing an example of a virtual image viewed by a user through the windshield, Figure 3(B) is a diagram showing the virtual image displayed on the virtual image display surface, and Figure 3(C) is a diagram showing an example of an image displayed on the display surface of the display unit. [Figure 4] FIG. 4(A) is a diagram showing the configuration of a HUD device mounted on a vehicle and an example of a virtual image display surface, and FIGS. 4(B) and 4(C) are diagrams showing examples of a method for realizing the virtual image display surface shown in FIG. 4(A). [Figure 5] FIG. 5 is a diagram showing an example of a display method of a HUD device (parallax type HUD device or 3D HUD device) that displays a stereoscopic virtual image. [Figure 6] FIG. 6 is a diagram showing an example of viewpoint-tracking warping control in a HUD device capable of displaying at least one of a virtual image of a standing image (including a pseudo standing image) and a virtual image of a tilted image (depth image). [Figure 7] Figure 7(A) shows a case where an inclined surface (for example, an inclined image HUD area of ​​a virtual image display surface) is placed in front, and the driver's viewpoint (assuming it is a single eye) looking at it moves along the width direction (left and right direction) of the vehicle from a state where it is located in the central divided area of ​​the eye box. Figures 7(B), (C), and (D) show how the inclined surface appears (the trapezoidal shape which is the outline of the inclined surface) when the viewpoint is located in each of the left, central, and right divided areas of the eye box. Figure 7(E) is a diagram showing an example of displaying a virtual image of an arrow on an inclined surface. Figures 7(F), (G), and (H) show how the virtual image of the arrow appears (the shape of the virtual image of the arrow) when the viewpoint is located in each of the left, central, and right divided areas of the eye box. Figures 7(I) and 7(J) show how the virtual image of the arrow appears with and without motion parallax correction. [Figure 8]Figure 8(A) shows a situation where a vertical surface (including a pseudo-vertical surface, for example, a vertical image HUD area of ​​a virtual image display surface) is placed in front, and the driver's viewpoint (assumed to be a single eye) looking at it moves along the width direction (left and right direction) of the vehicle from a state where it is located in the central divided area of ​​the eye box; Figures 8(B), (C), and (D) show how the vertical surface appears (the rectangular shape that is the outline of the vertical surface) when the viewpoint is located in each of the left, central, and right divided areas of the eye box; Figures 8(E), (F), and (G) show the content of image correction applied to the displayed image to suppress deformation of the rectangular outline of the vertical surface; and Figures 8(H), (I), and (J) show how the virtual image of the arrow appears (the shape of the virtual image of the arrow) when the viewpoint is located in each of the left, central, and right divided areas of the eye box. [Figure 9] 9A to 9D are diagrams for explaining two examples of image correction (examples of deforming the bottom or top side) that are applied to a display image in order to suppress deformation of the rectangular contour of the elevation. [Figure 10] Figure 10(A) is a diagram showing the contents of warping control for an inclined image (depth image) displayed on an inclined surface, Figure 10(B) is a diagram showing the contents of warping control for an upright image (frontal image) displayed on a vertical surface, Figures 10(C) to (E) are diagrams showing an example of generating an image to be displayed on a display surface by image rendering, and Figures 10(F) to (H) are diagrams showing examples of images (visual image, visual virtual image) viewed by the driver depending on the viewpoint position. [Figure 11] FIG. 1 is a diagram illustrating an example of the configuration of an HUD device. [Figure 12] Figure 12(A) shows an example of the main configuration of a HUD device when a figurine (pseudo figurine) is displayed fixed at a predetermined position in a viewpoint coordinate system regardless of the viewpoint position, and Figure 12(B) shows that the display (virtual image) appears to move in accordance with the movement of the viewpoint. [Figure 13] FIG. 10 is a diagram illustrating an example of a procedure in viewpoint tracking warping control. [Figure 14] 14(A) and 14(B) are diagrams showing modified examples of the virtual image display surface. DETAILED DESCRIPTION OF THE INVENTION

[0056] The best mode described below is used to facilitate understanding of the present invention, and therefore, those skilled in the art should be aware that the present invention is not unduly limited by the embodiments described below.

[0057] First, an overview of the present invention (including basic matters and an overview of the HUD device) will be explained in order with reference to FIGS. 1 to 5. Details of the present invention are shown in FIG. 6 and subsequent figures. In the following explanation, a depth image is an image with depth that is displayed on a virtual image display surface of an inclined surface, for example. However, this also includes the case where the inclination angle with respect to the road surface is zero (i.e., the case where the image is superimposed on the road surface). It may be simply referred to as a depth image or an inclined image. A frontal image is an image that is preferably displayed so as to face the driver. For example, it is an image displayed on a vertical surface that stands upright relative to the road surface (including not only a surface that stands upright perpendicular to the road surface, but also a "pseudo-vertical surface" that is at least partially inclined but can be treated as a vertical surface as a whole). It may be simply referred to as a frontal image or an image.

[0058] Please refer to Fig. 1. Fig. 1(A) is a diagram for explaining an outline of warping control (conventional warping control) in a conventional HUD device that displays a virtual image on a virtual image display surface that is erected relative to the road surface, and the manner in which the virtual image (and the virtual image display surface) displayed through the warping control is distorted, and Fig. 1(B) is a diagram showing an example of a virtual image that the driver sees through the windshield.

[0059] As shown in FIG. 1A, the HUD device 100 is mounted on a vehicle (which can be broadly interpreted) 1. The HUD device 100 includes a display unit (e.g., a light-transmitting screen) 101, a reflecting mirror 103, and a curved mirror (e.g., a concave mirror, the reflective surface of which may be a free-form surface) 105 as an optical element for projecting display light. An image displayed on the display unit 101 is projected onto a virtual image display area 5 on a windshield 2 as a projection target via the reflecting mirror 103 and the curved mirror 105. Reference numeral 4 denotes a projection area. The HUD device 100 may include multiple curved mirrors. In addition to the mirror (reflective optical element) of this embodiment, or instead of some (or all) of the mirror (reflective optical element) of this embodiment, a configuration including a refractive optical element such as a lens, a diffractive optical element, or other functional optical element may be employed.

[0060] A portion of the image display light is reflected by the windshield 2 and enters the viewpoint (eye) A of the driver or the like located inside (or on) a predetermined eye box EB (here, a rectangular shape of a predetermined area), and is imaged in front of the vehicle 1, whereby a virtual image V is displayed on a virtual virtual image display surface PS corresponding to the display surface 102 of the display unit 101.

[0061] The image on the display unit 101 is distorted due to the influence of the shape of the curved mirror 105, the shape of the windshield 2, and the like. In other words, distortion occurs due to optical components including the optical system of the HUD device 100 and the windshield 2. To offset this distortion, a distortion with the opposite characteristics to the distortion is applied to the image. This predistortion type image correction is referred to herein as warping processing (warping image correction processing).

[0062] Ideally, the warping process would result in the virtual image V displayed on the virtual image display surface PS being a flat image with no curvature. However, in a large HUD device 100 that projects display light onto a wide projection area 4 on the windshield 2 and sets the virtual image display distance over a fairly wide range, for example, some distortion will inevitably remain.

[0063] In the upper left of FIG. 1(A), PS' indicated by a dashed line indicates a virtual image display surface from which distortion has not been completely removed, and V' indicates a virtual image displayed on that virtual image display surface PS'.

[0064] Furthermore, the degree or manner of distortion of the virtual image V' that remains distorted differs depending on the position of the viewpoint A on the eyebox EB. Because the optical system of the HUD device 100 is designed assuming that the viewpoint A is located near the center, the distortion of the virtual image tends to be relatively small when the viewpoint A is near the center, and tends to become greater as the viewpoint A moves closer to the periphery.

[0065] FIG. 1(B) shows an example of a virtual image V viewed by a driver through the windshield 2. In FIG. 1(B), the virtual image V has a rectangular outer shape, and is provided with a total of 25 reference points (reference pixel points or coordinate points) GD(i,j) (where i and j are both variables that can take values ​​from 1 to 5), for example, five vertically and five horizontally. A distortion having the opposite characteristics to the distortion that occurs in the virtual image V due to the warping process is applied in advance to each reference point (each coordinate point) in the image (original image). Therefore, the applied distortion and the distortion that actually occurs cancel each other out, and ideally, a non-curved virtual image V such as that shown in FIG. 1(B) is displayed.

[0066] The number of reference points GD(i,j) can be increased as needed by interpolation processing, etc. In Fig. 1(B), reference numeral 7 denotes a steering wheel.

[0067] Next, let us refer to Figure 2. Figure 2(A) is a diagram for explaining an overview of the viewpoint position tracking warping process, and Figure 2(B) is a diagram showing an example of the configuration of an eyebox whose interior is divided into multiple partial regions. In Figure 2, parts that are common to Figure 1 are given the same reference numerals (this also applies to the following figures).

[0068] As shown in FIG. 2(A), the eyebox EB is divided into a plurality of (here, nine) partial regions J1 to J9, and the position of the driver's viewpoint A is detected for each of the partial regions J1 to J9.

[0069] Display light K of an image is emitted from the projection optical system 118 of the HUD device 100, and part of the light is reflected by the windshield 2 and enters the driver's viewpoint (eye) A. When viewpoint A is within the eyebox, the driver can see a virtual image of the image.

[0070] The HUD device 100 has a ROM 210, which incorporates an image conversion table 212. The image conversion table 212 stores warping parameters WP that determine polynomials, multipliers, constants, etc. for image correction (warping image correction) using, for example, a digital filter. The warping parameters WP are provided corresponding to each of the partial regions J1 to J9 in the eyebox EB. In FIG. 2(A), the warping parameters WP(J1) to WP(J9) corresponding to each partial region are shown. Note that in the figure, only the symbols WP(J1), WP(J4), and WP(J7) are shown.

[0071] When viewpoint A moves, the position of viewpoint A among the plurality of partial regions J1 to J9 is detected. Then, one of the warping parameters WP(J1) to WP(J9) corresponding to the detected partial region is read from ROM 210 (updating the warping parameter), and warping processing is performed using that warping parameter.

[0072] Figure 2(B) shows an eyebox EB with more partial regions than the example in Figure 2(A). The eyebox EB is divided into 6 partial regions vertically and 10 partial regions horizontally, for a total of 60 partial regions. Each partial region is represented as J(X, Y), with its coordinate position in the X and Y directions as parameters.

[0073] Next, let us refer to Fig. 3. Fig. 3(A) is a diagram showing an example of a virtual image viewed by a user through the windshield, Fig. 3(B) is a diagram showing the state of a virtual image displayed on a virtual image display surface, and Fig. 3(C) is a diagram showing an example of an image displayed on the display surface of the display unit.

[0074] In Fig. 3(A), the virtual image produced by the HUD device 100 is displayed in a virtual image display area 5 within the windshield 2. In Fig. 3(A), a virtual image SP displaying a vehicle speed (displaying "120 km / h") is displayed on the near side as seen by a user (such as a driver). In a broad sense, the virtual image of the vehicle speed display SP can be called a "standing image" virtual image, in other words, a "front-facing image" virtual image G1 displayed so as to face the user directly.

[0075] Furthermore, the virtual image G1 of the "standing image (frontal image)" may be a display of non-superimposed content (such as content that is not intended to be superimposed on an object and shows, for example, the state of vehicle 1 or the situation around vehicle 1) that is always displayed in front of the user, or it may be a navigation display consisting of at least one of letters, figures, symbols, etc.

[0076] On the other hand, a curved navigation arrow AW is displayed at the back, covering the road surface 40 extending linearly ahead of the vehicle 1 and gradually rising from the side closer to the vehicle 1 to the side farther away. The display of the arrow AW provides a unique three-dimensional visual sensation and is also an aesthetically pleasing display with a sense of realism. In a broad sense, the display of this arrow AW can be referred to as a virtual image G2 of a "depth image" that is displayed as an inclined image and has depth information (distance information). In this specification, the term "curved surface" may also include a part that is flat.

[0077] The virtual image G2 of the "depth image" may be an information image including an arrow figure relating to the progress of the vehicle 1 or other vehicles, and figures other than arrows (for example, a triangular figure indicating the direction of progress, a straight line indicating the center line, etc.), and the figure may be, for example, a figure that covers the road surface 40, with at least its main part separated from the road surface 40, and that is drawn along the road surface 40 (in other words, "a figure that gives the appearance of being superimposed on the road surface," or "a figure that extends along the road surface").

[0078] In the example of FIG. 3(A), an operation unit 9 capable of switching the HUD device on / off and setting the operation mode is provided near the steering wheel (or, in a broader sense, steering handle) 7. A display device (e.g., a liquid crystal display device) 13 is provided in the center of the front panel 11. The display device 13 can be used, for example, to assist the display by the HUD device. The display device 13 may also be a composite panel having a touch panel or the like.

[0079] 3(B), when viewed from viewpoint (eye) A of a user (such as a driver) aboard vehicle (host vehicle) 1, virtual image display surface PS1, which is an imaging surface, is a curved surface that extends integrally from a near end U1 on the side closer to vehicle 1 to a far end U3 on the side farther away, and a second distance h2 between road surface 40 and far end U3 is set to be larger than a first distance h1 between road surface 40 and near end U1.

[0080] Furthermore, the virtual image display surface PS1 is divided into a vertical image HUD area Z1 (shown surrounded by a dashed line in the figure) that includes a near end U1 and displays a virtual image G1 of a "vertical image (frontal image)" on the road surface 40, and an inclined image HUD area Z2 that is located farther away than the vertical image HUD area Z1 (forward in the longitudinal direction (Z direction) of the vehicle 1) and displays a virtual image G2 of an inclined image (depth image) that is inclined toward the road surface 40 more than the virtual image G1 of the "vertical image (frontal image)." The middle area U2 is a location (or point) on the virtual image display surface PS1 that is located at the boundary between the vertical image HUD area Z1 and the inclined image HUD area Z2.

[0081] Note that the virtual image G1 of the standing image (front-facing image) can be rephrased as the first virtual image G1. Also, the virtual image G2 of the tilted image (depth image) can be rephrased as the second virtual image G2.

[0082] Also, the distance from the user's viewpoint A (or a reference point corresponding to the viewpoint A set in the vehicle 1 or the like) to the proximal end U1 of the virtual image display surface PS1 (the "imaging distance", or in other words, the "virtual image display distance") is L1, the distance to the intermediate part U2 is L2, and the distance to the distal end U3 is L3. However, the relationship L1 < L2 < L3 holds.

[0083] The length (extending range) of the virtual image display surface PS1 along the road surface 40 can be, for example, about 10 m to 30 m.

[0084] Also, as is clear from FIG. 1(B), the area of the tilted image HUD region Z2 on the virtual image display surface PS1 is set larger than the area of the standing image HUD region Z1 (however, it is not limited to this).

[0085] By setting the relative area of the tilted image HUD region Z2 to be large, a region where a display with a sense of depth (in other words, a region with expressiveness in the depth direction) can be ensured, and it is easier to realize a vivid depth display. In other words, it is easier to perform a display that effectively utilizes the tilted image HUD region Z2, and there is no significant restriction just because the standing image HUD region Z1 exists. Therefore, for example, guiding information (such as an arrow) or the like can be presented to the user with an intuitive and vivid virtual image.

[0086] Next, refer to FIG. 3(C). FIG. 3(C) shows an example of display control by the display control unit (reference numeral 190 in FIG. 11). The specific configuration of the HUD device will be described later.

[0087] The display control unit (reference numeral 190 in FIG. 11) divides the image display area 45 of the display surface 164 of the display unit 160 into a first display area Z1' corresponding to the standing image HUD area Z1 and a second display area Z2' corresponding to the tilted image HUD area Z2, for example, based on a boundary position LN' which is the boundary between the near side and the far side when viewed from the user.

[0088] Then, the display control unit (reference numeral 190 in FIG. 11) displays a first image RG1 (here, a vehicle speed display SP') at a predetermined position in the first display area Z1', and displays a second image RG2 (here, a navigation arrow graphic AW') at a predetermined position in the second display area Z2'.

[0089] Depending on the configuration of the optical system, the "upper end of the angle of view" and the "lower end of the angle of view" in Fig. 3(C) may be reversed. In this case, the image displayed in Fig. 3(C) will be displayed upside down.

[0090] 3(C), points U1', U2', and U3' correspond to points U1, U2, and U3 in FIG. 3(B). The horizontal direction on the display surface 164 corresponds to the "left-right direction (X direction)" of the vehicle 1 in real space, and the vertical direction corresponds to the "height direction (Y direction)" that is the direction perpendicular to the road surface 40 in real space.

[0091] Moreover, the side line 51 and center line 53 of the road, which are depicted by dashed lines in FIG. 3(C), are detected by performing image processing on an image of the area ahead of the vehicle 1.

[0092] In FIG. 3(C), examples of the first image RG1 include vehicle information, information about the surroundings of the vehicle, and navigation information. These require accurate display and quick recognition, and vehicle speed information and the like is often displayed at all times. Therefore, the image HUD displays the information in front of the user in an easily viewable manner, standing at a certain angle or more (a predetermined angle or a threshold angle of the image (e.g., 45 degrees or more)) with respect to the road surface 40. Specific examples of this information include vehicle speed display, road speed limit information, turn-by-turn information (e.g., intersection name information, POI (point of interest on a map) information, etc.), and various displays using icons and characters.

[0093] The second image RG2 may include information including arrows indicating the progress of the vehicle or other vehicles, as well as graphics other than arrows. These images are primarily graphics, and it is important that they are intuitively comprehensible and allow the driver to perceive distance information without feeling uncomfortable. The second image RG2 may also include text, for example. Graphical information images with depth are displayed as virtual images with a sense of depth by the tilted image HUD. Specific examples of this information include arrow information as a route guide, white lines indicating the center line, colored graphic information indicating frozen road areas, and information from an advanced driver assistance system (ADAS) that assists the driver in operating the steering wheel or other devices.

[0094] By performing the display control shown in Fig. 3(C), a realistic virtual image display as shown in Fig. 3(A) is realized. In other words, by using a first virtual image G1, which is an image standing at a certain angle or more with respect to the road surface 40, and a second virtual image G2, which is, for example, an arrow-shaped image extending along the longitudinal direction of the vehicle 1 so as to cover the road surface 2, in combination, a realistic, easy-to-see, and highly expressive display can be realized. Therefore, the visibility of the virtual image display in the HUD device is improved.

[0095] Next, reference will be made to Figure 4. Figure 4(A) is a diagram showing an example of the configuration of a HUD device mounted on a vehicle and a virtual image display surface, and Figures 4(B) and (C) are diagrams showing examples of a method for realizing the virtual image display surface shown in Figure 4(A). Note that Figure 4(A) employs a different configuration from Figure 1(A). Therefore, different reference numerals are used even for the same parts.

[0096] Also, in Figure 4, the direction along the front of the vehicle 1 (also called the front-to-rear direction) is the Z direction, the direction along the width (horizontal width) of the vehicle 1 (left-to-right direction) is the X direction, and the height direction of the vehicle 1 (the direction away from the road surface 40 of a line segment perpendicular to the flat road surface 40) is the Y direction.

[0097] In the following description, the terms "up" and "down" are used to describe the shape of the virtual image display surface. For ease of explanation, the up-down direction is defined as the direction along a line (normal) perpendicular to the road surface 40 (which is also the height direction of the vehicle 1). When the road surface is horizontal, the vertically downward direction is the down direction, and the opposite direction is the up direction. This point can also be applied to the description of the drawings shown above.

[0098] As shown in FIG. 4(A), a HUD device 100 of this embodiment (specifically, a HUD device that can display only a "standing image," only an "inclined image," or both a "standing image and an inclined image" at the same time) is mounted inside a dashboard 41 of a vehicle (host vehicle) 1.

[0099] The HUD device 100 includes a display unit (sometimes called an image display unit, specifically, a screen) 160 having a display surface 164 for displaying an image; an optical system 120 including an optical element for projecting display light K for displaying the image onto a windshield (a reflective, translucent member 2); and a light projection unit (image projection unit) 150. The optical element includes a curved mirror (also called a concave mirror or a magnifying reflector) 170 having a reflecting surface 179. The reflecting surface 179 of the curved mirror 170 does not have a uniform radius of curvature, but can have a shape consisting of a collection of subregions with multiple radii of curvature. For example, a free-form surface design technique can be used (or the free-form surface itself can be used). Note that a free-form surface is a curved surface that cannot be expressed by a simple mathematical formula. It is expressed by setting several intersections and curvatures in space and interpolating the intersections using higher-order equations. The shape of the reflecting surface 179 significantly affects the shape of the virtual image display surface PS1 and its relationship to the road surface.

[0100] The shape of the virtual image display surface PS1 is affected not only by the shape of the reflecting surface 179 of the curved mirror (concave mirror) 130, but also by the curved surface shape of the windshield (reflective translucent member 2) and the shapes of other optical members (e.g., correcting mirrors) mounted in the optical system 120. It is also affected by the shape of the display surface 164 of the display unit 160 (which is generally flat, but may be non-flat in whole or in part) and the arrangement of the display surface 164 relative to the reflecting surface 179. However, the curved mirror (concave mirror) 170 is a magnifying reflecting mirror, and has a significant impact on the shape of the virtual image display surface. Furthermore, if the shape of the reflecting surface 179 of the curved mirror (concave mirror) 170 is different, the shape of the virtual image display surface actually changes.

[0101] Furthermore, the virtual image display surface PS1, which extends integrally from the near end U1 to the far end U3, can be formed by positioning the display surface 164 of the display unit 160 at an angle of less than 90 degrees to the optical axis of the optical system (the main optical axis corresponding to the chief ray).

[0102] The shape of the curved surface of the virtual image display surface PS1 may be adjusted by adjusting the optical characteristics of the entire area or a part of the area in the optical system, by adjusting the arrangement of the optical members and the display surface 164, by adjusting the shape of the display surface 164, or by a combination of these. In this way, the shape of the virtual image display surface can be adjusted in a variety of ways. This makes it possible to realize a virtual image display surface PS1 having a vertical image HUD area Z1 and an inclined image HUD area Z2.

[0103] A specific explanation will be given below. As shown in the left and lower left of Fig. 4(B), the manner and degree of overall tilt of the virtual image display surface PS is adjusted depending on the manner and degree of tilt of display surface 164 of display unit 160. In the example of Fig. 4(B), it is assumed that distortion of the virtual image display surface due to the curved surface of the windshield (reflective translucent member 2) is corrected by the curved shape of reflecting surface 179 of curved mirror (concave mirror, etc.) 170, resulting in the generation of a flat virtual image display surface PS.

[0104] Furthermore, as shown on the right and bottom left of Figure 4(B), by adjusting the positional relationship between the optical element (here, a curved mirror (concave mirror, etc.) 170) and the display surface 164, in other words, for example by rotating the display surface 164 to change the relative relationship with the optical element (curved mirror 170), the degree to which the virtual image display surface PS, which is an inclined surface, is separated from the road surface 40 can be adjusted.

[0105] Furthermore, as shown in Figure 4(C), by adjusting the shape of the reflective surface of the curved mirror (concave mirror, etc.) 170, which is an optical element (or by adjusting the shape of the display surface 164 of the display unit 160), the virtual image display distance near the end (near end) U1 of the virtual image display surface PS closer to the vehicle 1 is changed, so that the area near the near end U1 is bent toward the road surface and controlled to stand upright relative to the road surface (in other words, made vertical), thereby obtaining the virtual image display surface PS1.

[0106] As shown in the upper part of FIG. 4(C), the reflecting surface 179 of the curved mirror 170 can be divided into three parts (areas): Near (nearby display area), Center (middle (central) display area), and Far (far display area).

[0107] Here, Near is the part that generates display light E1 (shown by a dotted line in Figures 4(A) and (B)) corresponding to the near end U1 of the virtual image display surface PS1, Center is the part that generates display light E2 (shown by a dashed line) corresponding to the middle part (central part) U2 of the virtual image display surface PS1, and Far is the part that generates display light E3 (shown by a solid line) corresponding to the far end U3 of the virtual image display surface PS1.

[0108] In Fig. 4(C), the Center and Far portions are the same as the curved mirror (concave mirror or the like) 170 shown in Fig. 4(B) when generating a flat virtual image display surface PS. However, in Fig. 4(C), the curvature of the Near portion is set smaller than in Fig. 4(B). As a result, the magnification corresponding to the Near portion becomes larger.

[0109] The magnification of the HUD device (denoted as c) can be expressed as c = b / a, where a is the distance from the display surface 164 of the display unit 160 to the windshield 2 and b is the distance from the light reflected by the windshield (reflective translucent member 2) via viewpoint A to the image formation point. However, as the curvature of the near portion decreases, a decreases, the magnification increases, and the image is formed at a position farther away from the vehicle 1. In other words, in the case of Fig. 4(C), the virtual image display distance is greater than in the case of Fig. 4(B).

[0110] Therefore, the near end U1 of the virtual image display surface is pulled away from the vehicle 1, and the near end U1 is curved in a bowing manner toward the road surface 40, resulting in the formation of the standing image HUD area Z1. This results in a virtual image display surface PS1 having the standing image HUD area Z1 and the tilted image HUD area Z2.

[0111] Next, reference is made to FIG. 5. FIG. 5 is a diagram showing an example of a display method for a HUD device (a parallax HUD device or a 3D HUD device) that displays a three-dimensional virtual image. In FIG. 5, eye point P(C), which indicates the viewpoint position of the driver (user), is located at the center of eye box EB. If virtual imaging planes PS(L) and PS(R) corresponding to the left and right eyes are set in front of windshield 2, a virtual image V(C) is located at the center of the overlapping area. The convergence angle of virtual image V(C) is θd, and the virtual image V(C) is recognized as a three-dimensional image by the driver (user, viewer).

[0112] This three-dimensional virtual image V(C) can be displayed (formed) as follows. For example, light from an image IM displayed in a time-division manner is distributed using, for example, an MEMS scanner or the like to obtain display light L10 and R10 for the left and right eyes. The display light L10 and R10 are then reflected (reflected at least once) by, for example, a curved mirror (e.g., a concave mirror) 105 included in the optical system, and projected onto the windshield (projection target) 2 as display light K. The reflected light reaches the driver's eyes and forms an image in front of the windshield 2, thereby displaying (forming) a three-dimensional virtual image V(C) with a sense of depth. Note that the display method for an image with a sense of depth is not limited to the above. For example, a method may be used in which images for the left and right eyes are simultaneously displayed on a flat panel or the like, and the light from each image is separated using a lenticular lens or a parallax barrier to obtain display light L10 and R10 for each eye.

[0113] Furthermore, the present invention can be applied to a parallax-type HUD device that projects parallax images (different images) to each of the left and right eyes, but is not limited to this and can also be applied to a HUD device that projects the same image to each of the left and right eyes. These points will be described later.

[0114] Next, reference is made to Figure 6. Figure 6 is a diagram showing an example of viewpoint-tracking warping control in a HUD device capable of displaying at least one of a virtual image of a standing image (including a pseudo standing image) and a virtual image of an inclined image (depth image). The same reference numerals are used to designate parts common to Figures 1 and 3. In the following description, the inclined surface is to be interpreted in a broad sense, and as necessary, it is also possible to interpret it as including, for example, an inclined surface superimposed on the road surface (one with a zero inclination angle).

[0115] The display example in Fig. 6 is the same as that described using Fig. 3. The virtual image display surface PS1 has an elevation surface (pseudo elevation surface) PS1a including the vertical image HUD region Z1 and an inclined surface PS1b including the tilted image HUD region Z2. The vertical image HUD region Z1 displays a vehicle speed display SP of "120 km / h" as a first virtual image, which is a frontal image (vertical image). The inclined image HUD region Z2 displays a navigation arrow AW as a second virtual image, which is a depth image (tilted image).

[0116] The display method may be either a monocular method in which display light of the same image is incident on the left and right eyes, or a parallax method in which different parallax images are incident on the left and right eyes. Figure 6 shows an example of a parallax display.

[0117] An image of "120 km / h" as the first projection image G1' and images of arrows for the left and right eyes as the second projection images G2L' and G2R' are projected onto the projection area 4 of the windshield 2. Each image is given a distortion in the opposite direction to the distortion caused by the curved surface of the windshield 2.

[0118] Furthermore, the first projection image G1' is a standing image, and there is no need to express depth using parallax images, so the same image (common image) is projected.

[0119] When the driver's viewpoints A1 and A2 are both located at the center of the eye box EB, an arrow extending linearly along the road surface appears in front of the driver with a sense of perspective. Also, a vehicle speed display of "120 km / h" appears diagonally below and to the right of the image of the arrow in the foreground. In the example of FIG. 6, this vehicle speed display is displayed at a fixed position in the coordinate system of the real space. For example, this vehicle speed display can be displayed at all times.

[0120] In this state, let us assume that the driver's viewpoints A1 and A2 move (shift) along the width direction (left-right direction, X direction) of the vehicle. In FIG. 6, this shift in viewpoint is indicated by a dashed, two-way arrow SE. When this viewpoint shift occurs, the display seen by the driver is distorted by motion parallax. This deformation works advantageously for the arrow image AW, which is a depth image, by providing a natural sense of perspective, but works disadvantageously for the vehicle speed display SP, which is a frontal image, by reducing the recognizability of the information content. Therefore, it is better not to perform uniform warping on both images, but to perform warping using different image corrections individually. This will be explained in detail below.

[0121] Please refer to Fig. 7. Fig. 7(A) is a diagram showing a case where an inclined surface (for example, an inclined image HUD area of ​​a virtual image display surface) is placed in front of the vehicle, and the viewpoint (assumed to be a single eye) of the driver looking at it moves along the width direction (left-right direction) of the vehicle from a state where it is located in the central divided area of ​​the eye box, Fig. 7(B), (C), and (D) are diagrams showing how the inclined surface appears (the trapezoid shape that is the outline of the inclined surface) when the viewpoint is located in each of the left divided area, the central divided area, and the right divided area of ​​the eye box, Fig. 7(E) is a diagram showing an example of displaying a virtual image of an arrow on the inclined surface, Fig. 7(F), (G), and (H) are diagrams showing how the virtual image of the arrow appears (the shape of the virtual image of the arrow) when the viewpoint is located in each of the left divided area, the central divided area, and the right divided area of ​​the eye box, and Fig. 7(I) and Fig. 7(J) are diagrams showing how the virtual image of the arrow appears with and without motion parallax correction.

[0122] In Fig. 7(A), for convenience, the entire inclined surface PS1b on the virtual image display surface PS1 is considered as one display area visible to the driver, and the outline (shape) of the display area is rectangular in a plan view seen from a direction perpendicular to the road surface. Also, a grid formed by two orthogonal line segments is drawn inside the inclined surface PS1b. This indicates that each reference point (coordinate point) of the intersection can be grasped as a target for warping.

[0123] In the example of Figure 7(A), the eyebox EB is divided into left, center, and right partial regions ZL, ZC, and ZR. To simplify the explanation, it is assumed that an image is viewed with either the left or right eye. The symbols AL, AC, and AR attached to the eyes (viewpoints) correspond to the divided regions ZL, ZC, and ZR of the eyebox EB.

[0124] As mentioned above, the driver's eye (viewpoint) is initially located in the center of the eyebox EB (divided area ZC). The coordinate value in this state is X0. The coordinate values ​​when moving left or right are +X1 and -X1.

[0125] As shown in Figures 7(B), (C), and (D), the images seen by the driver corresponding to the viewpoints AL, AC, and AR change appropriately due to the influence of motion parallax. This is because the amount of apparent positional movement caused by lateral (left-right) movement of the viewpoint position is smaller the farther from vehicle 1 the image is, and larger the closer to vehicle 1 the image is. For example, when the image displayed in the background is sufficiently far away, the top side of the trapezoid is perceived as fixed, and the bottom side as it moves with the movement of the viewpoint position, which causes a deformation of the trapezoid (such as an isosceles trapezoid seen from the front).

[0126] When the virtual image display surface PS1b, which is an inclined surface disposed in front of the vehicle 1, is viewed from a viewpoint located at the center of the eye box EB (in other words, when viewed from directly in front of the driver), its contour is as follows: Because the width is perceived as narrowing toward the center due to perspective, it appears to be a trapezoid (including a parallelogram) (Figure 7(C)). The trapezoid in Figure 7(C) has parallel upper and lower bases, the length of the upper base is shorter than the length of the lower base, and the interior angles at both ends of the upper base are equal, as well as the interior angles at both ends of the lower base (a so-called isosceles trapezoid). However, there are also cases where the sense of depth can be almost ignored. In such cases, the lengths of the upper and lower bases are equal, and the left and right legs are parallel, so the isosceles trapezoid becomes a rectangle or a square (or, in a broader sense, a parallelogram with two pairs of opposite sides parallel). In other words, the trapezoid is a concept that includes parallelograms.

[0127] If the viewpoint moves (shifts) in the width direction (left and right direction) of vehicle 1, the relative position between the viewpoint and the virtual image will change unless the position of inclined surface PS1b changes, and the trapezoid will be distorted and deformed to the left or right due to the influence of motion parallax (see Figure 7(B) or Figure 7D)).

[0128] As can be seen from Figures 7(B) to (D), the basic shape of the outline (outline) of the virtual image display surface of the slope (depth HUD area Z2) is a trapezoid. When the viewpoint is in the center, an isosceles trapezoid is seen as shown in Figure 7(C). When the viewpoint shifts to the left, it becomes a left-inclined trapezoid (Figure 7(A)). When the viewpoint shifts to the right, it becomes a right-inclined trapezoid (Figure 7(D)).

[0129] This means that when warping is performed on the depth HUD area Z2 as a single image area during image correction (warping processing), if the contour (outline) is made trapezoidal rather than rectangular as in the past, an image (virtual image) with a more natural sense of depth will be seen after distortion caused by the optical system has been removed. Therefore, warping control is performed to deform the trapezoid so that the deformation of the trapezoid caused by movement of the viewpoint is reflected.

[0130] 7(E) to 7(G) show changes in the appearance (visual perception) of the arrow graphic AW (individual graphic) displayed in the depth HUD area Z2, rather than the depth HUD area Z2 itself.

[0131] Figures 7(E)-(H) correspond to Figures 7(A)-(D). When seen from the front, the arrow AW appears as in Figure 7(G). However, when the viewpoint shifts leftward, the arrow AW appears to tilt leftward as in Figure 7(F). When the viewpoint shifts rightward, the arrow AW appears to tilt rightward as in Figure 7(H).

[0132] Thus, when warping is performed on an image-by-image basis (in other words, for each image) during image correction (warping process), warping the image area of ​​each image based on a trapezoid, as in the cases of FIGS. 7(B) to 7(D), results in an image (virtual image) with a more natural sense of depth after distortion caused by the optical system is removed. Specifically, for example, arrows used in route guides and the like are designed to be visually perceived as approximately coinciding with the road surface. The road surface onto which the display is superimposed also undergoes a similar apparent shape change with the above-mentioned viewpoint shift. In other words, when the road surface and the display appear to change shape in a similar manner with viewpoint shift, humans perceive the display as approximately coinciding with the road surface. Therefore, this shape change can be said to aid human recognition of images intended to be matched (superimposed) with the road surface or displays intentionally positioned in the depth direction (images that are perceived as being far away).

[0133] Therefore, in this embodiment, for an image area of ​​an image with a sense of depth (which can mean the entire display area or the display area of ​​each image included in the display area, and can be interpreted as appropriate), image correction is performed by assuming that the contour when correctly viewed as a virtual image is a trapezoid. As a result, after distortion caused by the optical system is removed, an image (virtual image) with a more natural sense of depth is viewed by the driver (user, viewer).

[0134] Furthermore, since a depth image (tilt image) is an image generated to create a three-dimensional effect due to parallax, it is naturally expected from the beginning that the image will appear distorted due to the influence of motion parallax. As mentioned above, the motion parallax creates a natural (or nearly natural) three-dimensional effect (perspective), and therefore the trapezoidal deformation (distortion) shown in Figures 7(B) and (D) helps to create a natural visual perception of the depth image.

[0135] Therefore, when viewpoint-tracking warping is performed during image generation, if the viewpoint shifts to the left, the isosceles trapezoid is corrected so that it appears tilted to the left, making the perspective more natural. Then, the corrected image area is corrected by applying a distortion with the opposite characteristics to the distortion caused by the optical components (conventional warping). This results in a display like that shown in Figure 7(B). In other words, it is possible to display a depth image with a more natural perspective (three-dimensional effect).

[0136] However, in Figures 7(F) and (H), the degree of tilt of the arrow (in other words, the degree of distortion of the outline (trapezoid) of the image area displaying the arrow) is appropriate, and the images are not unnatural. To obtain this image, it may be necessary to appropriately set the degree of distortion, taking motion parallax into consideration. If this is not possible, for example, as shown in Figure 7(I), when the viewpoint shifts to the left, the arrow shape may tilt significantly to the left, resulting in an unnatural image (virtual image). In this case, by slightly suppressing motion parallax, it is possible to obtain an appropriate image (virtual image) like Figure 7(J) (this Figure 7(J) is the same as Figure 7(F)).

[0137] For example, when the displayed image is a road surface superimposed image (for example, an image of an arrow shape as in Figures 7(F) and (H)) that is displayed so as to be superimposed on the road surface, and the virtual image of the road surface superimposed image is displayed on an inclined surface that is inclined relative to the road surface, and the virtual image appears to be floating above the road surface (see the display examples in Figures 3 and 6), it is preferable to perform motion parallax correction that reduces the degree of deformation of the arrow (in other words, the trapezoid that is the outline of the display area of ​​the arrow image) compared to when the virtual image does not appear to be floating above the road surface (when the degree of superimposition on the road surface is relatively high).

[0138] This is because the position of the virtual image of the actual arrow (the virtual image of the road surface superimposed image) is closer to the driver (viewer) than the position of the road surface on which the arrow appears superimposed. In this case, motion parallax occurs for the virtual image of the arrow that appears in the foreground, while motion parallax also occurs for the road surface that appears behind it. Because the road surface that appears in the background is displayed in the distance, the motion parallax is smaller, while the virtual image of the arrow that appears in the foreground is displayed in the foreground, the motion parallax is perceived to be larger. Because human vision detects this relative difference, the motion parallax for the virtual image of the arrow is perceived to be larger than when the virtual image of the arrow is in close contact with the road surface, for example. Therefore, as in the example of Figure 7(I), the leftward tilt of the virtual image of the arrow is emphasized. Therefore, in this case, by performing motion parallax correction that reduces the leftward tilt while retaining it—in other words, that suppresses the deformation of the trapezoid that is the outline of the display area of ​​the arrow image—a display with appropriate motion parallax is obtained, as shown in Figure 7(J). However, unlike the front-facing image, the motion parallax is not cancelled out and some degree of motion parallax remains. Therefore, the amount of motion parallax suppression for the depth image is smaller than the amount of motion parallax suppression for the front-facing image.

[0139] Next, reference will be made to Fig. 8. Fig. 8(A) is a diagram showing a case where a vertical surface (including a pseudo vertical surface, for example, a vertical image HUD area of ​​a virtual image display surface) is placed in front of the driver, and the viewpoint (assumed to be a single eye) of the driver looking at it moves along the width direction (left-right direction) of the vehicle from a state where it is located in the central divided area of ​​the eye box, Fig. 8(B), (C), and (D) are diagrams showing how the vertical surface appears (the rectangular shape that is the contour of the vertical surface) when the viewpoint is located in each of the left divided area, the central divided area, and the right divided area of ​​the eye box, Fig. 8(E), (F), and (G) are diagrams showing the content of image correction applied to the display image to suppress deformation of the rectangular contour of the vertical surface, and Fig. 8(H), (I), and (J) are diagrams showing how the virtual image of the arrow appears (the shape of the virtual image of the arrow) when the viewpoint is located in each of the left divided area, the central divided area, and the right divided area of ​​the eye box.

[0140] 8(A) to 8(D) correspond to the previously shown FIGS. 7(A) to 7(D). In FIG. 8(A), a vehicle speed display SP of "120 km / h" is displayed on the vertical virtual image display surface PS1a. When viewed from the front, the vehicle speed display appears as shown in FIG. 8(C). However, when the viewpoint shifts to the left, the vehicle speed display SP appears to tilt to the left, as shown in FIG. 8(B). When the viewpoint shifts to the right, the vehicle speed display SP appears to tilt to the right, as shown in FIG. 8(D).

[0141] As shown in Figure 8(C), for a frontal image (image) displayed on an elevation (including a pseudo-elevation), the outline (outline) of the image area is rectangular (rectangle or square), as in the past. Vehicle speed displays SP and the like displayed on an elevation are often always displayed in a fixed position, and it is important that numbers, symbols, letters, etc. can be read accurately, and a sense of perspective is not particularly necessary. Therefore, there is no advantage to using a trapezoid, as in the case of an oblique image, and a rectangular outer shape is used here, as in the past.

[0142] However, if the viewpoint shifts left or right, the display will be distorted as shown in Figures 8(B) and (D). The reason for this distortion is that even though the distortion of the optical system is removed by warping and the display light of the rectangular image reaches the viewpoint (eye), the position of the vehicle speed display SP does not change, so the direction from which the display light arrives (the direction of incidence) is different before and after the viewpoint shift occurs. If a rectangle like the one in Figure 8(C) was visible without distortion before the viewpoint shift, after the viewpoint shift occurs, the brain receives light arriving from a different angle and interprets the image, resulting in the image appearing distorted due to motion parallax, as shown in Figures 8(B) and (D).

[0143] Image deformation due to motion parallax, as shown in Figures 8(B) and (D), is not desirable in terms of the recognizability of the display. Therefore, in this embodiment, when warping the vehicle speed display SP (broadly speaking, the frontal image (standing image)), the image (the outline of the image area: rectangular) is deformed in the direction opposite to the actual deformation due to the influence of motion parallax so that such deformation (deformation of a rectangle or other rectangular shape) is suppressed (cancelled), and then warping (conventional warping) is performed to impart distortion with characteristics opposite to the distortion caused by the optical system. Images in a state where distortion with characteristics opposite to the distortion that actually occurs is applied are shown in Figures 8(E) to (G).

[0144] Therefore, when the viewpoint shifts, the light from the image that is deformed in the opposite way to the deformation caused by motion parallax enters the human eye at that viewpoint.

[0145] At this time, the deformation due to motion parallax is cancelled out by the inverse deformation that was previously applied, and when the human brain interprets the image, it appears as a rectangular (rectangle or square) image. The image at this time is shown in Figures 8(H) to (J). As can be seen from Figures 8(H) and (J), even when viewpoint shift occurs, the outer shape (contour) of the image area of ​​the vehicle speed display SP is rectangular. In other words, regardless of the position of the viewpoint, the outer shape of the image area is maintained as a rectangle, and an easy-to-view frontal image is always displayed. This prevents a decrease in recognizability, and since a frontal image is always available, no discomfort occurs, thereby improving visibility.

[0146] As described above, in this embodiment, the contours of the image area of ​​the depth image (tilted image) and the contours of the image area of ​​the frontal image (standing image) are set separately and warped, so that different warping methods are essentially performed on each image. As a result, a display with improved visibility that gives a natural sense of perspective is realized for the depth image (tilted image), and a display with improved visibility and improved recognizability (e.g., a display suitable for accurately grasping information) is realized for the frontal image (standing image).

[0147] Next, reference will be made to Fig. 9. Fig. 9(A) to (D) are diagrams for explaining two examples of image correction (examples of deforming the bottom or top side) that are applied to a display image in order to suppress deformation of the rectangle that is the contour of the elevation.

[0148] 8(E) and 8(G), it was explained that a process for deforming a rectangle is performed during warping so as to cancel out the deformation of the rectangle that actually occurs due to motion parallax. There are two possible methods for deforming this rectangle. These methods will be explained in detail below.

[0149] In FIG. 9(A), the viewpoint is shifted to the right while the vehicle speed display SP is displayed. In this case, the vehicle speed display SP is distorted so as to tilt to the left, as explained above. To prevent this, it is necessary to impart a distortion with an inverse characteristic to the image of the vehicle speed display in advance so as to cancel out the distortion. In other words, it is necessary to correct the image shown in the left diagram of FIG. 9(B) to an image tilted to the right, as shown in the right diagram.

[0150] When performing such image correction, the bottom side BC of the rectangle may be fixed and the top side AD may be moved to distort the image, as shown in Figure 9(C). In Figure 9(C), the top side AD has been displaced to the top side A'D'.

[0151] Alternatively, as shown in Fig. 8(D), the upper side AD may be fixed and the lower side BC may be moved to distort the image. In Fig. 8(D), the lower side BD is displaced to the lower side B'D'.

[0152] Either method can impart a deformation (distortion) to an image region that cancels out the actual deformation caused by motion parallax, and in this respect, both methods have the same effect.

[0153] However, from the driver's (viewer's) perspective, the top edge of the rectangle displayed in front is far away and the bottom edge is close, and when motion parallax is taken into account, the further away the rectangle is, the smaller the perceived change in position. Therefore, when the rectangle is transformed by moving the top edge, it is thought that the amount of movement (actual amount of transformation) can be reduced compared to when the rectangle is transformed by moving the bottom edge, and in this respect there is a difference.

[0154] Next, reference will be made to Fig. 10. Fig. 10(A) is a diagram showing the contents of warping control for an inclined image (depth image) displayed on an inclined surface, Fig. 10(B) is a diagram showing the contents of warping control for an upright image (frontal image) displayed on an upright surface, Figs. 10(C) to (E) are diagrams showing an example of generating an image to be displayed on a display surface by image rendering, and Figs. 10(F) to (H) are diagrams showing examples of images (visual image, visual virtual image) visually recognized by the driver according to the viewpoint position.

[0155] As previously explained in Figure 3(C), the image display area 45 of the display surface 164 of the display unit 160 is divided into a first display area Z1' corresponding to the standing image HUD area Z1 and a second display area Z2' corresponding to the tilted image HUD area Z2, based on the boundary position LN', which is the boundary between the near side and the far side as seen from the user.

[0156] In the second display area Z2', a navigation arrow graphic AW' is displayed as the second image RG2. Note that in Fig. 10, the arrow graphic AW' is shown as a vertically long ellipse for convenience.

[0157] In this embodiment, during warping, as shown in a1 to a3 in FIG. 10A, the outer shape (contour) of the image area of ​​the arrow shape AW' is first corrected according to the position of the viewpoint. That is, when the viewpoint is located at the center of the eyebox EB, for example, an isosceles trapezoid is formed as shown in a2. When the viewpoint shifts to the left, the isosceles trapezoid is distorted by tilting to the right as shown in a1. When the viewpoint shifts to the right, the isosceles trapezoid is distorted by tilting to the left as shown in a3. In this case, as described above, it is preferable to variably control the motion parallax and perform warping to obtain an appropriate trapezoidal deformation. When the eye position shifts, for example, to the left, the motion parallax causes the upper edge of the image to deform so as to tilt significantly to the left. In response to this, the warping control corrects the upper edge by tilting it slightly to the right. This is because, as described above, since the virtual image plane is located in front of the road surface, the motion parallax deformation of the virtual image plane is greater than the motion parallax of the road surface, and it is therefore necessary to weaken the influence (deformation) of this motion parallax.

[0158] Next, as shown in a4 to a6, a distortion having the opposite characteristics to the distortion caused by the windshield or the optical system of the HUD device (collectively referred to as the optical member) is applied to each image (image area of ​​each image) according to the viewpoint position. In this way, viewpoint-tracking warping is performed on the front-facing image (tilted image).

[0159] Also, an image SP' displaying vehicle speed is displayed as the first image RG1 in the first display region Z1'. Note that in Fig. 10, the image SP' displaying vehicle speed is shown as a horizontally long ellipse for the sake of convenience.

[0160] As in the case of Fig. 10(A), first, the outer shape (outline) of the image area is corrected according to the viewpoint position, as shown in b1 to b3 of Fig. 10(B). Next, as shown in b4 to b6, distortion with characteristics opposite to the distortion caused by the optical members is applied to each image (image area of ​​each image) in accordance with the viewpoint position. In this way, viewpoint-tracking warping is performed on the depth image (tilt image).

[0161] Furthermore, it is possible that a single image may contain both a depth image and a frontal image. In this case, image correction specific to each image (region of each image) is performed individually, as described with reference to Fig. 10(A) or 10(B). In other words, different image correction methods are adopted depending on the type of image, and image correction is performed individually.

[0162] Then, as shown in FIGS. 10(C) to 10(D), the images (images after warping processing) corresponding to the viewpoint positions are synthesized by, for example, image rendering to generate a single image.

[0163] In each of Figures 10(C) to 10(E), the images obtained by synthesis are labeled Q1 to Q3. This image synthesis process can also be referred to as warping processing. The depth image (or image region of the depth image) and the front-facing image (or image region of the front-facing image) are treated separately and image correction is performed separately before they are synthesized, so that desired image correction can be quickly performed on each image. Furthermore, the image processing itself can be simplified. Therefore, the load on the image processing unit (which may include an image generation unit, an image rendering unit, etc.) associated with the image correction process is also reduced.

[0164] 10(F), (G), and (H) are diagrams showing how a virtual image appears when the viewpoint is located in the left, center, and right divided regions of the eyebox. When the viewpoint is located at the center of the eyebox EB, the tilted image HUD region Z2 is defined as a region surrounded by a trapezoid (e.g., an isosceles trapezoid) whose upper and lower edges are parallel to each other. If the first contour Za is a region within the tilted image HUD region Z2 seen from the center of the eyebox EB, the isosceles trapezoid tends to tilt left due to the influence of motion parallax. (In other words, the upper edge of the first contour Za tends to move leftward relative to the lower edge.) Here, by executing warping control to tilt the isosceles trapezoid right as shown in a1, the driver (whose viewpoint is shifted leftward from the center) sees a virtual image displayed on the first contour Za, in which the deformation of the isosceles trapezoid that tends to tilt left due to motion parallax is reduced, as shown in FIG. 10(F). On the other hand, if the viewpoint shifts from the center to the right, the isosceles trapezoid tends to tilt to the right due to the influence of motion parallax (in other words, the upper side of the first contour Za tends to move to the right relative to the lower side). Here, by executing warping control to tilt the isosceles trapezoid to the left as shown in a3, the driver (a driver whose viewpoint shifts from the center to the right) sees a virtual image displayed on the first contour Za, in which the deformation of the isosceles trapezoid that tends to tilt to the right due to motion parallax is reduced, as shown in FIG. 10(J). As a result, the motion parallax of the depth image approaches the motion parallax with the background, such as the road surface, thereby realizing a display with improved visibility and a natural sense of perspective. Note that the first contour Za may be a partial region of the tilted image HUD region Z2 viewed from the center of the eyebox EB, surrounded by a trapezoid (e.g., an isosceles trapezoid) whose upper and lower sides are parallel to each other, or may be the entire region if the tilted image HUD region Z2 itself is a trapezoid.

[0165] Furthermore, when the viewpoint is shifted from the center of the eyebox EB to the left, the second contour Zb is defined as an area surrounded by a rectangle, including a rectangle or a square, within the image HUD region Z1. If the viewpoint shifts from the center to the left, the rectangle will tend to tilt to the left due to the influence of motion parallax (in other words, the top edge of the second contour Zb will tend to be on the left side relative to the bottom edge). Here, as shown in b1, warping control is performed to tilt the rectangle to the right so as to cancel out the deformation of the second contour Zb caused by motion parallax. As shown in FIG. 10(F), a virtual image displayed on the second contour Zb, in which the deformation of the rectangle tending to tilt to the left due to motion parallax has been canceled, is viewed by the driver (a driver whose viewpoint has shifted from the center to the left). On the other hand, if the viewpoint shifts from the center to the right, the rectangle will tend to tilt to the right due to the influence of motion parallax (in other words, the top edge of the second contour Zb will tend to be on the right side relative to the bottom edge). Here, as shown in b3, warping control is performed to tilt the rectangle to the left so as to cancel out the deformation of the second contour Zb due to motion parallax, and as a result, a virtual image displayed on the second contour Zb in which the deformation of the rectangle that tries to tilt to the right due to motion parallax has been canceled is visually recognized by the driver (the driver whose viewpoint is shifted left from the center) as shown in Fig. 10(H). Note that the second contour Zb may be a part of the image HUD area Z1 surrounded by a rectangle when viewed from the center of the eyebox EB, or may be the entire area if the image HUD area Z1 itself is rectangular.

[0166] Furthermore, when performing the image processing described above, for example, the display layers may be separated depending on whether or not depth is used, and after performing separate shape correction processes for each layer, the images may be superimposed to form the final display image. That is, the display layer for the tilted image HUD region Z2 that expresses depth and the display layer for the standing image HUD region Z1 that does not express depth may be separated, and after performing separate shape correction processes for each layer, the images may be superimposed to form the final display image. Furthermore, the tilted image HUD region Z2 that expresses depth may be composed of multiple display layers, each of which is different for each region. Specifically, the tilted image HUD region Z2 may be composed of multiple display layers that are divided into multiple regions in the depth direction. Furthermore, shape correction may be performed individually according to the display attributes of each content.

[0167] Next, reference will be made to Fig. 11. Fig. 11 is a diagram showing an example of the configuration of an HUD device. The upper diagram of Fig. 11 is the same as Fig. 4(A). Here, the configuration of the display control unit 190 will be described.

[0168] The display control unit 190 has a viewpoint position detection unit 192 and a warping processing unit 194. The warping processing unit 194 has a warping control unit 192, a ROM 198 (having first and second image tables 199 and 200), a VRAM (for storing, for example, image data 196 and post-warping data 197) 201, and an image generation unit (image rendering unit) 202. Note that the warping control unit 195 may be configured to be provided outside the warping processing unit 194. The viewpoint position detection unit 192 may also be provided outside the HUD device 100.

[0169] The first image conversion table 199 stores warping parameters for frontal images (standing images), and the second image conversion table 200 stores warping parameters for depth images (tilted images).

[0170] The warping control unit 195 controls the image generation unit (image rendering unit) 202, ROM 198, VRAM 201, etc., so that the viewpoint position tracking warping described above is performed using warping parameters corresponding to the viewpoint position information supplied from the viewpoint position detection unit 192.

[0171] The VRAM stores (accumulates) original image data 196 of an image to be displayed. For example, the original image data 196 is read out, and warping parameters corresponding to the viewpoint position are applied to the read-out original image data, thereby performing the previously described deformation correction of the image area, image correction that preliminarily adds distortion with characteristics opposite to that of the distortion of the optical system, etc. Warping-processed data 197 is temporarily stored in the VRAM 201 and then supplied to an image generation unit (image rendering) 202, where image synthesis processing such as that shown in FIGS. 10(C) to 10(E) is performed. As a result, one image (display image) is generated. The generated image is supplied to a display unit (for example, a flat panel display such as a liquid crystal panel) 160 and displayed on a display surface (reference numeral 164 in FIG. 3(C)).

[0172] Next, let us refer to Figure 12. Figure 12(A) is a diagram showing an example of the main configuration of a HUD device when a figurine (pseudo figurine) is displayed fixed at a predetermined position in a viewpoint coordinate system regardless of the viewpoint position, and Figure 12(B) is a diagram showing that the display (virtual image) appears to move in accordance with the movement of the viewpoint.

[0173] In the example of FIG. 12, deformation (distortion) of the image area due to motion parallax is prevented for a frontal image (standing figure) by a method different from the image processing method of FIG. 8 described above.

[0174] In the example of Figure 8 described above, the problem of distortion of the front-facing image due to a shift in viewpoint position is addressed by previously providing an opposite distortion (inverse rectangular deformation) to the distortion (rectangular deformation) caused by motion parallax.In the example of Figure 12, however, the problem is addressed by shifting the display position of the front-facing image (virtual image) in accordance with the movement of the viewpoint.

[0175] As described above, the problem of distortion of the frontal image due to a shift in viewpoint position can arise because the display position of the frontal image (virtual image) is fixed in a coordinate system set in real space.

[0176] In the example of FIG. 12, the above problem is addressed by eliminating the assumption that the display position in the coordinate system set in real space is fixed.

[0177] In other words, in a viewpoint coordinate system (a coordinate system with a person's viewpoint at the center and axes set along the front-to-back, left-to-right, and up-and-down directions of the vehicle, and when the person's viewpoint (including face, etc.) moves, the coordinate system also moves in accordance with that movement), the position of the image (rectangular image area) on the display surface of the display unit (or the position on the image generation surface or image generation space of the image generation unit) is appropriately moved in response to the movement of the viewpoint so that the virtual image of the front-facing image is always in the same position.

[0178] This means that in a coordinate system set in real space, the position of the virtual image moves appropriately depending on the viewpoint deviation. In this case, for example, the relative positional relationship between the virtual image of the frontal image and the person's viewpoint is always constant, and there is no need to consider motion parallax, so there is no change in appearance. Therefore, this method can also solve the above-mentioned problem that the image is distorted by motion parallax, reducing recognition. Furthermore, the fact that the display position of the image (virtual image) does not change also has the effect of giving a sense of security to the viewer, such as a driver.

[0179] In Fig. 12(A), a viewpoint coordinate system movement amount (rotation amount) calculation unit 193 is added to the configuration shown in the lower part of Fig. 11. Since other parts are the same as in Fig. 11, Fig. 12(A) is simplified and shows only the main parts.

[0180] 12(A), when a change in the viewpoint position (displacement of the viewpoint) in a coordinate system (XYZ coordinate system) set in real space is detected by viewpoint position detection unit 192, the HUD device detects the change (displacement) as the amount of movement (including the amount of rotation) in the viewpoint coordinate system (X'Y'Z' coordinate system). Warping control unit 195 moves the viewpoint coordinate system based on the detected amount of movement, and controls so that an image is displayed at a predetermined position in the viewpoint coordinate system after the movement (a coordinate point indicating the viewpoint position before the movement).

[0181] As a result, as shown in FIG. 12(B), when viewpoint A moves from coordinate point X0 to +X1, the vehicle speed display SP also moves left and right by the same distance in the same direction as the movement of the coordinate point. For example, if the vehicle speed display SP was visible directly in front of the driver before viewpoint A moved, this state is maintained even after the viewpoint moves. Therefore, no motion parallax occurs. The driver can always see a virtual image of a frontal view (standing image) without distortion. Therefore, visibility is improved.

[0182] Next, reference will be made to Fig. 13. Fig. 13 is a diagram showing an example of a procedure in viewpoint tracking warping control.

[0183] In step S1, the viewpoint position is detected and warping processing is started. In step S2, the type of image to be displayed is determined, such as whether it is a depth image (tilted image, tilted image) or a frontal image (standing image, pseudo-standing image).

[0184] If it is determined in step S2 that the image is a depth image, the process proceeds to step S3.

[0185] In step S3, for example, warping is performed by regarding the outline (contour) of the image area as a trapezoid (including a parallelogram) when the outline (contour) is correctly viewed as a virtual image. Furthermore, when the viewpoint position is shifted in the width direction of the vehicle, image processing is performed to previously apply a deformation to the trapezoid with characteristics opposite to the change in appearance due to motion parallax, in order to reduce the degree of deformation while maintaining the direction of the change in appearance of the virtual image due to motion parallax. When the virtual image of the road surface superimposed image is displayed on a virtual image display surface of an inclined surface, for example, and appears to be floating above the road surface, correction is performed to suppress deformation of the virtual image due to motion parallax (however, appropriate motion parallax is left, and the motion parallax is not canceled out).

[0186] If it is determined in step S2 that the image is a frontal image, the process proceeds to step S4.

[0187] In step S4, for example, warping is performed by regarding the outline (contour) of the image area as a rectangle or a square (collectively referred to as a "rectangle") when the outline (contour) is correctly viewed as a virtual image. When the viewpoint position shifts in the width direction of the vehicle, "rectangle maintenance correction (motion parallax cancellation, suppression correction) that maintains the rectangular shape regardless of the viewpoint position" is performed, which applies distortion to the image (image area) in advance that has characteristics opposite to the change in appearance of the virtual image due to motion parallax. Furthermore, the rectangle maintenance correction is selectively performed in one of two ways: fixing the bottom side (bottom base) of the rectangle and moving the top side (top base) to distort it, or fixing the top side (top base) and moving the bottom side (bottom base) to distort it. Furthermore, if necessary, control is performed to fix the display position of the virtual image to a predetermined position in the viewpoint coordinate system (control to move the virtual image display position in response to movement of the viewpoint).

[0188] In step S5, it is determined whether image correction is complete. If it is Y, the viewpoint tracking warping process is completed. If it is N, the process returns to step S2.

[0189] As described above, according to the embodiments of the present invention, it is possible to realize warping control that ensures more natural visibility for, for example, depth images (tilted images) that are based on stereoscopic vision, or frontal images (standing images, pseudo-standing images) that, in principle, are not based on stereoscopic vision.

[0190] Recent HUD devices tend to be developed with the assumption that they will display virtual images over a fairly wide area in front of the vehicle, for example. In this case, the virtual image display area on the windshield is expanded, making it possible to display a variety of images.

[0191] It is also possible to simultaneously display different types of images, but in this case, it is expected that different warping methods will be required for each type of image. According to the present invention, the warping method can be changed depending on the type of image, so the visibility of various displays is not reduced. Therefore, it is possible to achieve high functionality and high performance of the HUD device.

[0192] The present invention can be used in both monocular HUD devices, in which display light for the same image is incident on each of the left and right eyes, and parallax HUD devices, in which images with parallax are incident on each of the left and right eyes.

[0193] In this specification, the term "vehicle" can be broadly interpreted as a vehicle. Furthermore, navigation-related terms (e.g., signs) are also broadly interpreted, taking into consideration the broader definition of navigation information useful for vehicle operation. Furthermore, the term "HUD" also includes devices used as simulators (e.g., aircraft simulators, simulators for game devices, etc.).

[0194] 14(A) and 14(B) are diagrams showing modified examples of the virtual image display surface. The cross-sectional shape of the virtual image display surface PS1 as viewed in the width direction (left-right direction, X direction) of the vehicle is not limited to the above-mentioned shape with the driver's side convex. The virtual image display surface PS1 may have a concave shape on the driver's side, as shown in FIG. 14(A). Furthermore, the virtual image display surface PS1 does not have to be curved, as shown in FIG. 14(B).

[0195] The present invention is not limited to the above-described exemplary embodiments, and those skilled in the art will be able to easily modify the above-described exemplary embodiments to the extent that they fall within the scope of the claims. [Explanation of symbols]

[0196] 1···vehicle (host vehicle), 2···projected member (reflective / translucent member, windshield, etc.), 4···projection area, 5···virtual image display area, 7···steering wheel, 51···display light, 100···HUD device, 120···optical system including optical members, 150···light projecting unit (image projection unit), 160···display unit (e.g., liquid crystal display device or screen, etc.), 164···display surface, 170···curved mirror (concave mirror, etc.), 179···reflecting surface, 188···imaging unit (pupil detection unit, face imaging unit, etc.), 190···display control unit, 192···viewpoint position detection unit, 193···movement amount (including rotation amount) of viewpoint coordinate system ) calculation unit, 194...warping processing unit, 195...warping control unit, 196...original image data, 197...warped data, 199...first image conversion table, 200...second image conversion table, 201...VRAM (image processing storage device), 202...image generation unit (image rendering unit), EB...eyebox, WP...warping parameter, PS1a...virtual image display surface of vertical surface (including pseudo-vertical surface), PS1b...virtual image display surface of inclined surface (including one approximately superimposed on the road surface), V...virtual image, Z1...vertical image HUD area, Z2...inclined image HUD area

Claims

1. The vehicle is equipped with a projection element that projects an image onto the vehicle, and the image is projected onto the projection element. A head-up display (HUD) device that allows a virtual image of the image to be visually recognized, an image generation unit that generates the image; a display unit that displays the image; an optical system including an optical member that reflects display light of the image and projects the image onto the projection target; The warping parameters are updated according to the driver's viewpoint position in the eyebox. A viewpoint-tracking warping system that corrects the image displayed on the display unit using warping parameters. a control unit that performs a switching control; and The control unit When warping is performed on a depth image, the image area of ​​the depth image is When the contour is correctly recognized as a virtual image, the contour is determined by checking whether at least one pair of opposite sides is parallel to each other. Warping control is performed as a trapezoid, When warping is performed on a front-facing image, the contour of the image area of ​​the front-facing image is The outline of the virtual image is correctly recognized as a rectangle or a square, and the outline is warped. Implementing control of A head-up display device.

2. The control unit When the warping process is performed on the depth image, the viewpoint is When the object moves along the width direction, the deformation of the trapezoid caused by the movement is reflected. warping control is performed to deform the trapezoid so as to When the warping process is performed on the front-view image, the viewpoint is When the object moves along the direction, the object is moved in a manner that cancels out the deformation of the rectangle caused by the movement. warping control is performed to deform the rectangle as follows:

2. The head-up display device according to claim 1, wherein:

3. The control unit performs warping control on the front-view image to deform the rectangle. When the rectangle is transformed to erase it, Fixing the bottom side of the rectangle and moving the top side to distort it. Or, Fix the top edge and move the bottom edge to distort it.

3. The head-up display device according to claim 2.

4. The control unit When the warping process is performed on the front-view image, the viewpoint is If moving along the direction, The virtual image of the frontal image is fixed at a predetermined position in a viewpoint coordinate system based on the viewpoint.

2. The head-up display device according to claim 1, wherein the position of the front-facing image is changed in response to the change of the front-facing image. Display device.

5. The control unit When the depth image and the front image are displayed together, After image processing is performed on the front-facing image and the front-facing image separately, the processed images are synthesized into one image. Let us take the image, 5. The head-up display device according to claim 1, 。

6. The control unit an image of the depth image as viewed from a predetermined first position in the left-right direction of the eyebox; Among the regions, a region surrounded by a trapezoid having at least one pair of opposite sides parallel to each other is defined as a first contour. When you do, When viewed from the left side of the first position, the deformation of the trapezoid caused by the movement In contrast, the upper side of the first contour is relatively to the right of the lower side, When viewed from the right side of the first position, the deformation of the trapezoid caused by the movement In contrast, the warping control is performed so that the upper side of the first contour is relatively to the left of the lower side. We will implement the following measures: an image area of ​​the frontal image as viewed from the first position in the left-right direction of the eyebox; When the area surrounded by a rectangle including a rectangle or a square is defined as the second contour, When viewed from the left and right sides of the first position, the upper side of the second contour is located at the and performing the warping control so that the position does not change relative to the The head-up display device according to any one of claims 1 to 5, Place.

7. When the control unit is performing warping processing on the depth image, When a point moves along the width direction of the vehicle, the deformation of the trapezoid caused by the movement is When performing warping control to deform the trapezoid so that the shape is reflected, the display image is a road surface superimposed image displayed so as to be superimposed on the road surface, The virtual image of the road surface superimposed image is displayed on an inclined surface that is inclined with respect to the road surface, and When the image appears to be lifted from the road surface, it is more noticeable than when it is not lifted from the road surface. ,performing warping control with motion parallax correction to reduce the degree of deformation of the trapezoid; 3. The head-up display device according to claim 2, wherein:

Images