Display control device, head-up display device, and display control method

Abstract

To provide a display control device, a head-up display device and a display control method which give a motion parallax that does not give a sense of discomfort to an observer.SOLUTION: A display control device comprises: first image correction processing of acquiring the eye position and / or head position of a user in the vertical direction of the vehicle and the eye position and / or head position in the left-right direction of the vehicle, and correcting the position of an image to be displayed on a display unit with a first correction amount based on at least a change in the eye position or the head position in the vertical direction and in the eye position or the head position in the left-right direction; and second image correction processing of correcting the position of the image to be displayed on the display unit with a second correction amount obtained by multiplying the first correction amount by a first prescribed coefficient equal to or less than 1. The display control device executes the first image correction processing when the distance from the vehicle to a real object in the foreground is equal to or less than a first distance threshold, and executes the second image correction processing when the distance from the vehicle to the real object in the foreground is greater than the first distance threshold.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] The present disclosure relates to a display control device, a head-up display device, a display control method, etc., that are used in a moving body such as a vehicle and that superimpose an image on the foreground of the moving body (the actual view in the forward direction of the moving body as seen by an occupant of the vehicle) and allow the image to be viewed.

[0002] Patent Document 1 describes a head-up display device (one example of a virtual image display device) in which display light projected onto a projection target such as the front windshield of the vehicle is reflected toward a vehicle occupant (observer) inside the vehicle, allowing the observer to visually recognize a virtual image superimposed on the foreground of the vehicle. In particular, the head-up display device described in Patent Document 1 allows the observer to virtually perceive a display object (virtual image) at a predetermined position (here, the position is referred to as a target position) in the depth or in the up, down, left, and right directions in the real space of the foreground, and controls an image displayed inside the head-up display device as if the display object were present at the target position in the foreground, even if the attitude of the vehicle changes or the eye position of the observer changes. In other words, such a head-up display device forms an augmented reality by adding and displaying virtual objects to the real scenery (foreground), and even when the vehicle's posture changes (which also leads to a change in the observer's eye position relative to the real scenery) or the observer's eye position changes inside the vehicle, the position of the image displayed inside the head-up display device is corrected in response to the change in the observer's eye position detected by an eye position detection unit such as a camera, thereby giving the virtual object motion parallax and allowing the observer to perceive the virtual object as if it were at the target position in the foreground (real scene). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-156608 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, an eye position detection unit such as a camera detects the eye positions (left and right eye positions) of a viewer from a captured image or the like using a complex algorithm. However, even if the viewer's eye position moves only a predetermined amount in the left and right direction, it may be detected (misdetected) as having moved by an amount greater than the predetermined amount due to a detection error or erroneous detection depending on the detection environment. Then, the motion parallax image of the virtual object may be corrected according to the erroneously detected amount of movement. As a result, the position correction of the image in the left and right direction may be greater than the amount of movement of the viewer's eye position, resulting in an amount of movement greater than the viewer's actual perception, which may cause the viewer to feel uncomfortable.

[0005] A summary of certain embodiments disclosed herein is provided below. It should be understood that these aspects are presented merely to provide the reader with an overview of these certain embodiments, and are not intended to limit the scope of the disclosure. Indeed, the disclosure may encompass a variety of aspects that are not set forth below.

[0006] The present disclosure relates to providing a display control device, a head-up display device, a display control method, and the like that provide a natural motion parallax to an observer. [Means for solving the problem]

[0007] Therefore, the display control device, head-up display device, and display control method described in this specification employ the following means to solve the above problems. This embodiment is a display control device (30) that executes display control in a head-up display device (20) that is composed of at least a display (40) that displays an image and a relay optical system (80) that projects the light of the image displayed by the display (40) onto a projection target member, and causes a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, the display control device (30) comprising one or more processors (33), a memory (37), one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33), and an outside-vehicle sensor (411) that detects real objects present around the vehicle, and the display control device (30) includes a display unit (40) that displays an image, a relay optical system (80) that projects the light of the image displayed by the display unit (40) onto a projection target member, and the display control device (30) executes display control in a head-up display device (20) that causes a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, the display control device (30) comprising one or more processors (33), a memory (37), one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33), and an outside-vehicle sensor (411) that detects real objects present around the vehicle, One or more The processor (33) determines the user's vertical eye position of the vehicle. Okimata is the head position (Py) and the eye position in the left and right direction of the vehicle Okimata Acquires the head position (Px), and the eye position or head position (Py) in the vertical direction Change in (ΔPy) And the change in eye position or head position (Px) in the left-right direction ( ΔPx and a second image correction process (S170) for correcting the position of the image to be displayed on the display device (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or smaller than 1. Superimposed on the virtual image Distance to the real object in the foreground The object distance is (D4) executes the first image correction process if the first distance threshold is less than or equal to the first distance threshold; The object distance When (D4) is greater than the first distance threshold, the second image correction process (S170) is executed. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing an example of application of a virtual image display system for a vehicle to a vehicle. [Diagram 2] FIG. 2 is a diagram showing a configuration of the head-up display device. [Diagram 3] FIG. 3 is a diagram showing an example of a foreground viewed by an observer while the host vehicle is traveling, and a virtual image displayed superimposed on the foreground. [Figure 4] FIG. 4 is a conceptual diagram showing the positional relationship between the left viewpoint virtual image and the right viewpoint virtual image displayed on the virtual image formation surface and the perceived image perceived by the observer from these left viewpoint virtual image and right viewpoint virtual image in an embodiment in which the HUD device is a 3D-HUD device. [Diagram 5] FIG. 5 is a diagram conceptually showing a virtual object placed at a target position in a real scene, and an image displayed in a virtual image display area so that the virtual object is visually recognized at the target position in the real scene. [Figure 6] FIG. 6 is a diagram for explaining a method of motion parallax adding processing in this embodiment. [Figure 7A] FIG. 7A is a comparative example showing a virtual image viewed from the position Px12 shown in FIG. 6 when the motion parallax adding process of the present embodiment is not performed. [Figure 7B] FIG. 7B is a diagram showing a virtual image viewed from the position Px12 shown in FIG. 6 when the motion parallax adding process of the present embodiment is performed. [Figure 8] FIG. 8 is a diagram for explaining a method of processing for adding motion parallax by moving the eye position (head position) in the up and down direction in this embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a foreground viewed by an observer while the host vehicle is traveling, and a virtual image displayed superimposed on the foreground. [Figure 10] FIG. 10 is a block diagram of a vehicle virtual image display system according to some embodiments. [Figure 11A] FIG. 11A is a flow diagram illustrating a method S100 for performing operations to correct an image based on an observer's eye position. [Figure 11B] FIG. 11B is a diagram for explaining a part of the flow diagram shown in FIG. 11A. [Figure 11C]FIG. 11C is a diagram for explaining a part of the flow diagram shown in FIG. 11A. [Figure 12] FIG. 12 is an image diagram showing the eye position detected at each predetermined period of time, the amount of change in the eye position, the moving speed of the eye position, etc. [Figure 13] FIG. 13 is a diagram for explaining the relationship between the object distance and the correction amount ratio for the amount of change in eye position in the left-right and up-down directions. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, in Figs. 1 to 6 and Figs. 7B to 13, an explanation of the configuration and operation of an exemplary vehicle display system will be provided. Note that the present invention is not limited to the following embodiments (including the contents of the drawings). Of course, modifications (including the deletion of components) can be made to the following embodiments. In addition, in the following explanation, explanations of known technical matters will be omitted as appropriate in order to facilitate understanding of the present invention.

[0010] Please refer to FIG. 1. FIG. 1 is a diagram showing an example of the configuration of a vehicle virtual image display system including a parallax-type 3D-HUD device. In FIG. 1, the left-right direction (in other words, the width direction of the vehicle 1) of a vehicle (an example of a moving body) 1 is the X-axis (the positive direction of the X-axis is the left direction when the vehicle 1 faces forward), the up-down direction (in other words, the height direction of the vehicle 1) along a line segment that is perpendicular to the left-right direction and perpendicular to the ground or a surface equivalent to the ground (here, the road surface 6) is the Y-axis (the positive direction of the Y-axis is the upward direction), and the front-rear direction along a line segment that is perpendicular to each of the left-right direction and the up-down direction is the Z-axis (the positive direction of the Z-axis is the straight-ahead direction of the vehicle 1). This point is the same in other drawings.

[0011] As shown in the figure, the vehicle display system 10 provided in a vehicle (host vehicle) 1 is composed of an eye position detection unit 409 for pupil (or face) detection that detects the position and gaze direction of the left eye 700L and right eye 700R of an observer (typically a driver sitting in the driver's seat of the host vehicle 1), an external sensor 411 composed of a camera (e.g., a stereo camera) that captures an image in front of the host vehicle 1 (broadly speaking, the surroundings), a head-up display device (hereinafter also referred to as a HUD device) 20, and a display control device 30 that controls the HUD device 20.

[0012] 2 is a diagram showing one embodiment of the configuration of a head-up display device. The HUD device 20 is arranged, for example, in a dashboard (reference numeral 5 in FIG. 1). The HUD device 20 has a stereoscopic display device (one example of a display device) 40, a relay optical system 80, and a housing 22 that houses the stereoscopic display device 40 and the relay optical system 80 and has a light exit window 21 that can emit display light K from the stereoscopic display device 40 from the inside to the outside.

[0013] Here, the stereoscopic display device 40 is a parallax 3D display device. The stereoscopic display device (parallax 3D display device) 40 is composed of a light modulation element 50, which is a naked-eye stereoscopic display device using a multi-viewpoint image display method that can control the depth expression by visually recognizing a left viewpoint image and a right viewpoint image, and a light source unit 60 that functions as a backlight.

[0014] The light modulation element 50 includes a light modulation element 51 that modulates the illumination light from the light source unit 60 to generate an image, and an optical layer (an example of a light beam separator) 52 that has, for example, a lenticular lens or a parallax barrier and separates the light emitted from the light modulation element 51 into left-eye display light (reference symbol K10 in FIG. 1) such as light beams K11, K12, and K13 for the left eye, and right-eye display light (reference symbol K20 in FIG. 1) such as light beams K21, K22, and K23 for the right eye. The optical layer 52 includes an optical filter such as a lenticular lens, a parallax barrier, a lens array, and a microlens array. However, this is an example and is not limited to the present invention. The embodiment of the optical layer 52 is not limited to the optical filter described above, and includes all forms of optical layers disposed on the front or rear surface of the light modulation element 51, as long as the optical layer generates the left eye display light (reference K10 in FIG. 1) and the right eye display light (reference K20 in FIG. 1) from the light emitted from the light modulation element 51. Some embodiments of the optical layer 52 may be electrically controlled to generate the left eye display light (reference K10 in FIG. 1) and the right eye display light (reference K20 in FIG. 1) from the light emitted from the light modulation element 51, and examples of such embodiments include a liquid crystal lens. That is, the embodiment of the optical layer 52 may include an electrically controlled one and an electrically non-controlled one.

[0015] Furthermore, the stereoscopic display device 40 may emit display light for the left eye (symbol K10 in FIG. 1), such as light rays K11, K12, and K13 for the left eye, and display light for the right eye (symbol K20 in FIG. 1), such as light rays K21, K22, and K23 for the right eye, by configuring the light source unit 60 with a directional backlight unit (example of a beam splitter) instead of or in addition to the optical layer (example of a beam splitter) 52. Specifically, for example, when the directional backlight unit irradiates illumination light toward the left eye 700L, the display control device 30 (described later) causes the light modulation element 51 to display a left viewpoint image, thereby directing left-eye display light K10 such as light rays K11, K12, and K13 for the left eye to the left eye 700L of the observer, and when the directional backlight unit irradiates illumination light toward the right eye 700R, causes the light modulation element 51 to display a right viewpoint image, thereby directing right-eye display light K20 such as light rays K21, K22, and K23 for the right eye to the right eye 700R of the observer. However, this is just an example and is not limited to the above.

[0016] The display control device 30 described later can control the aspect of the perceived virtual image FU displayed by the HUD device 20 (perceived by the observer) by directing the left eye display light K10 of the left viewpoint image V10 to the left eye 700L of the observer and the right eye display light K20 of the right viewpoint image V20 to the right eye 700R by executing, for example, image rendering processing (graphic processing) and display drive processing, and adjusting the left viewpoint image V10 and the right viewpoint image V20. Note that the display control device 30 described later may control the display (light modulation element 50) to generate a light field that reproduces (approximately) the light rays output in various directions from a point existing in a certain space.

[0017] The relay optical system 80 has curved mirrors (concave mirrors or the like) 81 and 82 that reflect light from the stereoscopic display device 40 and project image display lights K10 and K20 onto the windshield (projection target member) 2. However, it may further have other optical members (which may include refractive optical members such as lenses, diffractive optical members such as holograms, reflective optical members, or combinations of these).

[0018] In FIG. 1, a stereoscopic display device 40 of the HUD device 20 displays images with parallax (parallax images) for the left and right eyes. As shown in FIG. 1, each parallax image is displayed as V10 and V20 formed on a virtual image display surface (virtual image forming surface) VS. The focus of each eye of the observer (person) is adjusted to match the position of the virtual image display area VS. The position of the virtual image display area VS is referred to as an "adjustment position (or imaging position)," and the distance from a predetermined reference position (for example, the center 205 of the eye box 200 of the HUD device 20, the observer's viewpoint position, the observer's head position, or a specific position of the vehicle 1) to the virtual image display area VS (see reference symbol D10 in FIG. 4) is referred to as an adjustment distance (imaging distance).

[0019] However, in reality, since the human brain fuses each image (virtual image), the human recognizes that the perceived image (here, a graphic of an arrowhead for navigation) FU is displayed at a position further back than the adjustment position (for example, a position determined by the convergence angle between the left viewpoint image V10 and the right viewpoint image V20, and the smaller the convergence angle, the farther the position is perceived to be from the observer). The perceived virtual image FU may be called a "stereoscopic virtual image", and may also be called a "stereoscopic image" when the "image" is taken in a broad sense to include virtual images. It may also be called a "stereoscopic image", "3D display", etc. The HUD device 20 may display the left viewpoint image V10 and the right viewpoint image V20 so that the perceived virtual image FU is visually recognized at a position in front of the adjustment position.

[0020] Next, reference is made to Fig. 3 and Fig. 4. Fig. 3 is a diagram showing an example of a foreground viewed by an observer while the vehicle 1 is traveling, and a perceived image displayed superimposed on the foreground. Fig. 4 is a diagram conceptually showing the positional relationship between a left viewpoint virtual image and a right viewpoint virtual image displayed on the virtual image forming surface, and a perceived image perceived by the observer from these left viewpoint virtual image and right viewpoint virtual image.

[0021] In Fig. 3, the vehicle 1 is traveling on a straight road (road surface) 6. The HUD device 20 is disposed inside the dashboard 5. Display light K (K10, K20) is projected from a light exit window 21 of the HUD device 20 onto a projection target (the front windshield of the vehicle 1) 2. In the example of Fig. 3, a first content image FU1 that is superimposed on the road surface 6 and indicates the route of the vehicle 1 (indicating going straight here), and a second content image FU2 that also indicates the route of the vehicle 1 (indicating going straight here) and is perceived to be farther away than the first content image FU1 are displayed.

[0022] As shown in the left diagram of Fig. 4, the HUD device 20 (1) emits a left-eye display light K10 to the projection target unit 2 at a position and angle such that the light is reflected by the projection target unit 2 to the left eye 700L detected by the eye position detection unit 409, and forms a first left viewpoint content image V11 at a predetermined position of the virtual image display area VS as viewed from the left eye 700L, and (2) emits a right-eye display light K20 to the projection target unit 2 at a position and angle such that the light is reflected by the projection target unit 2 to the right eye 700R, and forms a first right viewpoint content image V21 at a predetermined position of the virtual image display area VS as viewed from the right eye 700R. The first content image FU1 perceived by the first left viewpoint content image V11 and the first right viewpoint content image V21 having parallax is visually recognized at a position that is a distance D21 behind the virtual image display area VS (a position that is a distance D31 away from the above-mentioned reference position).

[0023] Similarly, as shown in the right diagram of Fig. 4, the HUD device 20 (1) emits a left-eye display light K10 to the projection target unit 2 at a position and angle such that the light is reflected by the projection target unit 2 to the left eye 700L detected by the eye position detection unit 409, and forms a second left-viewpoint content image V12 at a predetermined position of the virtual image display area VS as viewed from the left eye 700L, and (2) emits a right-eye display light K20 to the projection target unit 2 at a position and angle such that the light is reflected by the projection target unit 2 to the right eye 700R, and forms a second right-viewpoint content image V22 at a predetermined position of the virtual image display area VS as viewed from the right eye 700R. The second content image FU2 perceived by the second left-viewpoint content image V12 and the second right-viewpoint content image V22 having parallax is visually recognized at a position that is a distance D22 behind the virtual image display area VS (a position that is a distance D32 away from the above-mentioned reference position).

[0024] Specifically, the distance from the above-mentioned reference position to the virtual image display area VS (imaging distance (first distance threshold) D10) is set to, for example, 4 m, the distance from the above-mentioned reference position to the first content image FU1 shown in the left diagram of Fig. 4 (first perceptual distance D31) is set to, for example, 7 m, and the distance from the above-mentioned reference position to the second content image FU2 shown in the right diagram of Fig. 4 (second perceptual distance D32) is set to, for example, 10 m. However, this is just an example and is not limiting.

[0025] FIG. 5 is a conceptual diagram showing a virtual object arranged at a target position in a real scene, and an image displayed in a virtual image display area so that the virtual object is visually recognized at the target position in the real scene. The HUD device 20 shown in FIG. 5 shows an example of 2D display, not 3D display. That is, the display 40 of the HUD device 20 shown in FIG. 5 is a 2D display device, not a stereoscopic display device (note that 2D display is also possible with a stereoscopic display device). As shown in FIG. 5, the depth direction is the Z-axis direction, the left-right direction (width direction of the host vehicle 1) is the X-axis direction, and the up-down direction (up-down direction of the host vehicle 1) is the Y-axis direction, as viewed from the viewer's eye position 700. The direction going away from the viewer is the positive direction of the Z-axis, the left direction from the viewer is the positive direction of the X-axis, and the up-down direction from the viewer is the positive direction of the Y-axis.

[0026] The viewer's eye position 700 perceives a virtual object FU at a predetermined target position PT in the real scene by viewing the virtual image V formed (imaged) in the virtual image display area VS via the projection unit 2. The viewer views the virtual image V of the image of the display light K reflected by the projection unit 2. At this time, if the virtual image V is, for example, an arrow indicating a course, the arrow of the virtual image V is displayed in the virtual image display area VS so that the virtual object FU is located and viewed at the predetermined target position PT in the foreground of the vehicle 1. Specifically, the HUD device 20 (display control device 30) renders an image to be displayed on the display 40 such that a virtual object FU of a predetermined size and shape located at the target position PT is projected and transformed into the virtual image display area VS, with the center between the viewer's left eye 700L and right eye 700R as the origin of the projective transformation. The HUD device 20 (display control device 30) changes the position of the virtual image V displayed in the virtual image display area VS so that the virtual object FU is perceived at the same target position PT as before the eye position was moved, even if the virtual object FU (virtual image V) is displayed at a position (virtual image display area VS) away from the target position PT, thereby making it possible to make the virtual object FU (virtual image V) be recognized as if it were at the target position PT. That is, the HUD device 20 (display control device 30) expresses natural motion parallax by changing the position (which may include a change in size or shape) of the image (virtual image V in the virtual image display area VS) on the display 40 based on the movement of the eye position (in other words, the HUD device 20 adds motion parallax to the virtual image (image) through image correction accompanying the movement of the eye position, making it easier to sense depth perception).

[0027] In the description of this embodiment, such correction of image position to express motion parallax in response to a change in eye position is called motion parallax addition processing (an example of eye tracking image correction processing). The motion parallax addition processing is not limited to image position correction to completely reproduce natural motion parallax, but may also include image position correction to approximate natural motion parallax. Note that the HUD device 20 (display control device 30) may execute not only the motion parallax addition processing (an example of eye tracking image correction processing) in response to a change in eye position 700, but also the motion parallax addition processing (an example of eye tracking image correction processing) based on the observer's head position 710 instead of the eye position 700.

[0028] 6 is a diagram for explaining a method of motion parallax addition processing in this embodiment. The display control device 30 (processor 33) of this embodiment controls the HUD device 20 and displays virtual images V41, V42, and V43 formed (imaged) in the virtual image display area VS via the projection unit 2. The virtual image V41 is set to a target position PT11 at a perceived distance D33 (a position that is a distance D23 behind the virtual image display area VS), the virtual image V42 is set to a target position PT12 at a perceived distance D34 (a position that is a distance D24 (>D23) behind the virtual image display area VS) that is longer than the perceived distance D33 of the virtual image V41, and the virtual image V43 is set to a target position PT13 at a perceived distance D35 (a position that is a distance D25 (>D24) behind the virtual image display area VS) that is longer than the perceived distance D34 of the virtual image V42. In addition, since the amount of correction of the image on the display 40 corresponds to the amount of correction of the virtual image in the virtual image display area VS, in Figure 6, the correction amounts of the virtual image corresponding to the correction amounts C1, C2, and C3 of the image on the display 40 also use the same symbols C1, C2, and C3 (the same applies to the symbols Cy11 (Cy) and Cy21 (Cy) in Figure 8).

[0029] When the head position 710 (eye position 700) of the observer moves from the position of Px11 to the right (negative direction of the X-axis) by ΔPx10, the display control device 30 (processor 33) executes a motion parallax addition process to correct the positions where the virtual images V41, V42, and V43 displayed in the virtual image display area VS are displayed by the correction amounts C1, C2 (>C1), and C3 (>C2), respectively, in the same direction as the movement of the head position 710 (eye position 700) of the observer. Fig. 7A is a comparative example showing the virtual images V41, V42, and V43 viewed from the position Px12 shown in Fig. 6 when the motion parallax addition process of this embodiment is not performed, and Fig. 7B is a diagram showing the virtual images V44, V45, and V46 viewed from the position Px12 shown in Fig. 6 when the motion parallax addition process of this embodiment is performed. Note that in FIG. 7B, the difference in the positions of the virtual images V44, V45, and V46 is exaggerated so that the difference in the correction amount can be easily seen. That is, the display control device 30 (processor 33) makes the correction amount of the positions of the virtual images V41, V42, and V43 associated with the movement of the eye position different depending on the difference in the perception distances D33, D34, and D35 of the virtual images V41, V42, and V43, so that the observer can sense the motion parallax only between the virtual images V41 (V44), V42 (V45), and V43 (V46). More specifically, the display control device 30 (processor 33) adds motion parallax to the virtual images V41 (V44), V42 (V45), and V43 (V46) by increasing the correction amount in the motion parallax adding process as the set perception distance D30 becomes longer.

[0030] 8 is a diagram for explaining a method of motion parallax addition processing when the eye position (head position) moves up and down in this embodiment. When the head position 710 (eye position 700) of the observer moves upward (Y-axis positive direction) from the position of reference symbol Py12, the display control device 30 (processor 33) executes motion parallax addition processing to correct the position where the virtual image V displayed in the virtual image display area VS is displayed in the same direction (upward (Y-axis positive direction)) as the head position 710 (eye position 700) of the observer moves, as shown in FIG. 8(a), by a correction amount Cy11 (changing the position of the virtual image V from the position of reference symbol V48 to reference symbol V47). In addition, when the head position 710 (eye position 700) of the observer moves downward (in the negative Y-axis direction) from the position of Py12, the display control device 30 (processor 33) executes a motion parallax addition process to correct the position where the virtual image V displayed in the virtual image display area VS is displayed in the same direction (downward (in the negative Y-axis direction)) as the head position 710 (eye position 700) of the observer has moved, by the correction amount Cy21 (changing the position of the virtual image V from the position of V48 to V49). This allows the virtual object FU (virtual image V) to be recognized as if it is at the target position PT, even though it is displayed at a position (virtual image display area VS) away from the target position PT (the sensation that the virtual object FU (virtual image V) is at the target position PT can be enhanced).

[0031] FIG. 9 is a diagram showing a real object 300 present in the foreground, which is visually recognized when an observer faces forward from the driver's seat of the vehicle 1, and a virtual image V displayed by the HUD device 20 of this embodiment. The virtual image V shown in FIG. 9 includes an AR virtual image V60 whose display position can be changed according to the position of the real object 300, and a non-AR virtual image V70, which is a virtual image whose display position is set regardless of the position of the real object 300. The AR virtual image V60 is displayed at a position (target position PT) corresponding to the position of the real object 300 present in the real scene. The AR virtual image V60 is displayed, for example, at a position superimposed on the real object 300 or in the vicinity of the real object 300, and the presence of the real object 300 is emphasized and notified. In other words, the "position (target position PT) corresponding to the position of the real object 300" is not limited to a position superimposed on the real object 300 and visually recognized by the observer, but may be a position in the vicinity of the real object 300. The AR virtual image V60 may be of any shape as long as it does not interfere with the visibility of the real object 300.

[0032] 9 includes navigation virtual images V61 and V62 indicating a guide route, enhanced virtual images V64 and V65 emphasizing and informing of an object of attention, and a POI virtual image V65 indicating a target object or a specific building. The position (target position PT) corresponding to the position of the real object 300 is the position of the road surface 311 (an example of the real object 300) on which the navigation virtual images V61 and V62 are superimposed, the position of the periphery of the person 313 (an example of the real object 300) in the enhanced virtual image V63, the position of the vicinity of another vehicle 314 (an example of the real object 300) in the enhanced virtual image V64, and the position of the periphery of the building 315 (an example of the real object 300) in the POI virtual image V65. As described above, the display control device 30 (processor 33) increases the correction amount C associated with the movement of the observer's eye position in the motion parallax adding process as the perception distance D30 set for the virtual image V becomes longer. 9, the order of the perception distance D30 set for the virtual image V is V65→V64→V63→V62→V61 from the longest, and the display control device 30 (processor 33) sets the correction amount C associated with the movement of the observer's eye position as follows: correction amount of V65>correction amount of V64>correction amount of V63>correction amount of V62>correction amount of V61. Note that since the virtual images V62 and V61 are the same type of virtual images and are displayed close to each other, the display control device 30 (processor 33) may set the correction amount of V62 associated with the movement of the observer's eye position and the correction amount of V61 to be the same.

[0033] In addition, the display control device 30 (processor 33) of some embodiments may set the correction amount C associated with the movement of the observer's eye position in the non-AR virtual image V70 to zero (no correction may be required associated with the movement of the observer's eye position).

[0034] In addition, the display control device 30 (processor 33) of some embodiments may correct the non-AR virtual image V70 according to the movement of the observer's eye position. In the example shown in Fig. 9, the non-AR virtual images V70 (V71, V72) are arranged below the virtual image display area VS, and the area of ​​the road surface 311 that is the real object 300 that overlaps with them is closer to the host vehicle 1 than the area of ​​the road surface 311 that overlaps with the navigation virtual image V61 in Fig. 9. That is, in some embodiments, the display control device 30 (processor 33) may set the perceived distance D30 of the non-AR virtual image V70 (V71, V72) shorter than the perceived distance D30 of the AR virtual image V60 (narrowly speaking, the navigation virtual image V61 located at the lowest among the AR virtual images V60), and set the correction amount C of the non-AR virtual image V70 associated with movement of the observer's eye position to be smaller than the correction amount C of the AR virtual image V60 (narrowly speaking, the navigation virtual image V61 located at the lowest among the AR virtual images V60) associated with movement of the observer's eye position.

[0035] FIG. 10 is a block diagram of a vehicle virtual image display system according to some embodiments. The display controller 30 includes one or more I / O interfaces 31, one or more processors 33, one or more image processing circuits 35, and one or more memories 37. The various functional blocks described in FIG. 3 may be configured in hardware, software, or a combination of both. FIG. 10 is just one embodiment, and the components shown may be combined into fewer components, or there may be additional components. For example, the image processing circuit 35 (e.g., a graphics processing unit) may be included in one or more processors 33.

[0036] As shown, the processor 33 and the image processing circuit 35 are operatively coupled to the memory 37. More specifically, the processor 33 and the image processing circuit 35 can execute programs stored in the memory 37 to control the vehicle display system 10 (display 40), for example, to generate and / or transmit image data. The processor 33 and / or the image processing circuit 35 can include at least one general-purpose microprocessor (e.g., a central processing unit (CPU)), at least one application specific integrated circuit (ASIC), at least one field programmable gate array (FPGA), or any combination thereof. The memory 37 can include any type of magnetic medium, such as a hard disk, any type of optical medium, such as a CD and a DVD, any type of semiconductor memory, such as a volatile memory, and a non-volatile memory. The volatile memory can include DRAM and SRAM, and the non-volatile memory can include ROM and NVRAM.

[0037] As shown in the figure, the processor 33 is operably connected to the I / O interface 31. The I / O interface 31 communicates with, for example, a vehicle ECU 401 (described later) and / or other electronic devices (reference numerals 403 to 419 described later) provided in the vehicle in accordance with the CAN (Controller Area Network) standard (also referred to as CAN communication). Note that the communication standard adopted by the I / O interface 31 is not limited to CAN, and may include, for example, wired communication interfaces such as CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport: MOST is a registered trademark), UART, or USB, or in-vehicle communication (internal communication) interfaces that are short-range wireless communication interfaces within several tens of meters, such as a personal area network (PAN) such as a Bluetooth (registered trademark) network, a local area network (LAN) such as an 802.11x Wi-Fi (registered trademark) network, etc. In addition, the I / O interface 31 may include an external communication (external communication) interface such as a wide area communication network (e.g., an Internet communication network) based on cellular communication standards such as a wireless wide area network (WWAN0, IEEE802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE802.16e-based (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, and 5G.

[0038] As shown in the figure, the processor 33 is interoperably connected to the I / O interface 31, thereby enabling information to be exchanged with various other electronic devices connected to the vehicle display system 10 (I / O interface 31). For example, a vehicle ECU 401, a road information database 403, a vehicle position detection unit 405, an operation detection unit 407, an eye position detection unit 409, an outside sensor 411, a brightness detection unit 413, an IMU 415, a mobile information terminal 417, and an external communication device 419 are operably connected to the I / O interface 31. The I / O interface 31 may include a function of processing (converting, calculating, analyzing) information received from other electronic devices connected to the vehicle display system 10.

[0039] The display 40 is operatively coupled to the processor 33 and the image processing circuitry 35. Thus, the image displayed by the light modulation elements 51 may be based on image data received from the processor 33 and / or the image processing circuitry 35. The processor 33 and the image processing circuitry 35 control the image displayed by the light modulation elements 51 based on information obtained from the I / O interface 31.

[0040] The vehicle ECU 401 acquires the state of the host vehicle 1 (for example, the ON / OFF state (one example of startup information) of the startup switch (for example, accessory switch: ACC or ignition switch: IGN), mileage, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine RPM, motor RPM, steering angle, shift position, drive mode, various warning states, posture (including roll angle and / or pitching angle), vehicle vibration (including vibration magnitude, frequency, and / or frequency)) from sensors and switches provided in the host vehicle 1, and collects and manages (which may also include control) the said state of the host vehicle 1, and as part of its function, can output a signal indicating a numerical value of the said state of the host vehicle 1 (for example, the vehicle speed of the host vehicle 1) to the processor 33 of the display control device 30. In addition to or instead of simply transmitting a value detected by a sensor or the like (e.g., the pitching angle is 3 degrees in the forward tilt direction) to the processor 33, the vehicle ECU 401 may transmit to the processor 33 a determination result based on one or more states of the host vehicle 1 including the value detected by the sensor (e.g., that the host vehicle 1 satisfies a predetermined condition for a forward tilt state) and / or an analysis result (e.g., that the vehicle has been in a forward tilt state due to braking in combination with information on the brake pedal opening degree). For example, the vehicle ECU 401 may output to the display control device 30 a signal indicating a determination result that the host vehicle 1 satisfies a predetermined condition previously stored in a memory (not shown) of the vehicle ECU 401. The I / O interface 31 may acquire the above-mentioned information from a sensor or a switch provided on the host vehicle 1 without going through the vehicle ECU 401.

[0041] In addition, the vehicle ECU 401 may output an instruction signal to the display control device 30 to indicate the image to be displayed by the vehicle display system 10, and at this time, the image coordinates, size, type, display mode, notification necessity of the image, and / or necessity-related information that serves as the basis for determining the notification necessity may be added to the instruction signal and transmitted.

[0042] The road information database 403 is included in a navigation device (not shown) provided in the vehicle 1, or an external server connected to the vehicle 1 via an external communication interface (I / O interface 31), and may read out information about the surroundings of the vehicle 1 (information related to real objects around the vehicle 1), such as road information (lanes, white lines, stop lines, pedestrian crossings, road widths, number of lanes, intersections, curves, forks, traffic regulations, etc.) on which the vehicle 1 is traveling, and information about features (buildings, bridges, rivers, etc.), such as presence or absence, position (including distance to the vehicle 1), direction, shape, type, detailed information, etc.), and transmit the information to the processor 33. The road information database 403 may also calculate an appropriate route (navigation information) from a departure point to a destination, and output a signal indicating the navigation information or image data indicating the route to the processor 33.

[0043] The vehicle position detection unit 405 is a GNSS (Global Navigation Satellite System) or the like provided in the vehicle 1, which detects the current position and direction of the vehicle 1, and outputs a signal indicating the detection result via the processor 33 or directly to the road information database 403, a mobile information terminal 417 described later, and / or an external communication device 419. The road information database 403, the mobile information terminal 417 described later, and / or the external communication device 419 may select / generate information about the surroundings of the vehicle 1 by acquiring the position information of the vehicle 1 from the vehicle position detection unit 405 continuously, intermittently, or for each predetermined event, and output the information to the processor 33.

[0044] The operation detection unit 407 is, for example, a hardware switch provided on a CID (Center Information Display) or an instrument panel of the vehicle 1, or a software switch combining an image and a touch sensor, and outputs operation information based on an operation by an occupant of the vehicle 1 (a user sitting in the driver's seat and / or a user sitting in the passenger seat) to the processor 33. For example, the operation detection unit 407 outputs to the processor 33 display area setting information based on an operation to move the virtual image display area 100 by a user's operation, eye box setting information based on an operation to move the eye box 200, information based on an operation to set the observer's eye position 700, and the like.

[0045] Eye position detection unit 409 may include a camera such as an infrared camera that detects eye position 700 (see FIG. 1) of an observer sitting in the driver's seat of host vehicle 1, and may output a captured image to processor 33. Processor 33 may obtain a captured image (one example of information from which eye position 700 can be estimated) from eye position detection unit 409, and analyze the captured image using a method such as pattern matching to detect the coordinates of observer's eye position 700, and output a signal indicating the coordinates of the detected eye position 700 to processor 33.

[0046] Furthermore, eye position detection unit 409 may output an analysis result of analyzing the image captured by the camera (for example, a signal indicating to which spatial region the observer's eye position 700 belongs in which a plurality of preset display parameters correspond) to processor 33. Note that the method of acquiring the observer's eye position 700 of the host vehicle 1 or information that allows the observer's eye position 700 to be estimated is not limited to these, and the information may be acquired using a known eye position detection (estimation) technique.

[0047] Furthermore, the eye position detection section 409 may detect the moving speed and / or moving direction of the viewer's eye position 700, and output a signal indicating the moving speed and / or moving direction of the viewer's eye position 700 to the processor 33.

[0048] The eye position detection unit 409 may also have a function as the gaze direction detection unit 409. The gaze direction detection unit 409 may include an infrared camera or a visible light camera that captures an image of the face of an observer sitting in the driver's seat of the vehicle 1, and may output the captured image to the processor 33. The processor 33 can specify the gaze direction (and / or the gaze position) of the observer by acquiring an image (one example of information that can estimate the gaze direction) from the gaze direction detection unit 409 and analyzing the captured image. The gaze direction detection unit 409 may analyze an image captured by a camera and output a signal indicating the gaze direction (and / or the gaze position) of the observer, which is the analysis result, to the processor 33. The method of acquiring information that can estimate the gaze direction of the observer of the vehicle 1 is not limited to these, and may be acquired using other known gaze direction detection (estimation) techniques such as an EOG (Electro-oculogram) method, a corneal reflection method, a scleral reflection method, a Purkinje image detection method, a search coil method, and an infrared fundus camera method.

[0049] The external sensor 411 detects real objects present around the vehicle 1 (in front, on the sides, and behind). The real objects detected by the external sensor 411 may include, for example, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), the road surface of a driving lane, dividing lines, roadside objects, and / or features (buildings, etc.) described below. The external sensor is composed of, for example, a detection unit made up of a radar sensor such as a millimeter wave radar, ultrasonic radar, or laser radar, a camera, or a combination of these, and a processing device that processes (data fuses) detection data from the one or more detection units. Conventional well-known methods are applied to object detection by these radar sensors and camera sensors. By detecting objects using these sensors, it is possible to detect the presence or absence of a real object in three-dimensional space, and if a real object is present, the position of the real object (relative distance from the vehicle 1, left / right position, up / down position, etc. when the direction of travel of the vehicle 1 is the forward / backward direction), size (size in the lateral direction (left / right), height direction (up / down), etc.), direction of movement (lateral direction (left / right), depth direction (front / back)), speed of movement (lateral direction (left / right), depth direction (front / back)), and / or type, etc.).

[0050] One or more external sensors 411 can detect a real object in front of the vehicle 1 at each detection period of the sensor, and output real object information (presence or absence of a real object, and if a real object exists, information on the position, size, and / or type of each real object, etc.) which is an example of real object information to the processor 33. The real object information may be transmitted to the processor 33 via another device (for example, the vehicle ECU 401). In addition, when a camera is used as a sensor, an infrared camera or a near-infrared camera is preferable so that a real object can be detected even when the surroundings are dark, such as at night. In addition, when a camera is used as a sensor, a stereo camera that can obtain distance and the like by parallax is preferable. Here, the relative distance of the detected real object 300 from the vehicle 1 is defined as an object distance (distance to the real object) D4.

[0051] The brightness detection unit 413 detects the illuminance or brightness of a predetermined range of the foreground present in front of the vehicle interior as the outside brightness (an example of brightness information), or detects the illuminance or brightness inside the vehicle interior as the inside vehicle brightness (an example of brightness information). The brightness detection unit 413 is, for example, a phototransistor or a photodiode, and is mounted on the instrument panel, the rearview mirror, the HUD device 20, or the like of the vehicle 1 shown in FIG.

[0052] The IMU 415 may include a combination of one or more sensors (e.g., an accelerometer and a gyroscope) configured to detect the position, orientation, and changes therein (speed of change, acceleration of change) of the host vehicle 1 based on inertial acceleration. The IMU 415 may output the detected values ​​(the detected values ​​include signals indicating the position, orientation, and changes therein (speed of change, acceleration of change) of the host vehicle 1, etc.) and the results of analyzing the detected values ​​to the processor 33. The results of the analysis may be signals indicating a determination result as to whether or not the detected values ​​satisfy a predetermined condition, and may be, for example, a signal indicating that the behavior (vibration) of the host vehicle 1 is small based on a value regarding the change (speed of change, acceleration of change) of the position or orientation of the host vehicle 1.

[0053] The mobile information terminal 417 is a smartphone, a notebook computer, a smartwatch, or other information device that can be carried by an observer (or another passenger of the vehicle 1). The I / O interface 31 can communicate with the mobile information terminal 417 by pairing with the mobile information terminal 417, and acquires data recorded in the mobile information terminal 417 (or a server via the mobile information terminal). The mobile information terminal 417 has, for example, the same functions as the above-mentioned road information database 403 and the vehicle position detection unit 405, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. The mobile information terminal 417 may also acquire commercial information (an example of real object related information) related to commercial facilities in the vicinity of the vehicle 1 and transmit it to the processor 33. In addition, the mobile information terminal 417 may transmit schedule information of the holder of the mobile information terminal 417 (e.g., the observer), information on incoming calls at the mobile information terminal 417, information on received emails, etc. to the processor 33, and the processor 33 and the image processing circuit 35 may generate and / or transmit image data related to these.

[0054] The external communication device 419 is a communication device that exchanges information with the vehicle 1, and is, for example, another vehicle connected to the vehicle 1 by vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), a pedestrian (a mobile information terminal carried by a pedestrian) connected to the vehicle 1 by vehicle-to-pedestrian communication (V2P: Vehicle To Pedestrian), and a network communication device connected by vehicle-to-roadside communication (V2I: Vehicle To Roadside Infrastructure), and in a broad sense, includes everything connected to the vehicle 1 by communication (V2X: Vehicle To Everything). The external communication device 419 may obtain the positions of, for example, pedestrians, bicycles, motorcycles, other vehicles (such as a preceding vehicle), road surfaces, dividing lines, roadside objects, and / or features (such as buildings), and output them to the processor 33. Moreover, the external communication device 419 may have the same function as the vehicle position detection unit 405 described above, and may acquire position information of the vehicle 1 and transmit it to the processor 33, and may also have the function of the road information database 403 described above, and may acquire the road information (an example of real object related information) and transmit it to the processor 33. Note that the information acquired from the external communication device 419 is not limited to the above.

[0055] The software components stored in the memory 37 include an eye position detection module 502, an eye position estimation module 504, an eye position prediction module 506, an eye position state determination module 508, a vehicle state determination module 510, an eye tracking image processing module 512, a graphics module 514, a light source driving module 516, and an actuator driving module 518, among others.

[0056] 11A, 11B, and 11C are flow diagrams showing a method S100 for performing an operation of correcting an image based on an eye position of a viewer. The method S100 is performed in a HUD device 20 including a display and a display control device 30 that controls the HUD device 20. Some operations of the method S100 shown below are optionally combined, the sequence of some operations is optionally changed, and some operations are optionally omitted.

[0057] First, the display control device 30 (processor 33) detects the viewer's eye position 700 (step S110).

[0058] (Step S112) In step S110 in some embodiments, the display control device 30 (processor 33) detects the eye position 700 of the viewer via the eye position detection unit 409 by executing the eye position detection module 502 in Fig. 10 (obtains information indicating the eye position 700). The eye position detection module 502 includes various software components for executing various operations related to detecting coordinates indicating the viewer's eye position 700 (positions in the X and Y axis directions, which are an example of a signal indicating the eye position 700), detecting coordinates indicating the height of the viewer's eyes (positions in the Y axis direction, which are an example of a signal indicating the eye position 700), detecting coordinates indicating the height and depth direction position of the viewer's eyes (positions in the Y and Z axis directions, which are an example of a signal indicating the eye position 700), and / or detecting coordinates indicating the viewer's eye position 700 (positions in the X, Y, and Z axis directions, which are an example of a signal indicating the eye position 700).

[0059] The eye position 700 detected by the eye position detection module 502 may include the right eye and left eye positions 700R and 700L, a predetermined one of the right eye position 700R and the left eye position 700L, one of the right eye position 700R and the left eye position 700L that is detectable (easy to detect), or a position calculated from the right eye position 700R and the left eye position 700L (for example, the midpoint between the right eye position and the left eye position), etc. For example, the eye position detection module 502 determines the eye position 700 based on the observation position acquired from the eye position detection unit 409 immediately before the timing of updating the display settings.

[0060] In addition, the eye position detection unit 409 may detect the movement direction and / or movement speed of the observer's eye position 700 based on multiple observation positions with different detection timings of the observer's eyes acquired from the eye position detection unit 409, and output a signal indicating the movement direction and / or movement speed of the observer's eye position 700 to the processor 33.

[0061] (Step S114) Also, in step S110 in some embodiments, the display control device 30 (processor 33) may acquire information from which the eye position can be estimated by executing the eye position estimation module 504 (step S114). The information from which the eye position can be estimated is, for example, the captured image acquired from the eye position detection unit 409, the position of the driver's seat of the host vehicle 1, the position of the observer's face, the height of the seat, or the observation positions of the eyes of multiple observers. The eye position estimation module 504 estimates the eye position 700 of the observer of the host vehicle 1 from the information from which the eye position can be estimated. The eye position estimation module 504 includes various software components for executing various operations related to estimating the observer's eye position 700, such as estimating the observer's eye position 700 from the captured image acquired from the eye position detection unit 409, the position of the driver's seat of the host vehicle 1, the position of the observer's face, the height of the seat, or the observation positions of the eyes of multiple observers. That is, the eye position estimation module 504 may include table data, arithmetic expressions, and the like for estimating the viewer's eye position 700 from information that enables the eye position to be estimated.

[0062] (Step S116) Also, in step S110 in some embodiments, the display control device 30 (processor 33) may execute the eye position prediction module 506 to obtain information capable of predicting the viewer's eye position 700 (step S116). The information capable of predicting the viewer's eye position 700 may be, for example, the latest observation position obtained from the eye position detection unit 409, or one or more observation positions obtained in the past. The eye position prediction module 506 includes various software components for performing various operations related to predicting the eye position 700 based on the information capable of predicting the viewer's eye position 700. Specifically, for example, the eye position prediction module 506 predicts the viewer's eye position 700 at the time when the viewer views the image to which the new display settings are applied. The eye position prediction module 506 may predict the next value using one or more past observation positions using a prediction algorithm such as, for example, the least squares method, a Kalman filter, an α-β filter, or a particle filter.

[0063] (Step S120) Next, the display control device 30 (processor 33) determines whether a predetermined condition is satisfied (step S120).

[0064] FIG. 11B is a diagram illustrating the operation of the eye position state determination module 508 in step S130.

[0065] (Step S130) In step S120 in some embodiments, the display control device 30 (processor 33) executes the eye position state determination module 508 of FIG. 10 to determine whether the eye position 700 (movement of the eye position 700) satisfies a specified condition based on the information indicating the eye position obtained in step S110, information that allows the eye position to be estimated, or information that allows the eye position to be predicted.

[0066] FIG. 12 shows (11) vertical eye position (or head position) Py (Y1, Y2, Y3, Y10), (12) change amount ΔPy (Py1 (=Y2-Y1), Py2 (=Y3-Y2), Py3 (=Y4-Y3), . . . Py9 (=Y10-Y9)) of the eye position (or head position) in the vertical direction, (13) movement speed Vy (Vy1 (=Py1 / Δt), Vy2 (=Py2 / Δt), Vy3 (=Py3 / Δt), . . . Vy9 (=Py9 / Δt)), (21) left and right Eye position (or head position) in the direction Px(X1, X2, X3, X10), (22) left and right The change in eye position (or head position) in the direction ΔPx (Px1 (=X2-X1), Px2 (=X3-X2), Px3 (=X4-X3), . . . Px9 (=X10-X9)), and (23) left and right 13 is a table showing the movement speed Vx (Vx1 (=Px1 / Δt), Vx2 (=Px2 / Δt), Vx3 (=Px3 / Δt), . . . Vx9 (=Px9 / Δt)) of the eye position (or head position) in the direction.

[0067] (Step S131) In step S130 in some embodiments, the display control device 30 (processor 33) may determine that a predetermined condition is satisfied when the eye position in the left-right direction changes continuously in one direction by executing the eye position state determination module 508 in Fig. 10. The eye position state determination module 508 may determine that the predetermined condition is satisfied when it is detected that the change amount ΔPx of the eye position in the left-right direction shown in Fig. 12 changes continuously in one direction (here, the right direction), such as Px3 (right direction) → Px4 (right direction).

[0068] (Step S132) Furthermore, in step S130 in some embodiments, the display control device 30 (processor 33) may execute the eye position state determination module 508 in Fig. 10 to determine that a predetermined condition is satisfied when the vertical eye position changes continuously in one direction. For example, the eye position state determination module 508 may determine that the predetermined condition is satisfied when it is detected that the change amount ΔPy of the vertical eye position shown in Fig. 12 changes continuously in one direction (here, downward), such as Py3 (downward) -> Py4 (downward).

[0069] (Step S133) In addition, in step S130 in some embodiments, the display control device 30 (processor 33) can execute the distance determination module 511 of Figure 10 to compare, for example, the object distance D4 with the imaging distance D10, and if the object distance D4 is greater than the imaging distance D10, it may determine that a specified condition is satisfied.

[0070] (Step S134) In addition, in step S130 in some embodiments, the display control device 30 (processor 33) may execute the distance determination module 511 in Fig. 10 to compare the object distance D4 with a predetermined second distance threshold (e.g., 20 m) stored in advance in the memory 37, and determine that the predetermined condition is satisfied. Here, the second distance threshold is set to about five times the imaging distance D10.

[0071] FIG. 11C is a diagram illustrating the operation of the vehicle state determination module 510 in step S140.

[0072] (Step S140) In some embodiments, in step S120, the display control device 30 (processor 33) may execute the vehicle state determination module 510 in FIG. 10 to determine whether the state of the vehicle satisfies a predetermined condition.

[0073] (Step S141) In some embodiments, in step S140, the display control device 30 (processor 33) executes the vehicle state determination module 510 of FIG. 10 to estimate whether or not the vehicle 1 is moving based on, for example, information such as mileage, vehicle speed, shift position, etc. obtained from the vehicle ECU 401, information indicating the vehicle position obtained from the vehicle position detection unit 405, and information obtained from the IMU 415, and if it is estimated that the vehicle 1 is moving, it may determine that the specified condition is satisfied.

[0074] (Step S150) 11A again, after it is determined in step S120 whether the predetermined condition is satisfied, the display control device 30 (processor 33) executes the eye tracking image processing module 512 to correct the position of the image to be displayed on the display 40 based on the observer's eye position 700 (specifically, the eye position 700 is the eye position Py in the up-down direction and the eye position Px in the left-right direction in FIG. 12, and may also include the eye position in the front-back direction) (eye tracking image processing).

[0075] The eye tracking image processing module 512 in FIG. 10 switches between a first image correction process S160 (step S160), a second image correction process S170 (step S170), and a third image correction process S180 (step S180) based on the determination result in step S120.

[0076] (First image correction process S160) When it is determined in step S120 that the predetermined condition is satisfied (step S131), the eye tracking image processing module 512 in FIG. 10 corrects the left-right position of the virtual image V by a first left-right correction amount (first correction amount) Cx according to the left-right eye position change amount ΔPx. The first left-right correction amount Cx (similar to the second left-right correction amount Cxq1 and the third left-right correction amount Cxq2 described later) is a parameter that gradually increases as the left-right eye position change amount ΔPx increases. In addition, the first left-right correction amount Cx (similar to the second left-right correction amount Cxq1 and the third left-right correction amount Cxq2 described later) is a parameter that gradually increases as the perception distance D30 set for the virtual image V becomes longer. The first image correction process S160 includes image position correction to perfectly reproduce natural motion parallax as if the virtual image V were fixed at the set target position PT even when viewed from each left-right eye position Px, and in a broader sense, may also include image position correction to approximate natural motion parallax. That is, the first image correction process S160 aligns the display position of the virtual image V with the intersection position of the virtual image display area VS and a straight line connecting the target position PT set on the virtual image V and the viewer's eye position 700 (brings the display position of the virtual image V closer).

[0077] (First image correction process S160) When it is determined in step S120 that the predetermined condition is satisfied (step S132), the eye tracking image processing module 512 in Fig. 10 corrects the vertical position of the virtual image V by a first vertical correction amount (first correction amount) Cy according to the vertical eye position change amount ΔPy. The first vertical correction amount Cy (similar to the second vertical correction amount Cyr1 and the third vertical correction amount Cyr2 described later) is a parameter that gradually increases as the vertical eye position change amount ΔPy increases. In addition, the first vertical correction amount Cy (similar to the second vertical correction amount Cyr1 and the third vertical correction amount Cyr2 described later) is a parameter that gradually increases as the perception distance D30 set for the virtual image V becomes longer. The first image correction process S160 includes image position correction to perfectly reproduce natural motion parallax as if the virtual image V were fixed at the set target position PT even when viewed from each vertical eye position Py, and in a broader sense may also include image position correction to approximate natural motion parallax. That is, the first image correction process S160 aligns the display position of the virtual image V with the intersection position of the virtual image display area VS and the straight line connecting the target position PT set on the virtual image V and the viewer's eye position 700 (brings the display position of the virtual image V closer).

[0078] (Second image correction process S170) When it is determined in step S120 that the predetermined condition is satisfied (step S133), the eye tracking image processing module 512 in FIG. 10 corrects the left-right position of the virtual image V by a second left-right correction amount (second correction amount) Cxq1 according to the left-right eye position change amount ΔPx. As shown in FIG. 13(a), the second left-right correction amount Cxq1 is a value obtained by multiplying the first left-right correction amount Cx for the left-right eye position change amount ΔPx in the first image correction process S160 by a first left-right coefficient (first predetermined coefficient) Q1 of 0.9, which is equal to or less than 1. Specifically, for example, when the first left-right correction amount Cx for the left-right eye position change amount ΔPx is 100%, the second left-right correction amount Cxq1 for the same up-down eye position change amount ΔPx is 90%. In a broad sense, the second left-right correction amount Cxq1 only needs to be smaller than the first left-right correction amount Cx to suppress the first left-right correction amount Cx, so it is sufficient for it to be less than 100% of the first left-right correction amount Cx, but it is preferable for it to be less than 90% of the first left-right correction amount Cx.

[0079] (Second image correction process S170) When it is determined in step S120 that the predetermined condition is satisfied (step S133), the eye tracking image processing module 512 in FIG. 10 corrects the vertical position of the virtual image V by a second vertical correction amount (second correction amount) Cyr1 according to the vertical eye position change amount ΔPy. As shown in FIG. 13(b), the second vertical correction amount Cyr1 is a value obtained by multiplying the first vertical correction amount Cy for the vertical eye position change amount ΔPy in the first image correction process S160 by a first vertical coefficient (first predetermined coefficient) R1 of 0.8, which is equal to or less than 1. Specifically, for example, when the first vertical correction amount Cy for the vertical eye position change amount ΔPy is set to 100%, the second vertical correction amount Cyr1 for the same vertical eye position change amount ΔPy is 80%. In a broad sense, the second vertical correction amount Cyr1 is set to be smaller than the first vertical correction amount Cy. First vertical correction amount Cy It is sufficient to suppress the above-mentioned correction amount Cy, so it is sufficient that the correction amount is less than 100% of the first vertical correction amount Cy, but it is preferable that the correction amount is less than 80% of the first vertical correction amount Cy.

[0080] (An example of step S170) The first left-right coefficient Q1 is preferably greater than the first up-down coefficient R1.

[0081] (An example of step S170) In step S170 in some embodiments, the display control device 30 (processor 33) may, after proceeding to the second image correction process as shown in Fig. 13(a), gradually decrease the second left-right correction amount Cxq1 for the left-right eye position change amount ΔPx over time as it becomes farther from the first left-right correction amount Cx for the left-right eye position change amount ΔPx (see Cxq11). For example, if the first left-right correction amount Cx for the left-right eye position change amount ΔPx is 100% and the second left-right correction amount Cxq1 is 90%, the display control device 30 (processor 33) may gradually decrease the second left-right correction amount Cxq1 over time as it becomes farther from the first left-right correction amount Cx, such as 100% -> 98% -> 96%... -> 92%. Similarly, the second vertical correction amount Cyr1 for the vertical eye position change amount ΔPy may be gradually decreased over time as it becomes farther from the first vertical correction amount Cy for the vertical eye position change amount ΔPy.

[0082] This correction amount Cxq11, which changes over time, gradually becomes smaller over time as the object distance becomes longer, and conversely, gradually becomes larger over time as the object distance becomes shorter, but the rate of change may be different when the object distance becomes longer and when the object distance becomes shorter.

[0083] (Third image correction process S180) When it is determined in step S120 that the predetermined condition is satisfied (step S134), the eye tracking image processing module 512 in FIG. 10 corrects the left-right position of the virtual image V by a third left-right correction amount (third correction amount) Cxq2 corresponding to the left-right eye position change amount ΔPx. As shown in FIG. 13(a), the third left-right correction amount Cxq2 is a value obtained by multiplying the first left-right correction amount Cx for the left-right eye position change amount ΔPx in the first image correction process S160 by a second left-right coefficient (second predetermined coefficient) Q2 of 0.5 which is smaller than the first left-right coefficient Q1. Specifically, for example, if the first left-right correction amount Cx for the left-right eye position change amount ΔPx is set to 100%, then the same left and right The third left-right correction amount Cxq2 for the change amount ΔPx of the eye position in the direction is 50%. In a broad sense, the third left-right correction amount Cxq2 should be smaller than the first left-right correction amount Cx to suppress the first left-right correction amount Cx, so it is sufficient if it is less than 100% of the first left-right correction amount Cx, but it is preferably less than 50% of the first left-right correction amount Cx.

[0084] (Third image correction process S180) When it is determined in step S120 that the predetermined condition is satisfied (step S134), the eye tracking image processing module 512 in FIG. 10 corrects the vertical position of the virtual image V by a third vertical correction amount (third correction amount) Cyr2 according to the vertical eye position change amount ΔPy. As shown in FIG. 13(b), the third vertical correction amount Cyr2 is a value obtained by multiplying the first vertical correction amount Cy for the vertical eye position change amount ΔPy in the first image correction process S160 by a second vertical coefficient (second predetermined coefficient) R2 that is smaller than the first vertical coefficient R1 of 0.4. Specifically, for example, when the first vertical correction amount Cy for the vertical eye position change amount ΔPy is 100%, the third vertical correction amount Cyr2 for the same vertical eye position change amount ΔPy is 40%. In a broad sense, the third vertical correction amount Cyr2 should be smaller than the first vertical correction amount Cy to suppress the first left / right correction amount Cx, so it is sufficient that the third vertical correction amount Cyr2 is less than 100% of the first vertical correction amount Cy, but it is preferable that the third vertical correction amount Cyr2 is less than 40% of the first vertical correction amount Cy.

[0085] (An example of step S180) In step S180 in some embodiments, the display control device 30 (processor 33) may, after proceeding to the third image correction process as shown in Fig. 13(a), gradually reduce the third left-right correction amount Cxq2 for the left-right eye position change amount ΔPx over time so as to move away from the second left-right correction amount Cxq1 for the left-right eye position change amount ΔPx (see Cxq21). For example, if the second left-right correction amount Cxq1 for the left-right eye position change amount ΔPx is 90% and the third left-right correction amount Cxq2 is 50%, the display control device 30 (processor 33) may gradually reduce the third left-right correction amount Cxq2 over time so as to move away from the second left-right correction amount Cxq1, such as 90% → 80% → 70% → ... → 50%. Similarly, the third vertical correction amount Cyr2 for the vertical eye position change amount ΔPy may be gradually decreased over time so as to move away from the second vertical correction amount Cyr1 for the vertical eye position change amount ΔPy.

[0086] This correction amount Cxq21, which changes over time, gradually becomes smaller over time as the object distance becomes longer, and conversely, gradually becomes larger over time as the object distance becomes shorter, but the rate of change may be different when the object distance becomes longer and when the object distance becomes shorter.

[0087] (Step S190) In some embodiments, in step S160, if the display control device 30 (processor 33) determines that a predetermined cancellation condition is satisfied, the process proceeds from the second image correction process S170 to the first image correction process S160.

[0088] The predetermined release condition includes the lapse of a predetermined time (e.g., 20 seconds) after transition to the second image correction process S170. The eye tracking image processing module 512 may execute timing after transition to the second image correction process S170, and may determine that the release condition is satisfied when the predetermined time previously stored in the memory 37 (or set by the operation detection unit 407) has elapsed.

[0089] (Step S190) In some embodiments, in step S160, if the display control device 30 (processor 33) determines that a predetermined cancellation condition is satisfied, the process proceeds from the third image correction process S180 to the second image correction process S170.

[0090] The predetermined release condition includes the lapse of a predetermined time (e.g., 20 seconds) after transition to the third image correction process S180. The eye tracking image processing module 512 may execute timing after transition to the third image correction process S180, and may determine that the release condition is satisfied when the predetermined time previously stored in the memory 37 (or set by the operation detection unit 407) has elapsed.

[0091] The predetermined cancellation condition may include the predetermined condition no longer being satisfied in step S120. That is, the predetermined cancellation condition may include detecting that at least one of steps S131 to S134 and step S141 has transitioned from a state in which the predetermined condition is satisfied to a state in which the predetermined condition is no longer satisfied. The predetermined cancellation condition may include the elapse of a predetermined time (e.g., 20 seconds) after the predetermined condition no longer being satisfied in step S120.

[0092] Referring again to Fig. 10, the graphics module 514 in Fig. 10 includes various known software components for generating image data by performing image processing such as rendering, and driving the display 40. The graphics module 514 may also include various known software components for changing the type (moving image, still image, shape), arrangement (position coordinates, angle), size, display distance (in the case of 3D), and visual effects (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the image to be displayed. The graphics module 514 can generate image data to be viewed by an observer in terms of the type of image (one example of a display parameter), the position coordinates of the image (one example of a display parameter), the angle of the image (a pitching angle about the X-direction axis, a yaw rate angle about the Y-direction axis, a rolling angle about the Z-direction axis, etc., one example of a display parameter), the size of the image (one example of a display parameter), the color of the image (one example of a display parameter set by hue, saturation, brightness, etc.), and the intensity of the perspective representation of the image (one display parameter set by the position of the vanishing point, etc.), and drive the light modulation element 50.

[0093] The light source driving module 516 includes various known software components for executing driving of the light source unit 24. The light source driving module 516 may drive the light source unit 24 based on the set display parameters.

[0094] The actuator driving module 518 includes various known software components for executing the actuation of the first actuator 28 and / or the second actuator 29. The actuator driving module 518 may drive the first actuator 28 and the second actuator 29 based on set display parameters.

[0095] As described above, the display control device 30 of this embodiment is a display control device (30) that is configured to at least include a display (40) that displays an image, and a relay optical system (80) that projects the light of the image displayed by the display (40) onto a projection target member, and executes display control in a head-up display device (20) that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, and includes one or more processors (33), a memory (37), one or more computer programs that are stored in the memory (37) and configured to be executed by the one or more processors (33), and an outside-vehicle sensor (411) that detects real objects present in the periphery of the vehicle, and One or more The processor (33) determines the user's vertical eye position of the vehicle. Okimata is the head position (Py) and the eye position in the left and right direction of the vehicle Okimata Acquires the head position (Px), and the eye position or head position (Py) in the vertical direction Change in (ΔPy) And the change in eye position or head position (Px) in the left-right direction ( ΔPx and a second image correction process (S170) for correcting the position of the image to be displayed on the display device (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or smaller than 1. Superimposed on the virtual image Distance to the real object in the foreground The object distance is (D4) executes the first image correction process if the first distance threshold is less than or equal to the first distance threshold; The object distance If (D4) is greater than the first distance threshold, the second image correction process (S170) is executed. As a result, even if the observer's eye position is detected (misdetected) as having moved an amount greater than a predetermined amount, the position correction of the image is suppressed, thereby reducing the sense of discomfort felt by the observer.

[0096] In some embodiments, the first distance threshold may be a distance (D10) from the vehicle to an area in which the virtual image is displayed.

[0097] In some embodiments, a third image correction process (S180) is provided to correct a position of the image to be displayed on the display (40) by a third correction amount (Cxq2, Cyr2) obtained by multiplying the first correction amount (Cx, Cy) by a second predetermined coefficient (Q2, R2), The object distance If (D4) is greater than the second distance threshold, the third image correction process (S180) may be executed.

[0098] Also, in some embodiments, the second distance threshold may be greater than the first distance threshold.

[0099] In some embodiments, the second predetermined coefficient (Q2, R2) may be smaller than the first predetermined coefficient (Q1, R1).

[0100] In some embodiments, the change in eye position or head position (Px) in the left-right direction ( ΔPx ) may be greater than the first predetermined coefficient (R1) when correcting the position of the image to be displayed on the display (40) based on a change (ΔPy) in the eye position or head position (Py) in the vertical direction.

[0101] In some embodiments, the first predetermined coefficient (Q1 ,R1) may change over time as the second correction amount (Cxq1, Cyr1) in the second image correction process becomes more distant from the first correction amount (Cx, Cy) in the first image correction process.

[0102] In some embodiments, the head-up display device 20 is at least composed of a display (40) that displays an image, and a relay optical system (80) that projects light of the image displayed by the display (40) onto a projection target member, and is a display control device (30) that executes display control in the head-up display device (20) that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, the display control device (30) comprising one or more processors (33), a memory (37), one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33), and an outside sensor (411) that detects real objects present in the periphery of the vehicle, One or more The processor (33) determines the user's vertical eye position of the vehicle. Okimata is the head position (Py) and the eye position in the left and right direction of the vehicle Okimata Acquires the head position (Px), and the eye position or head position (Py) in the vertical direction Change in (ΔPy) And the change in eye position or head position (Px) in the left-right direction ( ΔPx and a second image correction process (S170) for correcting the position of the image to be displayed on the display device (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or smaller than 1. Superimposed on a virtual image Distance to the real object in the foreground The object distance is (D4) executes the first image correction process if the first distance threshold is less than or equal to the first distance threshold; The object distance If (D4) is greater than the first distance threshold, the second image correction process (S170) is executed.

[0103] The operations of the above processing processes can be implemented by executing one or more functional modules of an information processing device, such as a general-purpose processor or an application-specific chip. These modules, combinations of these modules, and / or combinations with known hardware that can replace their functions, are all included in the scope of protection of the present invention.

[0104] The functional blocks of the vehicle display system 10 are optionally implemented by hardware, software, or a combination of hardware and software to implement the principles of the various embodiments described. It will be understood by those skilled in the art that the functional blocks described in FIG. 10 may be optionally combined or one functional block may be separated into two or more sub-blocks to implement the principles of the embodiments described. Therefore, the description in this specification optionally supports any possible combination or division of the functional blocks described in this specification. [Explanation of symbols]

[0105] 1: Vehicle 2:Projected area 5: Dashboard 6: Road surface 10: Vehicle display system 20:HUD device 21: Light exit window 22: Housing 24: Light source unit 28: First actuator 29: Second actuator 30: Display control device 31: I / O interface 33: Processor 35: Image processing circuit 37: Memory 40:Display device 205: Center 300: Real object 311: Road surface 313: person 314: Other vehicles 315: Building 401: Vehicle ECU 403: Road information database 405: Vehicle position detection unit 407: Operation detection unit 409: Eye position detection unit 411: External sensor 413: Brightness detection unit 417: Mobile information terminal 419: External communication equipment 502: Eye position detection module 504: Eye position estimation module 506: Eye position prediction module 508: Eye position state determination module 510: Vehicle state determination module 511: Distance determination module 512: Eye-tracking image processing module 514: Graphics module 516: Light source driving module 518: Actuator drive module 710: Head position Cx: First left / right correction amount (first correction amount) Cxq1: Second left / right correction amount (second correction amount) Cxq2: Third left / right correction amount (third correction amount) Cy: First vertical correction amount (first correction amount) Cyr1: Second vertical correction amount (second correction amount) Cyr2: Third up / down correction amount (third correction amount) Q1: First left-right coefficient (first specified coefficient) Q2: Second left-right coefficient (second specified coefficient) R1: First up / down coefficient (first predetermined coefficient) R2: Second up / down coefficient (second specified coefficient) D10: Imaging distance (first distance threshold) D30: Perceived Distance D4: Object distance (distance to real object) FU: Virtual object (perceptual image) K:Display light PS:Virtual image display area PT: Target position Px: Eye position (head position) in the left / right direction Py: Vertical eye position (head position) V: Virtual image V60: AR virtual image (upper virtual image) V61: Navigation Virtual Image V62: Navigation Virtual Image V63: Enhanced virtual image V64: Enhanced virtual image V65: Enhanced virtual image V65: POI virtual image V70: Non-AR virtual image (lower virtual image) VS: Virtual image display area Vx: Movement speed Vy: Movement speed t :period time ΔPx: Amount of change ΔPy: Amount of change

Claims

1. A display control device (30) that executes display control in a head-up display device (20) that is configured at least with a display device (40) that displays an image and a relay optical system (80) that projects light of the image displayed by the display device (40) onto a projection target member, and that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, one or more processors (33); A memory (37); one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33); an outside sensor (411) for detecting real objects present around the vehicle; a first image correction process (S160) in which the one or more processors (33) acquire the user's eye position or head position (Py) in a vertical direction of the vehicle and the user's eye position or head position (Px) in a horizontal direction of the vehicle, and correct the position of the image to be displayed on the display (40) with a first correction amount (Cx, Cy) based at least on a change (ΔPy) in the eye position or head position (Py) in the vertical direction and a change (ΔPx) in the eye position or head position (Px) in the horizontal direction; a second image correction process (S170) for correcting a position of the image to be displayed on the display (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or smaller than 1; When an object distance (D4) which is a distance from the vehicle to the real object in the foreground which is superimposed on the virtual image is equal to or less than a first distance threshold, the first image correction process is executed; When the object distance (D4) is greater than the first distance threshold, the second image correction process (S170) is executed. A display control device (30).

2. The first distance threshold is a distance (D10) from the vehicle to an area where the virtual image is displayed. A display control device (30) according to claim 1.

3. a third image correction process (S180) for correcting a position of the image to be displayed on the display (40) with a third correction amount (Cxq2, Cyr2) obtained by multiplying the first correction amount (Cx, Cy) by a second predetermined coefficient (Q2, R2); When the object distance (D4) is greater than a second distance threshold, the third image correction process (S180) is executed. A display control device (30) according to claim 1.

4. The second distance threshold is greater than the first distance threshold. A display control device (30) according to claim 3.

5. The second predetermined coefficient (Q2, R2) is smaller than the first predetermined coefficient (Q1, R1); A display control device (30) according to claim 3.

6. the first predetermined coefficient (Q1) when correcting the position of the image to be displayed on the display (40) based on the change (ΔPx) in the eye position or head position (Px) in the left-right direction is greater than the first predetermined coefficient (R1) when correcting the position of the image to be displayed on the display (40) based on the change (ΔPy) in the eye position or head position (Py) in the up-down direction; A display control device (30) according to claim 1.

7. The first predetermined coefficient (Q1, R1) is, in the second image correction process, the second correction amount (Cxq1, Cyr1) changes over time as it becomes farther from the first correction amount (Cx, Cy) in the first image correction process; A display control device (30) according to claim 1.

8. A display control device (30) that executes display control in a head-up display device (20) that is configured at least with a display device (40) that displays an image and a relay optical system (80) that projects light of the image displayed by the display device (40) onto a projection target member, and that allows a vehicle user to visually recognize a virtual image of the image superimposed on a foreground, one or more processors (33); A memory (37); one or more computer programs stored in said memory (37) and configured to be executed by said one or more processors (33); an outside sensor (411) for detecting real objects present around the vehicle; a first image correction process (S160) in which the one or more processors (33) acquire the user's eye position or head position (Py) in a vertical direction of the vehicle and the user's eye position or head position (Px) in a horizontal direction of the vehicle, and correct the position of the image to be displayed on the display (40) with a first correction amount (Cx, Cy) based at least on a change (ΔPy) in the eye position or head position (Py) in the vertical direction and a change (ΔPx) in the eye position or head position (Px) in the horizontal direction; a second image correction process (S170) for correcting a position of the image to be displayed on the display (40) with a second correction amount (Cxq1, Cyr1) obtained by multiplying the first correction amount (Cx, Cy) by a first predetermined coefficient (Q1, R1) that is equal to or smaller than 1; When an object distance (D4) which is a distance from the vehicle to the real object in the foreground which is superimposed on the virtual image is equal to or less than a first distance threshold, the first image correction process is executed; When the object distance (D4) is greater than the first distance threshold, the second image correction process (S170) is executed. A display control device (30) that executes display control in a head-up display device (20).

Images