Display control device, head-up display device, and display control method

Abstract

The present invention suppresses positional correction of an image which occurs along with detection of a shift in vertical eye position unintended by the user. First image correction processing S160 corrects the position of an image displayed on a display device 40 on the basis of a vertical eye position and a lateral eye position. Second image correction processing S170 corrects the position of the image displayed on the display device 40 on the basis of the vertical eye position and the lateral eye position, wherein a second correction amount Cy2 to the position of the image with respect to the amount of change in the vertical eye position is smaller than a first correction amount Cy1 to the position of the image with respect to the amount of change in the vertical eye position during the first image correction processing S160. A processor switches between the first image correction processing S160 and the second image correction processing S170 in accordance with whether or not a prescribed condition is satisfied.

Description

Technical Field

[0001] The present disclosure relates to a display control device, a head-up display device, a display control method, etc., which are used in a moving body such as a vehicle and visually recognize an image by superimposing it on the foreground of the moving body (the actual scene in the forward direction of the moving body as seen by the vehicle occupant).

[0002] In Patent Document 1, display light projected onto a projection target such as the front windshield of a vehicle is reflected toward a vehicle occupant (observer) inside the vehicle, allowing the observer to visually recognize a virtual image that overlaps with the foreground of the vehicle. In particular, the head-up display device described in Patent Document 1 virtually displays a virtual object (virtual image) at a depth in the foreground real space and at a predetermined position in the vertical, horizontal, and left-right directions (here, this position will be referred to as the target position). Even when there is a change in the vehicle's posture or a change in the observer's eye position, the head-up display device controls the image displayed inside the device as if a display object exists at the target position in the foreground. That is, such a head-up display device forms an augmented reality in which a virtual object is added to the real scenery (foreground) and displayed. Even when there is a change in the vehicle's posture (which also leads to a change in the observer's eye position relative to the real scene) or a change in the observer's eye position inside the vehicle, the position of the image displayed inside the head-up display device is corrected according to the change in the observer's eye position detected by an eye position detection unit such as a camera, giving the virtual object motion parallax and making the virtual object appear to the observer as if it were at the target position in the foreground (real scene).

[0003] Furthermore, Patent Document 2 discloses a head-up display device that tracks the position of the observer's right and left eyes detected by an eye position detection unit such as a camera, and controls the display to direct a right-eye display light showing a right-eye image to the tracked right-eye position and a left-eye display light showing a left-eye image to the tracked left-eye position, thereby giving binocular parallax to a virtual object and allowing the observer to perceive the virtual object as if it were actually located at the target position in the foreground (real scene).

[0004] Furthermore, Patent Document 3 discloses a head-up display device that emphasizes the position of real objects in a real scene by aligning the display position of an image (virtual image) with a position on a straight line when viewing a specific position on a real object in the foreground (or a position around a real object that has a specific positional relationship with the real object) from the observer's eye position detected by an eye position detection unit such as a camera. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2010-156608 [Patent Document 2] Japanese Patent Publication No. 2019-062532 [Patent Document 3] International Publication No. 2019 / 097918 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] Incidentally, when an observer moves their head from side to side or moves their neck from side to side, they may think they are moving their eyes horizontally, but in reality, they may also be moving them vertically. The eye position detection unit can detect vertical eye movement, and the display position of the image (virtual image) may be corrected vertically as well. In such cases, the image position correction due to the detection of vertical eye movement may cause discomfort to an observer who is conscious of moving their eyes horizontally.

[0007] Furthermore, while the eye position detection unit of a camera or similar device detects the observer's eye position (left and right eye position) using a complex algorithm based on the captured image, even if the observer's eye position is only moving left and right, depending on the way the head is moved left and right, the way the neck is moved from side to side, and / or the detection environment, detection errors or false detections may cause the eye position detection unit to also detect vertical eye movement, potentially correcting the display position of the image (virtual image) vertically. This correction of the image position due to the detection of vertical eye movement may cause discomfort to an observer who is only aware of moving their eye position horizontally.

[0008] A summary of specific embodiments disclosed herein is provided below. It should be understood that these embodiments are presented solely to provide the reader with an overview of these specific embodiments and do not limit the scope of this disclosure. In fact, this disclosure may encompass various embodiments not described below.

[0009] The outline of this disclosure relates to reducing the likelihood of causing discomfort to the observer. More specifically, it relates to providing a display control device, a head-up display device, and a display control method, etc., that reduce the discomfort caused to the user by suppressing image position correction that occurs when unintended vertical eye movement is detected.

[0010] Therefore, the display control device, head-up display device, and display control method described herein employ the following means to solve the above problem. This embodiment corrects the position of the image displayed on the display based at least on the vertical eye position or head position and the horizontal eye position or head position. Furthermore, the amount of correction for the image position in response to changes in the vertical eye position or head position is defined as the first correction amount (Cy1), and the amount of correction for the image position in response to changes in the horizontal eye position or head position is defined as the first horizontal correction amount (Cx1). First image correction processing Reason and, Eye position or head position in the left-right direction The amount of correction for the change is the second left-right correction amount (Cx2) Correct the position of the image displayed on the display unit. Tsutsu , the second correction amount (Cy2) for the image position with respect to the change in eye position or head position in the vertical direction is smaller than the first correction amount (Cy1) for the image position with respect to the change in eye position or head position in the vertical direction during the first image correction process, or the eye position or head position in the horizontal direction The second left-right correction amount (Cx2) is applied to the amount of change of The position of the image displayed on the display unit (40) is corrected. Tsutsu a second image correction process which sets the amount of image position correction for changes in vertical eye position or head position to zero, The system is equipped with the following features: In the second image correction process, the ratio of the second correction amount (Cy2) to the first correction amount (Cy1) is set to be smaller than the ratio of the second left-right direction correction amount (Cx2) to the first left-right direction correction amount (Cx1), such that (Cx2 / Cx1 > Cy2 / Cy1); and when at least one predetermined condition is met by the eye position, head position, or vehicle state, the system switches from the first image correction process to the second image correction process and executes accordingly. This is the gist of it.

[0011] Accordingly, the display control device described herein is a display control device that performs display control in a head-up display device that causes a vehicle user to view a virtual image of the image superimposed on the foreground, comprising at least a display unit that displays an image, and a relay optical system that projects the light of the image displayed by the display unit onto a projection member, and comprises one or more processors, a memory, and one or more computer programs stored in the memory and configured to be executed by one or more processors, wherein the processors acquire the user's vertical eye position and / or head position of the vehicle and the horizontal eye position and / or head position of the vehicle, and correct the position of the image to be displayed on the display unit based at least on the vertical eye position or head position and the horizontal eye position or head position. Furthermore, the amount of correction for the image position in response to changes in the vertical eye position or head position is defined as the first correction amount (Cy1), and the amount of correction for the image position in response to changes in the horizontal eye position or head position is defined as the first horizontal correction amount (Cx1). First image correction processing Reason and, Eye position or head position in the left-right direction The amount of correction for the change is the second left-right correction amount (Cx2) Correct the position of the image displayed on the display unit. Tsutsu, the second correction amount (Cy2) for the image position with respect to the change in eye position or head position in the vertical direction is smaller than the first correction amount (Cy1) for the image position with respect to the change in eye position or head position in the vertical direction during the first image correction process, or the eye position or head position in the horizontal direction The second left-right correction amount (Cx2) is applied to the amount of change of Correct the position of the image displayed on the display unit. Tsutsu a second image correction process which sets the amount of image position correction for changes in vertical eye position or head position to zero, The system is equipped with the following features: In the second image correction process, the ratio of the second correction amount (Cy2) to the first correction amount (Cy1) is set to be smaller than the ratio of the second left-right direction correction amount (Cx2) to the first left-right direction correction amount (Cx1), such that (Cx2 / Cx1 > Cy2 / Cy1); and when at least one predetermined condition is met by the eye position, head position, or vehicle state, the system switches from the first image correction process to the second image correction process and executes accordingly. It is a display control device. [Brief explanation of the drawing]

[0012] [Figure 1] Figure 1 shows an example of applying a virtual image display system for vehicles to a vehicle. [Figure 2] Figure 2 shows the configuration of a head-up display device. [Figure 3] Figure 3 shows an example of the foreground as seen by an observer while the vehicle is in motion, and a virtual image superimposed on the foreground. [Figure 4] Figure 4 is a conceptual diagram showing the positional relationship between the left-view virtual image and the right-view virtual image displayed on the virtual image forming surface, and the perceptual image perceived by the observer based on these left-view virtual image and right-view virtual image, in an embodiment where the HUD device is a 3D-HUD device. [Figure 5] Figure 5 is a conceptual diagram illustrating a virtual object placed at a target location in the real-world scene, and an image displayed in a virtual image display area so that the virtual object is visible at the target location in the real-world scene. [Figure 6] Figure 6 is a diagram illustrating the method of adding motion parallax in this embodiment. [Figure 7A] Figure 7A is a comparative example showing a virtual image viewed from position Px12 shown in Figure 6, when the motion parallax processing of this embodiment is not performed. [Figure 7B] Figure 7B shows the virtual image viewed from position Px12 shown in Figure 6 when the motion parallax processing of this embodiment is performed. [Figure 8]FIG. 8 is a diagram for explaining a method of adding motion parallax by moving the vertical eye position (head position) in the present embodiment. [Figure 9] FIG. 9 is a diagram showing an example of a foreground visually recognized by an observer and a virtual image displayed superimposed on the foreground while the host vehicle is traveling. [Figure 10] FIG. 10 is a block diagram of a vehicle virtual image display system according to some embodiments. [Figure 11A] FIG. 11A is a flowchart showing a method S100 of performing an operation of correcting an image based on the eye position of an observer. [Figure 11B] FIG. 11B is a diagram for explaining a part of the flowchart shown in FIG. 11A. [Figure 11C] FIG. 11C is a diagram for explaining a part of the flowchart shown in FIG. 11A. [Figure 12] FIG. 12 is an image diagram showing the eye position, the amount of change in the eye position, the moving speed of the eye position, etc. detected every predetermined cycle time. [Figure 13] FIG. 13 is a diagram for explaining a HUD device in some embodiments in which an eyebox can be moved in the vertical direction by rotating a relay optical system.

BEST MODE FOR CARRYING OUT THE INVENTION

[0013] Hereinafter, FIGS. 1 to 6 and FIGS. 7B to 12 provide an explanation of the configuration and operation of an exemplary vehicle display system. Note that the present invention is not limited to the following embodiments (including the contents of the drawings). Of course, changes (including deletion of components) can be made to the following embodiments. In the following description, for the sake of easy understanding of the present invention, descriptions of known technical matters are appropriately omitted. <00001​​Refer to Figure 1. Figure 1 shows an example of the configuration of a vehicle virtual image display system including a parallax 3D-HUD device. In Figure 1, the left-right direction of the vehicle (an example of a moving object) 1 (in other words, the width direction of the vehicle 1) is defined as the X-axis (the positive direction of the X-axis is to the left when the vehicle 1 is facing forward), the up-down direction (in other words, the height direction of the vehicle 1) along a line segment perpendicular to the left-right direction and perpendicular to the ground or a surface equivalent to the ground (in this case, the road surface 6) is defined as the Y-axis (the positive direction of the Y-axis is upward), and the front-back direction along a line segment perpendicular to each of the left-right and up-down directions is defined as the Z-axis (the positive direction of the Z-axis is the straight-ahead direction of the vehicle 1). This is the same in other drawings as well.

[0015] As shown in the figure, the vehicle display system 10 installed in the vehicle (own vehicle) 1 consists of an eye position detection unit 409 for pupil (or face) detection that detects the position and gaze direction of the left eye 700L and right eye 700R of the observer (typically the driver seated in the driver's seat of vehicle 1), an external sensor 411 consisting of a camera (e.g., a stereo camera) that captures images of the area in front of (or broadly speaking, the surroundings of) vehicle 1, a head-up display device (hereinafter also referred to as a HUD device) 20, and a display control device 30 that controls the HUD device 20.

[0016] Figure 2 shows one configuration of a head-up display device. The HUD device 20 is installed, for example, in the dashboard (reference numeral 5 in Figure 1). The HUD device 20 comprises a stereoscopic display device (an example of a display) 40, a relay optical system 80, and a housing 22 that houses the stereoscopic display device 40 and the relay optical system 80, and has a light emission window 21 that allows display light K from the stereoscopic display device 40 to be emitted from the inside to the outside.

[0017] The stereoscopic display device 40 is, in this case, a parallax-type 3D display device. This stereoscopic display device (parallax-type 3D display device) 40 is a naked-eye stereoscopic display device that uses a multi-view image display method capable of controlling depth representation by allowing the viewer to see a left-view image and a right-view image, and consists of a light modulation element 50 and a light source unit 60 that functions as a backlight.

[0018] The optical modulation element 50 includes an optical modulation element 51 that modulates illumination light from the light source unit 60 to generate an image, and an optical layer (an example of a light ray separation unit) 52 that separates the light emitted from the optical modulation element 51 into left-eye display light (reference numeral K10 in Figure 1), such as left-eye rays K11, K12, and K13, and right-eye display light (reference numeral K20 in Figure 1), such as right-eye rays K21, K22, and K23. The optical layer 52 includes optical filters such as lenticular lenses, parallax barriers, lens arrays, and microlens arrays. However, this is just an example and is not limited to this. Embodiments of the optical layer 52 are not limited to the optical filter described above, but include all forms of optical layers arranged on the front or rear surface of the optical modulation element 51, as long as they generate left-eye indicator light (reference numeral K10 in Figure 1) and right-eye indicator light (reference numeral K20 in Figure 1) from the light emitted from the optical modulation element 51. Some embodiments of the optical layer 52 may generate left-eye indicator light (reference numeral K10 in Figure 1) and right-eye indicator light (reference numeral K20 in Figure 1) from the light emitted from the optical modulation element 51 by being electrically controlled, such as a liquid crystal lens. In other words, embodiments of the optical layer 52 may include those that are electrically controlled and those that are not electrically controlled.

[0019] Furthermore, the stereoscopic display device 40 may emit left-eye display light (reference numeral K10 in Figure 1), such as left-eye rays K11, K12, and K13, and right-eye display light (reference numeral K20 in Figure 1), such as right-eye rays K21, K22, and K23, by configuring the light source unit 60 with a directional backlight unit (reference numeral K10 in Figure 1) instead of or in addition to the optical layer (example of a light ray separation unit) 52. Specifically, for example, the display control device 30, described later, directs the left-eye display light K10, such as the left-eye rays K11, K12, and K13, towards the observer's left eye 700L when the directional backlight unit emits illumination light directed towards the left eye 700R, by displaying a left-viewpoint image on the light modulation element 51, thereby directing the right-eye display light K20, such as the right-eye rays K21, K22, and K23, towards the observer's left eye 700R. However, this is just one example and is not limited to this.

[0020] The display control device 30, described later, can control the appearance of the perceptual virtual image FU displayed by the HUD device 20 (as perceived by the observer) by performing, for example, image rendering processing (graphics processing) and display drive processing, directing the left-eye display light K10 of the left-viewpoint image V10 to the observer's left eye 700L and the right-eye display light K20 of the right-viewpoint image V20 to the right eye 700R, and adjusting the left-viewpoint image V10 and the right-viewpoint image V20. The display control device 30, described later, may also control the display (light modulation element 50) to generate a light field that reproduces (approximately) the light rays emitted in various directions from a point existing in a certain space.

[0021] The relay optical system 80 has curved mirrors (concave mirrors, etc.) 81 and 82 that reflect light from the stereoscopic display device 40 and project the image display light K10 and K20 onto the windshield (projection target) 2. However, it may further include other optical elements (refractive optical elements such as lenses, diffractive optical elements such as holograms, reflective optical elements, or combinations thereof).

[0022] In Figure 1, the stereoscopic display device 40 of the HUD device 20 displays parallax images for each of the left and right eyes. Each parallax image is displayed as V10 and V20, which are imaged onto the virtual image display surface (virtual image forming surface) VS, as shown in Figure 1. The focus of each of the observer's (person's) eyes is adjusted to match the position of the virtual image display area VS. The position of the virtual image display area VS is referred to as the "adjustment position (or image forming position)," and the distance from a predetermined reference position (for example, the center 205 of the eye box 200 of the HUD device 20, the observer's viewpoint position, or a specific position on the vehicle 1) to the virtual image display area VS (see symbol D10 in Figure 4) is referred to as the adjustment distance (image forming distance).

[0023] However, in reality, because the human brain fuses each image (virtual image), people perceive the perceived image (in this case, the arrowhead shape for navigation) FU as being displayed at a position further back than the accommodation position (for example, a position determined by the convergence angle between the left viewpoint image V10 and the right viewpoint image V20, where the smaller the convergence angle, the further away the image is perceived to be from the observer). The perceived virtual image FU is sometimes called a "stereoscopic virtual image," and if "image" is taken in a broad sense to include virtual images, it can also be called a "stereoscopic image." It may also be called a "stereoscopic image," "3D display," etc. The HUD device 20 can display the left viewpoint image V10 and the right viewpoint image V20 so that the perceived image FU is visible at a position closer to the user than the accommodation position.

[0024] Next, refer to Figures 3 and 4. Figure 3 is a diagram showing an example of the foreground seen by the observer while the vehicle 1 is in motion, and a perceptual image superimposed on the foreground. Figure 4 is a diagram conceptually showing the positional relationship between the left-view virtual image and the right-view virtual image displayed on the virtual image forming surface, and the perceptual image perceived by the observer by these left-view virtual image and right-view virtual image.

[0025] In Figure 3, vehicle 1 is traveling on a straight road (road surface) 6. The HUD device 20 is installed in the dashboard 5. Display light K (K10, K20) is projected from the light emission window 21 of the HUD device 20 onto the projection area (the front windshield of vehicle 1) 2. In the example in Figure 3, a first content image FU1 is superimposed on the road surface 6 and indicates the path of vehicle 1 (in this case, going straight), and a second content image FU2 is displayed, which also indicates the path of vehicle 1 (in this case, going straight) and is perceived as being further away from the first content image FU1.

[0026] As shown in the left diagram of Figure 4, the HUD device 20 (1) emits a left-eye display light K10 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the left eye 700L detected by the eye position detection unit 409, and forms a first left-viewpoint content image V11 at a predetermined position in the virtual image display area VS as seen from the left eye 700L, and (2) emits a right-eye display light K20 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the right eye 700R, and forms a first right-viewpoint content image V21 at a predetermined position in the virtual image display area VS as seen from the right eye 700R. The first content image FU1 perceived by the first left-viewpoint content image V11 and the first right-viewpoint content image V21, which have parallax, is visible at a position that is a distance D21 behind the virtual image display area VS (a position that is a distance D31 from the above reference position).

[0027] Similarly, as shown in the right diagram of Figure 4, the HUD device 20 (1) emits a left-eye display light K10 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the left eye 700L detected by the eye position detection unit 409, and forms a second left-viewpoint content image V12 at a predetermined position in the virtual image display area VS as seen from the left eye 700L, and (2) emits a right-eye display light K20 to the projection unit 2 at a position and angle such that it is reflected by the projection unit 2 to the right eye 700R, and forms a second right-viewpoint content image V22 at a predetermined position in the virtual image display area VS as seen from the right eye 700R. The second content image FU2 perceived by the second left-viewpoint content image V12 and the second right-viewpoint content image V22, which have parallax, is visible at a position that is a distance D22 behind the virtual image display area VS (a position that is a distance D31 from the above reference position).

[0028] Specifically, the distance from the above reference position to the virtual image display area VS (imaging distance D10) is set to, for example, "4m", the distance from the above reference position to the first content image FU1 shown in the left diagram of Figure 4 (first perceptual distance D31) is set to, for example, "7m", and the distance from the above reference position to the second content image FU2 shown in the right diagram of Figure 4 (second perceptual distance D32) is set to, for example, "10m". However, this is just an example and is not limited to this.

[0029] Figure 5 is a conceptual diagram showing a virtual object placed at a target position in the real-world scene, and an image displayed in the virtual image display area so that the virtual object is visible at the target position in the real-world scene. Note that the HUD device 20 shown in Figure 5 is an example of 2D display, not 3D display. That is, the display unit 40 of the HUD device 20 shown in Figure 5 is a 2D display device, not a stereoscopic display device (however, 2D display is also possible with stereoscopic display devices). As shown in Figure 5, from the perspective of the viewer 700, the depth direction is the Z-axis direction, the left-right direction (width direction of the vehicle 1) is the X-axis direction, and the up-down direction (up-down direction of the vehicle 1) is the Y-axis direction. Note that the direction away from the viewer is the positive Z-axis direction, the left direction from the viewer is the positive X-axis direction, and the up direction from the viewer is the positive Y-axis direction.

[0030] The observer 700 perceives a virtual object FU at a predetermined target position PT in the real scene by viewing the virtual image V formed (imaged) in the virtual image display area VS via the projection unit 2. The observer views the virtual image V of the display light K reflected by the projection unit 2. At this time, if the virtual image V is, for example, an arrow indicating a path, the arrow of the virtual image V is displayed in the virtual image display area VS so that the virtual object FU is positioned and visible at a predetermined target position PT in the foreground of the vehicle 1. Specifically, the HUD device 20 (display control device 30) renders an image to be displayed on the display unit 40 so that a virtual image V of a predetermined size and shape is displayed, which is obtained by projecting a virtual object FU of a predetermined size and shape, placed at the target position PT, onto the virtual image display area VS, using the center between the observer's left eye 700L and right eye 700R as the origin of the projection transformation. Furthermore, the HUD device 20 (display control device 30) changes the position of the virtual image V displayed in the virtual image display area VS so that even when the observer moves their eye position, the virtual object FU is perceived as being at the same target position PT as before the eye position moved. This allows the virtual object FU (virtual image V) to be perceived as being at the target position PT, even though it is displayed at a position far from the target position PT (in the virtual image display area VS). In other words, the HUD device 20 (display control device 30) expresses natural motion parallax by changing the position of the image on the display unit 40 (virtual image V in the virtual image display area VS) based on the movement of the eye position (this may also involve changes in size or shape). (In other words, the HUD device 20 makes it easier to perceive depth by adding motion parallax to the virtual image (image) through image correction accompanying the movement of the eye position.) In this embodiment, such correction of the image position that expresses motion parallax in response to changes in eye position is called motion parallax addition processing (an example of eye-tracking image correction processing). The motion parallax addition process is not limited to correcting the image position to perfectly reproduce natural motion parallax, but may also include correcting the image position to approximate natural motion parallax. The HUD device 20 (display control device 30) performs motion parallax addition processing (an example of eye-tracking image correction processing) in response to changes in the eye position 700.In addition to the above, motion parallax processing (an example of eye-tracking image correction processing) may be performed based on the observer's head position 710 instead of the eye position 700.

[0031] Figure 6 is a diagram illustrating the method of motion parallax addition processing in this embodiment. The display control device 30 (processor 33) in this embodiment controls the HUD device 20 and displays the virtual images V41, V42, and V43 formed (imaged) in the virtual image display area VS via the projection unit 2. Virtual image V41 is set to target position PT11 at a perceptual distance D33 (a position that is a distance D23 further back than the virtual image display area VS), virtual image V42 is set to target position PT12 at a perceptual distance D34 which is longer than the perceptual distance D33 of virtual image V41 (a position that is a distance D24 (>D23) further back than the virtual image display area VS), and virtual image V43 is set to target position PT13 at a perceptual distance D35 which is longer than the perceptual distance D34 of virtual image V42 (a position that is a distance D25 (>D24) further back than the virtual image display area VS). Furthermore, since the image correction amount in the display unit 40 corresponds to the virtual image correction amount in the virtual image display area VS, in Figure 6, the virtual image correction amounts corresponding to the image correction amounts C1, C2, and C3 in the display unit 40 also use the same symbols C1, C2, and C3 (the same applies to the symbols Cy11(Cy) and Cy21(Cy) in Figure 8).

[0032] When the observer's head position 710 (eye position 700) moves ΔPx10 to the right (negative X-axis direction) from the position of symbol Px11, the display control device 30 (processor 33) performs motion parallax addition processing to correct the positions where the virtual images V41, V42, and V43 displayed in the virtual image display area VS are shown, in the same direction as the movement of the observer's head position 710 (eye position 700), by correction amounts C1, C2 (>C1) and C3 (>C2), respectively. Figure 7A is a comparative example showing the virtual images V41, V42, and V43 as viewed from position Px12 shown in Figure 6 when the motion parallax addition processing of this embodiment is not performed, and Figure 7B shows the virtual images V44, V45, and V46 as viewed from position Px12 shown in Figure 6 when the motion parallax addition processing of this embodiment is performed. Note that in Figure 7B, the differences in the positions of the virtual images V44, V45, and V46 are exaggerated to make the differences in correction amounts easier to understand. In other words, the display control device 30 (processor 33) makes the correction amount for the positions of the multiple virtual images V41, V42, and V43 that occur with the movement of the eye position differ depending on the differences in the perceptual distances D33, D34, and D35 of the multiple virtual images V41, V42, and V43, thereby allowing the observer to perceive motion parallax even if only between the multiple virtual images V41 (V44), V42 (V45), and V43 (V46). More specifically, the display control device 30 (processor 33) adds motion parallax to the multiple virtual images V41 (V44), V42 (V45), and V43 (V46) by increasing the correction amount in the motion parallax addition process as the set perceptual distance D30 increases.

[0033] Figure 8 is a diagram illustrating the method of adding motion parallax when the eye position (head position) moves vertically in this embodiment. When the observer's head position 710 (eye position 700) moves upward (in the positive Y-axis direction) from the position of symbol Py12, the display control device 30 (processor 33) executes motion parallax addition processing to correct the position where the virtual image V displayed in the virtual image display area VS is displayed by a correction amount Cy11 in the same direction (upward (positive Y-axis direction)) as the observer's head position 710 (eye position 700) moved, as shown in Figure 8(a) (changing the position of the virtual image V from the position of symbol V48 to symbol V47). Furthermore, if the observer's head position 710 (eye position 700) moves downward (in the negative Y-axis direction) from the position of code Py12, the display control device 30 (processor 33) performs motion parallax processing to correct the position where the virtual image V displayed in the virtual image display area VS is displayed by a correction amount Cy21 in the same direction (upward (negative Y-axis direction)) as the observer's head position 710 (eye position 700) moved, as shown in Figure 8(c) (changing the position of the virtual image V from the position of code V48 to code V49). As a result, even though it is displayed at a position far from the target position PT (in the virtual image display area VS), it can be perceived as if the virtual object FU (virtual image V) is at the target position PT (the feeling that the virtual object FU (virtual image V) is at the target position PT can be enhanced).

[0034] Figure 9 shows a real object 300 in the foreground, as seen by an observer looking forward from the driver's seat of vehicle 1, and a virtual image V displayed by the HUD device 20 of this embodiment. The virtual image V shown in Figure 9 includes an AR virtual image V60 whose displayed position can be changed according to the position of the real object 300, and a non-AR virtual image V70 whose displayed position is set regardless of the position of the real object 300. The AR virtual image V60 is displayed at a position (target position PT) corresponding to the position of the real object 300 in the real scene. The AR virtual image V60 is displayed, for example, at a position superimposed on the real object 300, or in the vicinity of the real object 300, to emphasize and notify the presence of the real object 300. In other words, the "position corresponding to the position of the real object 300 (target position PT)" is not limited to a position that is superimposed on the real object 300 as seen by the observer, but may also be a position in the vicinity of the real object 300. Furthermore, the AR virtual image V60 is optional, as long as it does not obstruct the visibility of the real object 300.

[0035] The AR virtual images V60 shown in Figure 9 include navigation virtual images V61 and V62 that indicate the guidance route, enhancement virtual images V64 and V65 that highlight and notify the object of attention, and POI virtual image V65 that indicates a target object or a designated building. The position corresponding to the position of the real object 300 (target position PT) is the position of the road surface 311 (an example of a real object 300) on which the navigation virtual images V61 and V62 are superimposed, the position around the person 313 (an example of a real object 300) in the enhancement virtual image V63, the position near another vehicle 314 (an example of a real object 300) in the enhancement virtual image V64, and the position around the building 315 (an example of a real object 300) in the POI virtual image V65. As described above, the display control device 30 (processor 33) increases the correction amount C associated with the movement of the observer's eye position in the motion parallax addition processing as the perceptual distance D30 set for the virtual image V becomes longer. In other words, if the perceptual distance D30 set for the virtual images V shown in Figure 9 is in the order of V65 → V64 → V63 → V62 → V61 from longest to longest, the display control device 30 (processor 33) sets the correction amount C associated with the movement of the observer's eye position in the order of correction amount for V65 > correction amount for V64 > correction amount for V63 > correction amount for V62 > correction amount for V61. Note that virtual images V62 and V61 are the same type of virtual image and are displayed in close proximity, so the display control device 30 (processor 33) may set the correction amount for V62 and the correction amount for V61 associated with the movement of the observer's eye position to be the same.

[0036] Furthermore, in some embodiments, the display control device 30 (processor 33) may set the correction amount C associated with the movement of the observer's eye position to zero in the non-AR virtual image V70 (it may be unnecessary to correct for the movement of the observer's eye position).

[0037] Furthermore, in some embodiments, the display control device 30 (processor 33) may correct the non-AR virtual image V70 in accordance with the movement of the observer's eye position. In the example shown in Figure 9, the non-AR virtual images V70 (V71, V72) are positioned below the virtual image display area VS, and the area of ​​the road surface 311, which is a real object 300 that overlaps with them, is closer to the vehicle 1 than the area of ​​the road surface 311 that overlaps with the navigation virtual image V61 in Figure 9. In other words, in some embodiments, the display control device 30 (processor 33) may set the perceptual distance D30 of the non-AR virtual image V70 (V71, V72) to be shorter than the perceptual distance D30 of the AR virtual image V60 (more precisely, the navigation virtual image V61 located at the lowest position among the AR virtual images V60), and set the correction amount C of the non-AR virtual image V70 in response to the movement of the observer's eye position to be smaller than the correction amount C of the AR virtual image V60 (more precisely, the navigation virtual image V61 located at the lowest position among the AR virtual images V60) in response to the movement of the observer's eye position.

[0038] Figure 10 is a block diagram of a virtual image display system for a vehicle according to several embodiments. The display control device 30 comprises one or more I / O interfaces 31, one or more processors 33, one or more image processing circuits 35, and one or more memories 37. The various functional blocks shown in Figure 3 may consist of hardware, software, or a combination of both. Figure 10 is only one embodiment, and the illustrated components may be combined with fewer components, or additional components may be included. For example, the image processing circuit 35 (e.g., a graphics processing unit) may be included in one or more processors 33.

[0039] As shown in the figure, the processor 33 and the image processing circuit 35 are operably connected to the memory 37. More specifically, the processor 33 and the image processing circuit 35 can control the vehicle display system 10 (display device 40), for example, by executing a program stored in the memory 37, such as generating and / or transmitting image data. The processor 33 and / or the image processing circuit 35 may include at least one general-purpose microprocessor (e.g., a central processing unit (CPU)), at least one application-specific integrated circuit (ASIC), at least one field-programmable gate array (FPGA), or any combination thereof. The memory 37 includes any type of magnetic medium such as a hard disk, any type of optical medium such as CDs and DVDs, any type of semiconductor memory such as volatile memory, and non-volatile memory. The volatile memory may include DRAM and SRAM, and the non-volatile memory may include ROM and NVRAM.

[0040] As shown in the figure, the processor 33 is operably connected to the I / O interface 31. The I / O interface 31 communicates (also referred to as CAN communication) with, for example, the vehicle ECU 401 and / or other electronic devices (reference numerals 403 to 419 described later) installed in the vehicle, in accordance with the CAN (Controller Area Network) standard. The communication standard adopted by the I / O interface 31 is not limited to CAN, and includes, for example, wired communication interfaces such as CANFD (CAN with Flexible Data Rate), LIN (Local Interconnect Network), Ethernet (registered trademark), MOST (Media Oriented Systems Transport: MOST is a registered trademark), UART, or USB, or in-vehicle communication (internal communication) interfaces, which are short-range wireless communication interfaces within tens of meters, such as personal area networks (PANs) such as Bluetooth (registered trademark) networks, and local area networks (LANs) such as 802.11x Wi-Fi (registered trademark) networks. Furthermore, the I / O interface 31 may also include an external communication interface for outside vehicles, such as a wide-area communication network (e.g., an internet communication network) using cellular communication standards such as wireless wide-area network (WWAN0, IEEE802.16-2004 (WiMAX: Worldwide Interoperability for Microwave Access)), IEEE802.16e-based (Mobile WiMAX), 4G, 4G-LTE, LTE Advanced, and 5G.

[0041] As shown in the figure, the processor 33 is interconnected with the I / O interface 31 so as to be able to exchange information with various other electronic devices connected to the vehicle display system 10 (I / O interface 31). For example, the vehicle ECU 401, road information database 403, vehicle position detection unit 405, operation detection unit 407, eye position detection unit 409, external sensor 411, brightness detection unit 413, IMU 415, portable information terminal 417, and external communication device 419 are interconnected to the I / O interface 31 so as to be able to operate. The I / O interface 31 may also include a function to process (convert, calculate, analyze) information received from other electronic devices connected to the vehicle display system 10.

[0042] The display unit 40 is operably connected to the processor 33 and the image processing circuit 35. Therefore, the image displayed by the optical modulation element 51 may be based on image data received from the processor 33 and / or the image processing circuit 35. The processor 33 and the image processing circuit 35 control the image displayed by the optical modulation element 51 based on information obtained from the I / O interface 31.

[0043] The vehicle ECU 401 acquires information such as the state of the vehicle 1 from sensors and switches installed on the vehicle 1 (for example, the ON / OFF state of the start switch (e.g., accessory switch: ACC or ignition switch: IGN) (an example of start information), mileage, vehicle speed, accelerator pedal opening, brake pedal opening, engine throttle opening, injector fuel injection amount, engine speed, motor speed, steering angle, shift position, drive mode, various warning states, attitude (including roll angle and / or pitching angle), and vehicle vibration (including magnitude, frequency, and / or frequency of vibration)) and collects and manages (may also include control) the aforementioned state of the vehicle 1. As part of its function, it can output a signal indicating a numerical value of the aforementioned state of the vehicle 1 (for example, the vehicle speed of the vehicle 1) to the processor 33 of the display control device 30. Furthermore, in addition to simply transmitting a numerical value detected by a sensor, etc. (for example, a pitching angle of 3 degrees in the forward tilt direction) to the processor 33, or instead, the vehicle ECU 401 may transmit to the processor 33 a judgment result based on one or more states of the vehicle 1, including the numerical value detected by the sensor (for example, that the vehicle 1 satisfies the predetermined conditions for a forward tilt state), or / or an analysis result (for example, that the vehicle has become tilted forward due to braking, combined with brake pedal opening information). For example, the vehicle ECU 401 may output a signal to the display control device 30 indicating a judgment result that the vehicle 1 satisfies predetermined conditions stored in the memory (not shown) of the vehicle ECU 401. Furthermore, the I / O interface 31 may acquire the above-mentioned information from sensors and switches provided on the vehicle 1 without going through the vehicle ECU 401.

[0044] Furthermore, the vehicle ECU 401 may output an instruction signal to the display control device 30 that instructs the vehicle display system 10 to display an image. In this case, the coordinates, size, type, display mode, notification requirement of the image, and / or necessity-related information that serves as the basis for determining the notification requirement may be added to the instruction signal and transmitted.

[0045] The road information database 403 is included in a navigation device (not shown) installed in the vehicle 1, or in an external server connected to the vehicle 1 via an external communication interface (I / O interface 31). Based on the position of the vehicle 1 obtained from the vehicle position detection unit 405 (described later), the database may read information about the area around the vehicle 1 (information related to real objects around the vehicle 1), such as road information (lanes, white lines, stop lines, pedestrian crossings, road width, number of lanes, intersections, curves, forks in the road, traffic regulations, etc.) and feature information (buildings, bridges, rivers, etc.), including presence or absence, location (including distance to the vehicle 1), direction, shape, type, and detailed information, and transmit it to the processor 33. The road information database 403 may also calculate an appropriate route (navigation information) from the starting point to the destination and output a signal indicating the navigation information or image data indicating the route to the processor 33.

[0046] The vehicle position detection unit 405 is a GNSS (Global Navigation Satellite System) or the like installed on the vehicle 1, which detects the current position and direction of the vehicle 1 and outputs a signal indicating the detection result to the road information database 403, the portable information terminal 417 (described later), and / or external communication device 419, either via the processor 33 or directly. The road information database 403, the portable information terminal 417 (described later), and / or external communication device 419 may acquire the position information of the vehicle 1 from the vehicle position detection unit 405 continuously, intermittently, or at predetermined event intervals, and select and generate information about the area around the vehicle 1, which may then output to the processor 33.

[0047] The operation detection unit 407 is, for example, a hardware switch provided on the CID (Center Information Display) or instrument panel of the vehicle 1, or a software switch that combines an image and a touch sensor, and outputs operation information based on operations performed by the occupants of the vehicle 1 (the user seated in the driver's seat and / or the user seated in the passenger seat) to the processor 33. For example, the operation detection unit 407 outputs to the processor 33 display area setting information based on an operation to move the virtual image display area 100, eye box setting information based on an operation to move the eye box 200, and information based on an operation to set the observer's eye position 700.

[0048] The eye position detection unit 409 may include a camera, such as an infrared camera, to detect the eye position 700 (see Figure 1) of an observer seated in the driver's seat of the vehicle 1, and may output the captured image to the processor 33. The processor 33 may acquire the captured image (an example of information that can estimate the eye position 700) from the eye position detection unit 409, and by analyzing this captured image using a method such as pattern matching, detect the coordinates of the observer's eye position 700, and output a signal indicating the detected coordinates of the eye position 700 to the processor 33.

[0049] Furthermore, the eye position detection unit 409 may output to the processor 33 the analysis results obtained by analyzing the camera's captured image (for example, a signal indicating where in the spatial region corresponding to a set of multiple display parameters the observer's eye position 700 belongs). Note that the method for obtaining the observer's eye position 700 in the vehicle 1, or information that can estimate the observer's eye position 700, is not limited to these, and may be obtained using known eye position detection (estimation) techniques.

[0050] Furthermore, the eye position detection unit 409 may detect the movement speed and / or direction of the observer's eye position 700 and output a signal indicating the movement speed and / or direction of the observer's eye position 700 to the processor 33.

[0051] Furthermore, if the eye position detection unit 409 detects (10) a signal indicating that the observer's eye position 700 is outside the eye box 200, (20) a signal indicating that the observer's eye position 700 is estimated to be outside the eye box 200, or (30) a signal indicating that the observer's eye position 700 is predicted to be outside the eye box 200, it may determine that a predetermined condition has been met and output a signal indicating this state to the processor 33.

[0052] (20) Signals that indicate the observer's eye position 700 is outside the eye box 200 include (21) a signal indicating that the observer's eye position 700 cannot be detected, (22) a signal indicating that the observer's eye position 700 cannot be detected after a movement of the observer's eye position 700 has been detected, and / or (23) a signal indicating that either the observer's eye position 700R or 700L is in the vicinity of the boundary 200A of the eye box 200 (the vicinity includes, for example, being within a predetermined coordinate range from the boundary 200A).

[0053] (30) The signals that predict the observer's eye position 700 will be outside the eye box 200 include (31) a signal indicating that the newly detected eye position 700 is greater than or equal to an eye position movement distance threshold pre-stored in memory 37 relative to previously detected eye positions 700 (i.e., the movement of the eye position within a predetermined unit time is greater than the specified range), (32) a signal indicating that the movement speed of the eye position is greater than or equal to an eye position movement speed threshold pre-stored in memory 37, and so on.

[0054] Furthermore, the eye position detection unit 409 may also function as a gaze direction detection unit 409. The gaze direction detection unit 409 may include an infrared camera or a visible light camera that captures the face of an observer seated in the driver's seat of the vehicle 1, and may output the captured image to the processor 33. The processor 33 can acquire the captured image (an example of information that can estimate the gaze direction) from the gaze direction detection unit 409 and determine the observer's gaze direction (and / or the gaze position) by analyzing this captured image. The gaze direction detection unit 409 may also analyze the captured image from the camera and output a signal indicating the observer's gaze direction (and / or the gaze position), which is the result of the analysis, to the processor 33. The method for acquiring information that can estimate the gaze direction of the observer of the vehicle 1 is not limited to these, and may be acquired using other known gaze direction detection (estimation) techniques such as the EOG (Electro-oculogram) method, corneal reflection method, scleral reflection method, Purkinje image detection method, search coil method, and infrared fundus camera method.

[0055] The external sensor 411 detects real objects present in the vicinity of the vehicle 1 (front, side, and rear). The real objects detected by the external sensor 411 may include, for example, obstacles (pedestrians, bicycles, motorcycles, other vehicles, etc.), the road surface of the driving lane, lane markings, roadside objects, and / or ground features (buildings, etc.), as described later. The external sensor consists of, for example, a detection unit consisting of a radar sensor such as a millimeter-wave radar, ultrasonic radar, or laser radar, a camera, or a combination thereof, and a processing unit that processes (data fuses) the detection data from one or more of these detection units. Conventional and well-known methods are applied to object detection using these radar sensors and camera sensors. These sensors can detect objects to determine whether or not a real object exists in three-dimensional space. If a real object exists, it can detect its position (relative distance from the vehicle 1, position in the left-right direction when the vehicle 1 is moving in the front-rear direction, position in the up-down direction, etc.), size (size in the lateral direction (left-right direction), height direction (up-down direction), direction of movement (lateral direction (left-right direction), depth direction (front-rear direction)), speed of movement (lateral direction (left-right direction), depth direction (front-rear direction)), and / or type. One or more external sensors 411 can detect a real object in front of the vehicle 1 at each detection cycle and output real object information (information such as whether or not a real object exists, and if a real object exists, its position, size, and / or type) to the processor 33. This real object information may also be transmitted to the processor 33 via other equipment (e.g., the vehicle ECU 401). Furthermore, if a camera is used as a sensor, an infrared camera or near-infrared camera is preferable so that real objects can be detected even when the surroundings are dark, such as at night. Furthermore, when using a camera as a sensor, a stereo camera that can acquire distance and other parameters through parallax is preferable.

[0056] The brightness detection unit 413 detects the illuminance or brightness of a predetermined range of the foreground in front of the passenger compartment of the vehicle 1 as the external brightness (an example of brightness information), or the illuminance or brightness inside the passenger compartment as the interior brightness (an example of brightness information). The brightness detection unit 413 is, for example, a phototransistor or photodiode, and is mounted on the instrument panel, rearview mirror, or HUD device 20 of the vehicle 1 as shown in Figure 1.

[0057] The IMU415 may include a combination of one or more sensors (e.g., an accelerometer and a gyroscope) configured to detect the position, orientation, and changes thereto (change rate, change acceleration) of the vehicle 1 based on inertial acceleration. The IMU415 may output the detected values ​​(including signals indicating the position, orientation, and changes thereto (change rate, change acceleration) of the vehicle 1) and the results of analyzing the detected values ​​to the processor 33. The results of the analysis may include signals indicating whether the detected values ​​meet predetermined conditions, and for example, a signal indicating that the vehicle 1's behavior (vibration) is minimal based on values ​​related to changes in the position or orientation (change rate, change acceleration) of the vehicle 1.

[0058] The personal information terminal 417 is a smartphone, laptop computer, smartwatch, or other information device that can be carried by an observer (or other occupants of the vehicle 1). The I / O interface 31 can communicate with the personal information terminal 417 by pairing with it and acquire data recorded in the personal information terminal 417 (or a server via the personal information terminal). The personal information terminal 417 may, for example, have the same functions as the road information database 403 and the vehicle position detection unit 405 described above, acquire the road information (an example of real object-related information), and transmit it to the processor 33. The personal information terminal 417 may also acquire commercial information (an example of real object-related information) related to commercial facilities near the vehicle 1 and transmit it to the processor 33. The personal information terminal 417 may transmit schedule information of the person holding the personal information terminal 417 (e.g., an observer), incoming call information on the personal information terminal 417, email reception information, etc., to the processor 33, and the processor 33 and the image processing circuit 35 may generate and / or transmit image data related to these.

[0059] The external communication device 419 is a communication device that exchanges information with the vehicle 1, and is, for example, a communication device that connects with the vehicle 1 via vehicle-to-vehicle communication (V2V: Vehicle To Vehicle), a communication device that connects with a pedestrian via vehicle-to-pedestrian communication (V2P: Vehicle To Pedestrian) (a portable information terminal carried by a pedestrian), or a communication device that connects via vehicle-to-roadside infrastructure (V2I: Vehicle To Roadside Infrastructure). In a broader sense, it includes everything that is connected via communication with the vehicle 1 (V2X: Vehicle To Everything). The external communication device 419 may, for example, acquire the location of pedestrians, bicycles, motorcycles, other vehicles (such as preceding vehicles), the road surface, lane markings, roadside objects, and / or landmarks (such as buildings), and output this information to the processor 33. Furthermore, the external communication device 419 may have the same functions as the vehicle position detection unit 405 described above, acquire the position information of the vehicle 1 and transmit it to the processor 33, and may also have the functions of the road information database 403 described above, acquire the road information (an example of real object-related information) and transmit it to the processor 33. Note that the information acquired from the external communication device 419 is not limited to what is described above.

[0060] The software components stored in memory 37 include an eye position detection module 502, an eye position estimation module 504, an eye position prediction module 506, an eye position state determination module 508, a vehicle state determination module 510, an eye-tracking image processing module 512, a graphics module 514, a light source drive module 516, and an actuator drive module 518, among others.

[0061] Figures 11A, 11B, and 11 are flowcharts illustrating a method S100 that performs an image correction operation based on the observer's eye position. Method S100 is performed in a HUD device 20 including a display and a display control device 30 that controls the HUD device 20. Some of the operations of method S100 described below can be optionally combined, some of the procedures of some operations can be optionally modified, and some operations can be optionally omitted.

[0062] First, the display control device 30 (processor 33) detects the observer's eye position 700 (step S110).

[0063] (Step S112) In step S110 of some embodiments, the display control device 30 (processor 33) detects the observer's eye position 700 via the eye position detection unit 409 by executing the eye position detection module 502 shown in Figure 10 (acquires information indicating the eye position 700). The eye position detection module 502 includes various software components for performing various operations related to detecting coordinates indicating the observer's eye position 700 (positions in the X,Y axis directions, which are examples of signals indicating the eye position 700), detecting coordinates indicating the observer's eye height (positions in the Y axis direction, which are examples of signals indicating the eye position 700), detecting coordinates indicating the observer's eye height and depth position (positions in the Y and Z axis directions, which are examples of signals indicating the eye position 700), and / or detecting coordinates indicating the observer's eye position 700 (positions in the X,Y,Z axis directions, which are examples of signals indicating the eye position 700).

[0064] The eye position 700 detected by the eye position detection module 502 may include the respective positions 700R and 700L of the right and left eyes, a predetermined position among the right eye position 700R and the left eye position 700L, a detectable (easily detectable) position among the right eye position 700R and the left eye position 700L, or a position calculated from the right eye position 700R and the left eye position 700L (for example, the midpoint between the right eye position and the left eye position). For example, the eye position detection module 502 determines the eye position 700 based on the observed position acquired from the eye position detection unit 409 immediately before updating the display settings.

[0065] Furthermore, the eye position detection unit 409 may detect the direction and / or speed of movement of the observer's eye position 700 based on multiple observation positions with different detection timings of the observer's eyes acquired from the eye position detection unit 409, and output a signal indicating the direction and / or speed of movement of the observer's eye position 700 to the processor 33.

[0066] (Step S114) Furthermore, in step S110 of some embodiments, the display control device 30 (processor 33) may obtain information that allows for the estimation of eye position by executing the eye position estimation module 504 (step S114). Information that allows for the estimation of eye position includes, for example, an image captured from the eye position detection unit 409, the position of the driver's seat of the vehicle 1, the position of the observer's face, the height of their sitting position, or the observation positions of the eyes of multiple observers. The eye position estimation module 504 estimates the eye position 700 of the observer in the vehicle 1 from the information that allows for the estimation of eye position. The eye position estimation module 504 includes various software components for performing various operations related to estimating the observer's eye position 700, such as estimating the observer's eye position 700 from an image captured from the eye position detection unit 409, the position of the driver's seat of the vehicle 1, the position of the observer's face, the height of their sitting position, or the observation positions of the eyes of multiple observers. In other words, the eye position estimation module 504 may include table data, calculation formulas, etc., for estimating the observer's eye position 700 from information that allows for the estimation of eye position.

[0067] (Step S116) Furthermore, in step S110 of some embodiments, the display control device 30 (processor 33) may execute the eye position prediction module 506 to obtain information that can predict the observer's eye position 700 (step S116). The information that can predict the observer's eye position 700 is, for example, the most recent observation position obtained from the eye position detection unit 409, or one or more past observation positions. The eye position prediction module 506 includes various software components for performing various operations related to predicting the eye position 700 based on the information that can predict the observer's eye position 700. Specifically, for example, the eye position prediction module 506 predicts the observer's eye position 700 at the time when the image to which the new display settings have been applied is viewed by the observer. The eye position prediction module 506 may use a prediction algorithm such as the least squares method, a Kalman filter, an α-β filter, or a particle filter to predict the next value using one or more past observation positions.

[0068] (Step S120) Next, the display control device 30 (processor 33) determines whether predetermined conditions are met (step S120).

[0069] Figure 11B is a diagram illustrating the operation of the eye position state determination module 508 in step S130.

[0070] (Step S130) In step S120 of some embodiments, the display control device 30 (processor 33) executes the eye position state determination module 508 shown in Figure 10 to determine whether the eye position 700 (movement of the eye position 700) satisfies predetermined conditions based on information indicating the eye position, information that can estimate the eye position, or information that can predict the eye position acquired in step S110.

[0071] Figure 12 shows the following detected at predetermined period times t(t1, t2, t3...t10): (11) vertical eye position (or head position) Py(Y1, Y2, Y3...Y10), (12) change in vertical eye position (or head position) ΔPy(Py1(=Y2-Y1), Py2(=Y3-Y2), Py3(=Y4-Y3),...Py9(=Y10-Y9)), (13) vertical movement velocity Vy(Vy1(=Py1 / Δt), Vy2(=Py2 / Δt), Vy3(=Py3 / Δt),... This table shows (21) the vertical eye position (or head position) Px(X1,X2,X3···X10), (22) the change in the vertical eye position (or head position) ΔPx(Px1(=X2-X1),Px2(=X3-X2),Px3(=X4-X3),···Px9(=X10-X9)), and (23) the vertical movement speed Vx(Vx1(=Px1 / Δt),Vx2(=Px2 / Δt),Vx3(=Px3 / Δt),···Vx9(=Px9 / Δt)).

[0072] (Step S131) In step S130 of some embodiments, the display control device 30 (processor 33) may execute the eye position state determination module 508 shown in Figure 10 and determine that a predetermined condition is satisfied if the eye position in the left-right direction changes continuously in one direction. The eye position state determination module 508 may determine that the predetermined condition is satisfied if, for example, the amount of change ΔPx of the eye position in the left-right direction shown in Figure 12 is detected to have changed continuously in one direction (in this case, to the right), such as Px3 (right) → Px4 (right).

[0073] (Step S132) Furthermore, in step S130 of some embodiments, the display control device 30 (processor 33) can compare the eye position movement speed Vy in the left-right direction with a predetermined first threshold (not shown) that is pre-stored in the memory 37 (or set by the operation detection unit 407) by executing the eye position state determination module 508 shown in Figure 10, and may determine that a predetermined condition is satisfied if the eye position movement speed Vy in the left-right direction is faster than the predetermined first threshold.

[0074] (Step S133) Furthermore, in step S130 of some embodiments, the display control device 30 (processor 33) can, for example, compare the ratio of the change in the vertical eye position to the change in the horizontal eye position (Px / Py) with a predetermined second threshold (not shown) that is pre-stored in memory 37 (or set by the operation detection unit 407) by executing the eye position state determination module 508 shown in Figure 10. If the ratio of the change in the vertical eye position to the change in the horizontal eye position (Px / Py) is faster than the predetermined first threshold, the device may determine that a predetermined condition is met.

[0075] (Step S134) Furthermore, in step S130 of some embodiments, the display control device 30 (processor 33) can, for example, compare the amount of change ΔPy of the eye position in the vertical direction with a predetermined third threshold (not shown) that is pre-stored in the memory 37 (or set by the operation detection unit 407) by executing the eye position state determination module 508 shown in Figure 10, and may determine that a predetermined condition is met if the movement speed Vy of the eye position in the horizontal direction is faster than the predetermined first threshold.

[0076] (Step S135) Furthermore, in some embodiments, the eye position state determination module 508 in Figure 10 may determine whether the observer's eye position 700 is in an unstable state, and if it is determined that the observer's eye position 700 is in an unstable state, it may determine that the predetermined conditions are met. The eye position state determination module 508 includes various software components for performing various operations related to determining whether the stability of the observer's eye position is low (unstable), and if the stability of the observer's eye position is low, determining that it is in an unstable state (step S135). That is, the eye position state determination module 508 may include thresholds, table data, calculation formulas, etc., in order to determine whether the observer's eye position 700 is in an unstable state from detection information, estimation information, or prediction information of the eye position 700.

[0077] (An example of step S135) The eye position detection module 502 calculates the variance of each position data of multiple observation positions acquired from the eye position detection unit 409 within a predetermined measurement time, and the eye position state determination module 508 may determine that the stability of the observer's eye position is low (unstable) if the variance calculated by the eye position detection module 502 is greater than a predetermined threshold stored in the memory 37 (or set by the operation detection unit 407).

[0078] (An example of step S135) The eye position detection module 502 calculates the deviation of each position data of multiple observation positions acquired from the eye position detection unit 409 within a predetermined measurement time, and the eye position state determination module 508 may determine that the stability of the observer's eye position is low (unstable) (not in an unstable state) if the deviation calculated by the eye position detection module 502 is greater than a predetermined threshold stored in the memory 37 (or set by the operation detection unit 407).

[0079] Alternatively, without using the variance or deviation in step S135, the eye position detection module 502 may be capable of identifying the eye box 200 into multiple partial fields of view (for example, 25 regions divided into 5 vertically and 5 horizontally), and may determine that the observer's eye position is unstable (not unstable) when the number of these partial fields of view to which the eye position 700 has moved per predetermined unit time exceeds a predetermined threshold. Alternatively, the eye position detection module 502 may determine that the observer's eye position is unstable (not unstable) when the total distance traveled by the eye position 700 per predetermined unit time (the sum of the distances between multiple observation positions acquired multiple times per unit time) exceeds a predetermined threshold.

[0080] (Step S136) Furthermore, in some embodiments, the eye position state determination module 508 in Figure 10 determines whether the detection operation of the observer's eye position 700 is in an unstable state, and if it is determined to be in an unstable state, it determines that the predetermined conditions are met. The eye position state determination module 508 (10) determines whether the observer's eye position 700 can be detected, and if the eye position 700 cannot be detected, it determines that it is in an unstable state (an example of step S136), (20) determines whether it can be estimated that the detection accuracy of the observer's eye position 700 has decreased, and if it can be estimated that the detection accuracy of the eye position 700 has decreased, it determines that it is in an unstable state (an example of step S136), (30) determines whether the observer's eye position 700 is outside the eye box 200, and if it is outside the eye box 200, it determines that it is in an unstable state The system includes various software components for performing various operations related to (1) determining whether the observer's eye position 700 is outside the eye box 200 (an example of step S136), (2) determining whether the observer's eye position 700 is estimated to be outside the eye box 200, and if it is estimated to be outside the eye box 200, determining that it is in an unstable state (an example of step S136), or (3) determining whether the observer's eye position 700 is predicted to be outside the eye box 200, and if it is predicted to be outside the eye box 200, determining that it is in an unstable state (an example of step S136). In other words, the eye position state determination module 508 may include thresholds, table data, calculation formulas, etc., to determine whether the detection operation of the observer's eye position 700 is in an unstable state based on detection information, estimation information, or prediction information of the eye position 700.

[0081] (An example of step S136) A method for determining whether or not the observer's eye position 700 can be detected includes (1) obtaining a signal from the eye position detection unit 409 indicating that the eye position 700 cannot be detected, (2) determining that some (for example, more than a predetermined number of times) or all of the observer's eye position obtained from the eye position detection unit 409 within a predetermined period cannot be detected, or (3) determining that the observer's eye position 700 cannot be detected (that the detection of the observer's eye position 700 is unstable) based on a combination of these factors (however, the determination method is not limited to these).

[0082] (An example of step S136) A method for determining that the detection accuracy of the observer's eye position 700 has decreased includes: (1) obtaining a signal from the eye position detection unit 409 indicating that the detection accuracy of the eye position 700 is estimated to have decreased; (2) finding that some (e.g., more than a predetermined number of times) or all of the observer's eye position obtained from the eye position detection unit 409 within a predetermined period cannot be detected; (3) finding that the eye position detection module 502 cannot detect the observer's eye position 700 in normal operation; (4) finding that the eye position estimation module 504 cannot estimate the observer's eye position 700 in normal operation; (5) finding that the eye position prediction module 506 cannot predict the observer's eye position 700 in normal operation; or determining that the detection accuracy of the observer's eye position 700 has decreased based on a combination of these factors (however, the determination method is not limited to these).

[0083] (An example of step S136) A method for determining whether the observer's eye position 700 is outside the eye box 200 includes (1) acquiring some (for example, a predetermined number of times or more) or all of the observer's eye position obtained from the eye position detection unit 409 within a predetermined period of time outside the eye box 200, (2) the eye position detection module 502 detecting the observer's eye position 700 outside the eye box 200, or determining that the observer's eye position 700 is outside the eye box 200 (the observer's eye position 700 is in an unstable state) based on a combination of these (however, the determination method is not limited to these).

[0084] (An example of step S136) A method for determining whether the observer's eye position 700 can be estimated to be outside the eye box 200 includes (1) the observation of the observer's eye position 700 becoming undetectable after the eye position detection unit 409 detects movement of the observer's eye position 700, (2) the eye position detection module 502 detecting the observer's eye position 700 near the boundary of the eye box 200, and (3) the eye position detection module 502 detecting either the observer's right eye position 700R or left eye position 700L near the boundary of the eye box 200, or a combination thereof, determining that the observer's eye position 700 can be estimated to be outside the eye box 200 (i.e., the observer's eye position 700 is in an unstable state) (however, the determination method is not limited to these).

[0085] (An example of step S136) A method for determining whether the observer's eye position 700 is predicted to be outside the eye box 200 includes (1) the eye position prediction module 506 predicting that the observer's eye position 700 will be outside the eye box 200 after a predetermined time; and (2) the eye position detection module 502 determining that the newly detected eye position 700 is greater than or equal to an eye position movement distance threshold pre-stored in the memory 37 relative to previously detected eye positions 700 (i.e., the movement speed of the eye position 700 is greater than or equal to an eye position movement speed threshold pre-stored in the memory 37), or, based on a combination of these, that the observer's eye position 700 is predicted to be outside the eye box 200 (i.e., the observer's eye position 700 is in an unstable state) (however, the determination method is not limited to these).

[0086] Figure 11C illustrates the operation of the vehicle state determination module 510 in step S140.

[0087] (Step S140) In step S120 of some embodiments, the display control device 30 (processor 33) may determine whether the vehicle state satisfies predetermined conditions by executing the vehicle state determination module 510 shown in Figure 10.

[0088] (Step S141) In step S140 of some embodiments, the display control device 30 (processor 33) may execute the vehicle status determination module 510 shown in Figure 10 to estimate whether or not vehicle 1 is running based on information such as mileage, vehicle speed, and shift position obtained from the vehicle ECU 401, information indicating the vehicle's position obtained from the vehicle position detection unit 405, and information obtained from the IMU 415. If it is estimated that vehicle 1 is running, it may determine that the predetermined conditions are met.

[0089] (Step S142) In step S140 of some embodiments, the display control device 30 (processor 33) may execute the vehicle state determination module 510 shown in Figure 10 to estimate whether or not vehicle 1 is vibrating (more specifically, to estimate whether or not vehicle 1 is driving on a rough road based on, for example, the frequency and / or amplitude of vibrations of the vehicle obtained from the vehicle ECU 401, and information obtained from the IMU 415), and if it estimates that vehicle 1 is vibrating, it may determine that the predetermined conditions are met.

[0090] (Step S143) In step S140 of some embodiments, the display control device 30 (processor 33) may execute the vehicle status determination module 510 shown in Figure 10 to determine whether the vehicle 1 is in manual driving mode, for example, based on information regarding the driving mode (automatic driving mode or manual driving mode) obtained from the vehicle ECU 401, and if the vehicle 1 is in manual driving mode, it may determine that the predetermined conditions are met.

[0091] (Step S150) Refer to Figure 11A again. After it is determined in step S120 whether the predetermined conditions are met, the display control device 30 (processor 33) executes the eye-tracking image processing module 512 to correct the position of the image to be displayed on the display 40 based on the observer's eye position 700 (specifically, the eye position 700 is the vertical eye position Py and the horizontal eye position Px in Figure 12, and the front-to-back eye position may also be included) (eye-tracking image processing).

[0092] The eye-tracking image processing module 512 in Figure 10 switches between the first image correction process S160 (step S160) and the second image correction process S170 (step S170) based on the determination result in step S120.

[0093] (First image correction process S160) The eye-tracking image processing module 512 in Figure 10, if it is determined in step S120 that the predetermined conditions are not met, corrects the vertical position of the virtual image V by a first correction amount Cy1 corresponding to the change in the vertical eye position ΔPy, and corrects the horizontal position of the virtual image V according to the change in the horizontal eye position ΔPx. The first correction amount Cy1 (and the second correction amount Cy2 described later) is a parameter that gradually increases as the change in the vertical eye position ΔPy increases. The first correction amount Cy1 (and the second correction amount Cy2 described later) is a parameter that gradually increases as the perceptual distance D30 set for the virtual image V increases. The first image correction process S160 includes image position correction that completely reproduces natural motion parallax, as if the virtual image V were fixed to the set target position PT from the perspective of each vertical eye position Py, and may also include image position correction that approaches natural motion parallax. In other words, the first image correction process S160 adjusts the display position of the virtual image V to the position of the intersection of the line connecting the target position PT set for the virtual image V and the observer's eye position 700, and the virtual image display area VS (brings the display position of the virtual image V closer).

[0094] (An example of the second image correction process S170.) The eye-tracking image processing module 512 in Figure 10, if it is determined in step S120 that the predetermined conditions are met, corrects the vertical position of the virtual image V by a second correction amount Cy2 corresponding to the change in the vertical eye position ΔPy, and corrects the horizontal position of the virtual image V according to the change in the horizontal eye position ΔPx. The second correction amount Cy2 is smaller than the first correction amount Cy1 for the change in the vertical eye position ΔPy in the first image correction processing S160. Specifically, for example, if the first correction amount Cy1 for the change in the vertical eye position ΔPy is set to 100%, then the second correction amount Cy2 for the same change in the vertical eye position ΔPy is lower than 25%. In a broader sense, the second correction amount Cy2 only needs to be smaller than the first correction amount Cy1, so it should be less than 100% of the first correction amount Cy1, but preferably it should be less than 60% of the first correction amount Cy1.

[0095] (An example of the second image correction process S170.) Furthermore, in some embodiments, the eye-tracking image processing module 512 in Figure 10 may, if it is determined in step S120 that the predetermined conditions are met, set the second correction amount Cy2 corresponding to the change in the vertical eye position ΔPy to zero. In this case, the eye-tracking image processing module 511 corrects the horizontal position of the virtual image V only according to the change in the horizontal eye position ΔPx (horizontal eye position Py).

[0096] (An example of the second image correction process S170.) Incidentally, the image position correction amount Cx2 for the change in eye position ΔPx in the left-right direction in the second image correction process S170 is equal to the image position correction amount Cx1 for the change in eye position ΔPx in the left-right direction in the first image correction process S160. However, it is not limited to this, and the image position correction amount Cx2 for the change in eye position ΔPx in the left-right direction in the second image correction process S170 may be set lower than the image position correction amount Cx1 for the change in eye position ΔPx in the left-right direction in the first image correction process S160, but may be set higher than the ratio of the second correction amount Cy2 to the first correction amount Cy1 for the change in eye position ΔPy in the up-down direction (Cx2 / Cx1 > Cy2 / Cy1).

[0097] (Step S181) In step S170 of some embodiments, if the display control device 30 (processor 33) determines that a predetermined release condition has been met, it proceeds from the second image correction process S170 to the first image correction process S160.

[0098] The predetermined release condition includes the elapsed time (for example, 20 seconds) since the transition to the second image correction process S170. The eye-tracking image processing module 512 may perform a timer after the transition to the second image correction process S170, and determine that the release condition has been satisfied when the predetermined time, which has been stored in the memory 37 in advance (or set by the operation detection unit 407), has elapsed.

[0099] Furthermore, the predetermined release condition may include the fact that the predetermined condition is no longer satisfied in step S120. That is, the predetermined release condition may include detecting that at least one of steps S131 to S136 and steps S141 to S143 has transitioned from a state in which the predetermined condition is satisfied to a state in which the predetermined condition is no longer satisfied. Furthermore, the predetermined release condition may include the fact that a predetermined time (for example, 20 seconds) has elapsed since the predetermined condition was no longer satisfied in step S120.

[0100] (Step S182) In step S170 of some embodiments, if the display control device 30 (processor 33) determines that a predetermined release condition has been met, it proceeds to a third image correction process in which a third correction amount Cy3 for the image position with respect to the change amount ΔPy of the eye position in the vertical direction is smaller than the first correction amount Cy1 in the first image correction process S160 and larger than the second correction amount Cy2 in the second image correction process S170. For example, if the first correction amount Cy1 for the change amount ΔPy of the eye position in the vertical direction is 100% and the second correction amount Cy2 is 20%, then the third correction amount Cy3 is 50%.

[0101] (An example of step S182) In step S182 of some embodiments, the display control device 30 (processor 33) may, after proceeding to the third image correction process, gradually increase the third correction amount Cy3 for the change in vertical eye position ΔPy over time so that it approaches the first correction amount Cy1 for the change in vertical eye position ΔPy. For example, if the first correction amount Cy1 for the change in vertical eye position ΔPy is 100%, the second correction amount Cy2 is 20%, and the third correction amount Cy3 is 50%, the display control device 30 (processor 33) may gradually increase the third correction amount Cy3 over time, such as 55% → 60% → 65% → ... → 100%, so that it approaches the first correction amount Cy1.

[0102] Refer again to Figure 10. The graphics module 514 in Figure 10 includes various known software components for generating image data through image processing such as rendering and for driving the display 40. The graphics module 514 may also include various known software components for changing the type (moving image, still image, shape), arrangement (position coordinates, angle), size, display distance (in the case of 3D), and visual effects (e.g., brightness, transparency, saturation, contrast, or other visual characteristics) of the displayed image. The graphics module 514 can generate image data that can be viewed by an observer based on the following parameters: image type (one example of a display parameter), image position coordinates (one example of a display parameter), image angle (such as pitching angle around the X direction, yaw rate angle around the Y direction, and rolling angle around the Z direction, and is one example of a display parameter), image size (one example of a display parameter), image color (one example of a display parameter set by hue, saturation, brightness, etc.), and the intensity of the image's perspective representation (one example of a display parameter set by the vanishing point position, etc.), and drive the optical modulation element 50.

[0103] The light source drive module 516 includes various known software components for performing the task of driving the light source unit 24. The light source drive module 516 can drive the light source unit 24 based on set display parameters.

[0104] The actuator drive module 518 includes various known software components for performing the driving of the first actuator 28 and / or the second actuator 29. Based on set display parameters, the actuator drive module 518 can drive the first actuator 28 and the second actuator 29.

[0105] Figure 13 illustrates a HUD device 20 in several embodiments, in which the eyebox 200 can be moved vertically by rotating the relay optical system 80 (curved mirror 81). In some embodiments, the display control device 30 (processor 33) can rotate the relay optical system 80 (curved mirror 81) and move the eyebox 200 vertically (in the Y-axis direction) by controlling, for example, the first actuator 28. Typically, when the eyebox 200 is positioned as the relatively upper eyebox 201 shown in Figure 13, the position of the virtual image display area PS is at the position indicated by the relatively lower reference numeral PS1, and when the eyebox 200 is positioned as the lower eyebox 203 shown in Figure 13, the position of the virtual image display area PS is at the position indicated by the relatively upper reference numeral PS3. In some embodiments, the display control device 30 (processor 33) may, by executing the eye-tracking image processing module 512, reduce the correction amount Cy of the position of the image displayed on the optical modulation element 50 with respect to the change in vertical eye position (or head position) when the eye box 200 is positioned above a predetermined height threshold (in other words, when the control value of the first actuator 28 exceeds an actuator control threshold such that the eye box 200 is positioned above a predetermined height threshold). The actuator drive module 518 may automatically change the height of the eye box 200 according to the vertical position of the eye position 700 (or head position 710), or it may change the height of the eye box 200 according to user operation by the operation detection unit 407. In other words, the eye-tracking image processing module 512 may include thresholds, table data, and calculation formulas for switching the correction amount Cy of the image position displayed on the optical modulation element 50 in response to the change in the vertical direction of the eye position (or head position), based on information regarding the height of the eye box 200, information regarding the control value of the actuator, information regarding the vertical position of the eye position 700 (or head position 710) which can automatically adjust the height of the eye box 200, or operation information from the operation detection unit 407 which adjusts the height of the eye box 200.

[0106] Furthermore, in some embodiments, the display control device 30 (processor 33) may gradually or continuously decrease the correction amount Cy of the position of the image displayed on the optical modulation element 50 in response to the change in vertical eye position (or head position) as the eye box 200 rises to a predetermined height (in other words, as the control value of the first actuator 28 is changed to raise the eye box 200). That is, the eye-tracking image processing module 512 may include thresholds, table data, calculation formulas, etc., for adjusting the correction amount Cy of the position of the image displayed on the optical modulation element 50 in response to the change in vertical eye position (or head position) based on information about the height of the eye box 200, information about the control value of the actuator, information about the vertical position of the eye position 700 (or head position 710) which can automatically adjust the height of the eye box 200, or operation information from the operation detection unit 407 which adjusts the height of the eye box 200.

[0107] As described above, the display control device 30 of this embodiment comprises at least a display unit 40 that displays an image, and a relay optical system 80 that projects the light of the image displayed by the display unit 40 onto a projection member, and is a display control device 30 that performs display control in a head-up display device 20 that causes the user of a vehicle to view a virtual image of the image superimposed on the foreground, and comprises one or more processors 33, a memory 37, and one or more computer programs stored in the memory 37 and configured to be executed by one or more processors 33, wherein the processor 33 acquires the user's vertical eye position (and / or head position) Py and the horizontal eye position (and / or head position) Px of the vehicle, and the vertical eye position (or head position) Py and the horizontal eye position (and / or The system switches between a first image correction process S160, which corrects the position of the image displayed on the display 40 based at least on the head position (Px), and a second image correction process S170, which corrects the position of the image displayed on the display 40 based at least on the vertical eye position (or head position) Py and the horizontal eye position (or head position) Px, and ensures that the second correction amount Cy2 of the image position with respect to the change amount ΔPy of the vertical eye position (or head position) is smaller than the first correction amount Cy1 of the image position with respect to the change amount ΔPy of the vertical eye position (or head position) in the first image correction process S160, or corrects the position of the image displayed on the display 40 based at least on the horizontal eye position (or head position) Px, and sets the correction amount of the image position with respect to the change amount ΔPy of the vertical eye position (or head position) to zero.

[0108] Furthermore, in some embodiments, the processor 33 may select the second image correction process S170 if at least one of the following conditions is met: (1) the left-right eye position (or head position) Px changes continuously in one direction; (2) a change in the up-down eye position (and / or head position) and a change in the left-right eye position (and / or head position) are detected, and the ratio of the change amount ΔPy in the up-down eye position (or head position) to the change amount ΔPx in the left-right eye position (or head position) is less than a predetermined first threshold; and (3) a change in the up-down eye position (or head position) Py and a change in the left-right eye position (or head position) Px are detected, and the change amount ΔPy in the up-down eye position (or head position) is less than a predetermined second threshold. This reduces the discomfort caused to the observer by detecting up-down eye position (head position) movement that the observer is not aware of when the observer moves their eye position (head position) in the left-right direction.

[0109] Furthermore, in some embodiments, the processor 33 may select the second image correction process S170 if, after it becomes impossible to obtain the vertical eye position (and / or head position) Py and / or the horizontal eye position (and / or head position) Px, a change in the vertical eye position (or head position) Py and a change in the horizontal eye position (or head position) Px are detected. In other words, the processor 33 may proceed to the second image correction process S170 if, in the first image correction process S160, one or more of the vertical eye position Py, vertical head position Py, horizontal eye position Px, and horizontal head position Px are no longer detectable.

[0110] Furthermore, in some embodiments, the processor 33 may, in the second image correction process S170, after a predetermined time has elapsed, correct the position of the image displayed on the display 40 based at least on the vertical eye position (or head position) Py and the horizontal eye position (or head position) Px, and switch to a third image correction process S182 in which the third correction amount Cy3 of the image position with respect to the change amount ΔPy of the vertical eye position (or head position) is smaller than the first correction amount Cy1 in the first image correction process S160 and larger than the second correction amount Cy2 in the second image correction process S170.

[0111] Furthermore, in some embodiments, if the processor 33 detects in the second image correction process S170 that the change amount ΔPy of the vertical eye position (or head position) has become greater than a predetermined third threshold, it may correct the position of the image displayed on the display 40 based at least on the vertical eye position (or head position) Py and the horizontal eye position (or head position) Px, and switch to a third image correction process S182 in which the third correction amount Cy3 of the image position with respect to the change amount ΔPy of the vertical eye position (or head position) is smaller than the first correction amount Cy1 in the first image correction process S160 and larger than the second correction amount Cy2 in the second image correction process S170.

[0112] Furthermore, in some embodiments, the processor 33 may change the third correction amount Cy3 over time in the third image correction process S182 so that it approaches the first correction amount Cy1 from the first image correction process S160.

[0113] Furthermore, in some embodiments, the head-up display device 20 displays a distant virtual image V1 (for example, virtual images V64-V65 shown in Figure 9) perceived at a position a first distance away from a reference point set on the vehicle side, and a nearby virtual image V2 (for example, virtual images V61-V63 shown in Figure 9) perceived at a position a second distance shorter than the first distance. The processor 33 may switch between displaying the distant virtual image V1 using a first image correction process S160 and a second image correction process S170 depending on whether predetermined conditions are met, and may display the nearby virtual image V2 using the second image correction process S170 regardless of whether predetermined conditions are met. In other words, the eye position state determination module 508 may include thresholds, table data, calculation formulas, etc., for determining whether each virtual image V is a distant virtual image V1 or a nearby virtual image V2, based on position information of the real object 300 to which the virtual image V acquired from the external sensor 411 is associated, and information regarding the perceptual distance D30 set for the virtual image V based on the position information of the real object 300.

[0114] Furthermore, in some embodiments, if the area in which the head-up display device 20 can display a virtual image V is defined as the virtual image display area VS, then, as shown in Figure 9, the device displays an upper virtual image V60 displayed in an upper area VSα that includes the upper end VSu of the virtual image display area VS as seen from the driver's seat of the vehicle, and a lower virtual image V70 displayed in a lower area VSβ that includes the lower end VSb of the virtual image display area VS and is below the upper area VSα. The processor 33 may switch between displaying the upper virtual image V60 between the first image correction process S160 and the second image correction process S170 depending on whether predetermined conditions are met, and may display the lower virtual image V70 without performing image position correction based on eye position or head position.

[0115] Furthermore, in some embodiments, the head-up display device 20 displays an AR virtual image V60 whose display position changes according to the position of real objects present in the foreground of the vehicle, as shown in Figure 9, and a non-AR virtual image V70 whose display position does not change according to the position of real objects. The processor 33 switches between displaying the AR virtual image V60 between a first image correction process S160 and a second image correction process S170 depending on whether predetermined conditions are met, and displays the non-AR virtual image V70 without performing image position correction based on eye position or head position.

[0116] The operation of the processing steps described above can be carried out by having one or more functional modules of an information processing device, such as a general-purpose processor or an application-specific chip, execute. All of these modules, combinations of these modules, and / or combinations with known hardware that can substitute for their functions are all within the scope of protection of the present invention.

[0117] The functional blocks of the vehicle display system 10 are optionally implemented by hardware, software, or a combination of hardware and software to carry out the principles of the various embodiments described. Those skilled in the art will understand that the functional blocks described in Figure 10 may optionally be combined, or one functional block may be separated into two or more subblocks, to carry out the principles of the embodiments described. Therefore, the description herein optionally supports any possible combination or division of the functional blocks described herein. [Explanation of Symbols]

[0118] 1: Vehicle 2:Projected area 5: Dashboard 6: Road surface 10: Vehicle display system 20:HUD device 21: Light-emitting window 22: Cabinet 24: Light source unit 28: First Actuator 29: Second actuator 30: Display control device 31: I / O Interface 33: Processor 35: Image processing circuit 37: Memory 40:Display device 205: Center 300: Actual object 311: Road surface 313: person 314: Other vehicles 315: Building 401: Vehicle ECU 403: Road Information Database 405: Vehicle position detection unit 407: Operation detection unit 409: Eye position detection unit 411: External vehicle sensor 413: Brightness detection unit 417: Mobile Information Terminal 419: External communication devices 502: Eye position detection module 504: Eye position estimation module 506: Eye position prediction module 508: Eye position state determination module 510: Vehicle status determination module 511: Eye-position tracking image processing module 512: Eye-tracking image processing module 514: Graphics module 516: Light source driving module 518: Actuator drive module 710: Head position Cy1: First correction amount Cy2: Second correction amount Cy3: Third correction amount D10: Image formation distance D30: Perceptual distance FU: Virtual object (perceptual illusion) K:Display light PS:Virtual image display area PT: Target position Px: Eye position (head position) in the left-right direction Py: Eye position (head position) in the vertical direction V: Illusion V60: AR virtual image (upper virtual image) V61: Navigation illusion V62: Navigation illusion V63: Enhanced Illusion V64: Enhanced Illusion V65: Enhanced Illusion V65: POI (Point of Interest) Illusion V70: Non-AR virtual image (lower virtual image) VS: Virtual image display area VSb: Bottom end VSu :Top edge VSα: Upper area VSβ: lower area Vx: Movement speed Vy: Movement speed t :period time ΔPx: Change ΔPy: Change

Claims

1. A display control device (30) that performs display control in a head-up display device (20) which is composed of at least a display unit (40) that displays an image, and a relay optical system (80) that projects the light of the image displayed by the display unit (40) onto a projection member, and which causes the user of the vehicle to see a virtual image of the image superimposed on the foreground, One or more processors (33), Memory (37) and, The system comprises one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33), The aforementioned processor (33) The user's eye position and / or head position in the vertical direction of the vehicle, and the eye position and / or head position in the horizontal direction of the vehicle are obtained. The position of the image displayed on the display unit (40) is corrected based at least on the aforementioned vertical eye position or head position and the aforementioned horizontal eye position or head position. The amount of correction for the position of the image with respect to the amount of change in the vertical eye position or head position is defined as the first correction amount (Cy1). A first image correction process in which the amount of correction of the image position with respect to the amount of change in the eye position or head position in the left-right direction is defined as the first left-right correction amount (Cx1), While correcting the position of the image displayed on the display unit (40) with a second left-right correction amount (Cx2) for the amount of change in the left-right eye position or head position, the second correction amount (Cy2) for the position of the image with respect to the amount of change in the up-down eye position or head position is smaller than the first correction amount (Cy1) for the position of the image with respect to the amount of change in the up-down eye position or head position during the first image correction process, or The system includes a second image correction process which corrects the position of the image displayed on the display unit (40) with respect to the amount of change in the eye position or head position in the left-right direction by the second left-right correction amount (Cx2), while setting the correction amount for the image position with respect to the amount of change in the eye position or head position in the up-down direction to zero. In the second image correction process described above, The ratio of the second correction amount (Cy2) to the first correction amount (Cy1) is, The ratio of the second left-right correction amount (Cx2) to the first left-right correction amount (Cx1) is smaller. It is set so that (Cx2 / Cx1 > Cy2 / Cy1), If any of the eye position, head position, and vehicle state satisfies at least one predetermined condition, The process switches from the first image correction process to the second image correction process and executes it. Display control device (30).

2. The aforementioned processor (33) The eye position or head position in the left-right direction changed continuously in one direction. The changes in the vertical eye position and / or head position and the changes in the horizontal eye position and / or head position are detected, and in this case, the ratio of the change in the vertical eye position or head position to the change in the horizontal eye position or head position is less than a predetermined first threshold, and If a change in the vertical eye position or head position and a change in the horizontal eye position or head position are detected, and at least one of the following conditions is met—that the amount of change in the vertical eye position or head position is less than a predetermined second threshold—then the second image correction process is selected. The display control device (30) according to claim 1.

3. The aforementioned processor (33) If, after the vertical eye position and / or head position and / or the horizontal eye position and / or head position can no longer be obtained, a change in the vertical eye position or head position and a change in the horizontal eye position or head position are detected, the second image correction process is selected. The display control device (30) according to claim 1 or 2.

4. The aforementioned processor (33) In the second image correction process described above, after a predetermined time has elapsed, The position of the image displayed on the display unit (40) is corrected based at least on the vertical eye position or head position and the horizontal eye position or head position, and the system switches to a third image correction process in which a third correction amount (Cy3) for the position of the image with respect to the change in the vertical eye position or head position is smaller than the first correction amount (Cy1) in the first image correction process and larger than the second correction amount (Cy2) in the second image correction process. The display control device (30) according to claim 1.

5. The aforementioned processor (33) In the second image correction process described above, if it is detected that the amount of change in the vertical eye position or head position has become greater than a predetermined third threshold, The position of the image displayed on the display unit (40) is corrected based at least on the vertical eye position or head position and the horizontal eye position or head position, and the system switches to a third image correction process in which a third correction amount (Cy3) for the position of the image with respect to the change in the vertical eye position or head position is smaller than the first correction amount (Cy1) in the first image correction process and larger than the second correction amount (Cy2) in the second image correction process. The display control device (30) according to claim 1.

6. The aforementioned processor (33) In the image correction process described above (third step), The third correction amount (Cy3) changes over time so that it approaches the first correction amount (Cy1) during the first image correction process. The display control device (30) according to claim 4.

7. The head-up display device (20) displays a distant virtual image (V1) perceived at a position a first distance away from a reference point set on the vehicle side, and a nearby virtual image (V2) perceived at a position a second distance shorter than the first distance. The aforementioned processor (33) The distant virtual image (V1) is displayed by switching between the first image correction process and the second image correction process depending on whether predetermined conditions are met. The nearby virtual image (V2) is displayed by the second image correction process, regardless of whether the predetermined conditions are met. The display control device (30) according to claim 1.

8. If the area in which the head-up display device (20) can display the virtual image (V) is defined as the virtual image display area (VS), then the head-up display device (20) displays an upper virtual image (V60) that is displayed in an upper area (VSα) including the upper end (VSu) of the virtual image display area (VS) as seen from the driver's seat of the vehicle, and a lower virtual image (V70) that is displayed in a lower area (VSβ) below the upper area (VSα) and including the lower end (VSb) of the virtual image display area (VS), The aforementioned processor (33) The upper virtual image (V60) is displayed by switching between the first image correction process and the second image correction process depending on whether predetermined conditions are met. The lower virtual image (V70) is displayed without performing positional correction of the image based on the eye position or the head position. The display control device (30) according to claim 1.

9. The head-up display device (20) displays an AR virtual image (V60) whose display position changes according to the position of a real object in the foreground of the vehicle, and a non-AR virtual image (V70) whose display position does not change according to the position of the real object. The aforementioned processor (33) The AR virtual image (V60) is displayed by switching between the first image correction process and the second image correction process depending on whether predetermined conditions are met. The non-AR virtual image (V70) is displayed without performing positional correction of the image based on the eye position or the head position. The display control device (30) according to claim 1.

10. Display unit (40) for displaying images, A relay optical system (80) that projects the light of the image displayed by the display unit (40) onto the projection member, One or more processors (33), Memory (37) and, A head-up display device comprising one or more computer programs stored in the memory (37) and configured to be executed by the one or more processors (33), which causes a vehicle user to view a virtual image of the image superimposed on the foreground, The aforementioned processor (33) The user's eye position and / or head position in the vertical direction of the vehicle, and the eye position and / or head position in the horizontal direction of the vehicle are obtained. The position of the image displayed on the display unit (40) is corrected based at least on the aforementioned vertical eye position or head position and the aforementioned horizontal eye position or head position. The amount of correction for the position of the image with respect to the amount of change in the vertical eye position or head position is defined as the first correction amount (Cy1). A first image correction process in which the amount of correction of the image position with respect to the amount of change in the eye position or head position in the left-right direction is defined as the first left-right correction amount (Cx1), While correcting the position of the image displayed on the display unit (40) with a second left-right correction amount (Cx2) for the amount of change in the left-right eye position or head position, the second correction amount (Cy2) for the position of the image with respect to the amount of change in the up-down eye position or head position is smaller than the first correction amount (Cy1) for the position of the image with respect to the amount of change in the up-down eye position or head position during the first image correction process, or The system includes a second image correction process which corrects the position of the image displayed on the display unit (40) with respect to the amount of change in the eye position or head position in the left-right direction by the second left-right correction amount (Cx2), while setting the correction amount for the image position with respect to the amount of change in the eye position or head position in the up-down direction to zero. In the second image correction process described above, The ratio of the second correction amount (Cy2) to the first correction amount (Cy1) is, The ratio of the second left-right correction amount (Cx2) to the first left-right correction amount (Cx1) is smaller. It is set so that (Cx2 / Cx1 > Cy2 / Cy1), If any of the eye position, head position, and vehicle state satisfies at least one predetermined condition, The process switches from the first image correction process to the second image correction process and executes it. Head-up display device (20).

11. A display control method for a head-up display device (20) comprising at least a display unit (40) that displays an image, and a relay optical system (80) that projects the light of the image displayed by the display unit (40) onto a projection member, wherein the device causes the user of a vehicle to view a virtual image of the image superimposed on the foreground, The acquisition of the user's eye position or head position in the vertical direction of the vehicle, and the eye position or head position in the horizontal direction of the vehicle, The position of the image displayed on the display unit (40) is corrected based at least on the aforementioned vertical eye position or head position and the aforementioned horizontal eye position or head position. The amount of correction for the position of the image with respect to the amount of change in the vertical eye position or head position is defined as the first correction amount (Cy1). A first image correction process in which the amount of correction of the image position with respect to the amount of change in the eye position or head position in the left-right direction is defined as the first left-right correction amount (Cx1), While correcting the position of the image displayed on the display unit (40) with a second left-right correction amount (Cx2) for the amount of change in the left-right eye position or head position, the second correction amount (Cy2) for the position of the image with respect to the amount of change in the up-down eye position or head position is smaller than the first correction amount (Cy1) for the position of the image with respect to the amount of change in the up-down eye position or head position during the first image correction process, or A second image correction process is performed, in which the position of the image displayed on the display unit (40) is corrected by the second left-right correction amount (Cx2) for the amount of change in the left-right eye position or head position, while the correction amount for the image position for the amount of change in the up-down eye position or head position is set to zero, In the second image correction process described above, The ratio of the second correction amount (Cy2) to the first correction amount (Cy1) is, The ratio of the second left-right correction amount (Cx2) to the first left-right correction amount (Cx1) is smaller. It is set so that (Cx2 / Cx1 > Cy2 / Cy1), If any of the eye position, head position, and vehicle state satisfies at least one predetermined condition, This includes switching from the first image correction process to the second image correction process and executing it, Display control method.

Images