Display control device, display device, and display control method

Abstract

To display an image with reduced distortion not only to a primary viewer but also a secondary viewer, the image being visually recognized by the primary viewer and the secondary viewer.SOLUTION: A display control apparatus 159 includes an information acquisition unit 135 which acquires line-of-sight information, and a control unit 160 which changes a correction parameter to be used for correcting image data. The control unit 160 uses, when only a first occupant 7a views an image, a correction parameter suitable for the first occupant, uses, when only a second occupant 7b views the image, a correction parameter suitable for the second occupant, sets, when both first and second occupants views the image, a first region S1 and a second regions S2 in the image and uses a correction parameter WP-a suitable for the first occupant as a correction parameter corresponding to the first region and uses a correction parameter WP-b suitable for the second occupant as a correction parameter corresponding to the second region.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to a display control device, a display device, a display control method, and the like.

Background Art

[0002] It is conceivable that a passenger other than the driver may view an image displayed by a display device mounted on a vehicle (vehicle display device).

[0003] Regarding this point, for example, in

[0004] Patent Document 1, it is described that "··· when a passenger other than the driver is also riding (seated) in the passenger seat and it is desired that the passenger seated in the passenger seat also views the display image composed of vehicle information and the like, in addition to the above-described head-up display device (that is, the head-up display device for the driver's seat), it is conceivable to separately arrange another head-up display device for the passenger seat inside the instrument panel in front of the passenger seated in the passenger seat."

[0004] Further, as an example of a process for correcting distortion of a displayed image, for example, Patent Document 2 describes a warping process (a correction process for previously imparting distortion having characteristics opposite to those of distortion caused by an optical system or the like to image data) in a HUD device.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] It is conceivable that the image displayed by the display device can be seen not only by the primary viewer (the occupant facing the image directly) but also by a secondary viewer (an occupant viewing the image from an oblique angle).

[0007] Traditionally, image distortion correction has been performed assuming only the primary viewer. As a result, the image may appear highly distorted to secondary viewers.

[0008] For example, if a crew member, who is a secondary viewer, views a displayed image from an oblique angle to obtain information, the image will appear significantly distorted to the secondary viewer because it is designed to be viewed from the front by the primary viewer. This can lead to discomfort, annoyance, or difficulty in confirming necessary information.

[0009] The aforementioned Patent Documents 1 and 2 do not mention this problem, nor do they describe any countermeasures.

[0010] One of the objectives of the present invention is to provide a display in which distortion is reduced not only for the primary viewer but also for the secondary viewer when the image can be viewed by both the primary viewer and the secondary viewer.

[0011] Other objects of the present invention will become apparent to those skilled in the art by referring to the embodiments and best embodiments described below, as well as the accompanying drawings. [Means for solving the problem]

[0012] The following are examples of embodiments of the present invention to facilitate understanding of its outline.

[0013] In the first embodiment, the display control device is A display control device that controls the image display of a display device mounted in a vehicle, which allows the occupants of the vehicle to view an image, An information acquisition unit that acquires gaze direction information including at least one of the occupant's gaze direction and the direction of their face, A control unit having a function to change correction parameters used when correcting image data of the image in order to suppress distortion of the image as seen from the occupant, based on the line of sight direction information, It has, When the display device designates the first occupant as the primary observer and the second occupant as the secondary observer, The control unit, If the image is visible only to the first occupant, the correction parameter used shall be a correction parameter suitable for the first occupant. If only the second occupant is viewing the aforementioned image, then a correction parameter suitable for the second occupant shall be used as the correction parameter. If the aforementioned image is visible to the first and second crew members, The image is divided into a first area prioritizing the first occupant and a second area prioritizing the second occupant. As a correction parameter corresponding to the first region, a correction parameter suitable for the first occupant is used. As the correction parameter corresponding to the second region, a correction parameter suitable for the second occupant is used.

[0014] In the first embodiment, if there are occupants (in other words, secondary viewers) who view the displayed image from an oblique angle in the left-right direction (in other words, the width direction of the vehicle), correction parameters are selected to reduce the distortion of the image as seen by these secondary viewers, taking their perspective into consideration. Therefore, it is possible to provide a display with reduced distortion not only for the primary viewer (for example, the viewer directly facing the image) but also for secondary viewers.

[0015] Even when there are multiple occupants in the vehicle, the quality of the displayed image (at least the minimum required image quality) can be ensured for all occupants (viewers). Optimal display can be provided for all viewers, thereby reducing discomfort and inconvenience for each viewer, and ensuring that each occupant can obtain the necessary information.

[0016] For example, assuming there are a first and a second passenger sitting adjacent to each other, the first passenger is viewing an image displayed facing the second passenger obliquely from the left - right direction, while the second passenger may also be viewing the image displayed facing the first passenger obliquely from the left - right direction (in the case of line - of - sight intersection).

[0017] In such a case, in the distortion correction process of the image data of each image, correction parameters suitable for the passenger viewing obliquely (the passenger who is the secondary viewer) are applied.

[0018] Also, it is possible to assume a case where a common display image is being viewed by both the first and second passengers (in the case of line - of - sight convergence or line - of - sight overlap).

[0019] For example, if the second passenger is viewing the image obliquely, considering this point, in the display image, a region (the second region) that suppresses the reduction in visibility as seen from the second passenger is set. In other words, the display image is divided into a first region that prioritizes the first passenger and a second region that prioritizes the second passenger.

[0020] When performing distortion correction on the image data corresponding to the first region, correction parameters suitable for the first passenger are applied, and when performing distortion correction on the image data corresponding to the second region, correction parameters suitable for the second passenger are applied.

[0021] By this process, the image quality of the display image is ensured for both the first and second passengers, the sense of discomfort and annoyance is reduced, and the recognition of the image is improved.

[0022] In a second aspect subordinate to the first aspect, when the width direction of the vehicle is the left - right direction, the control unit, when setting the first and second regions, when the second passenger is located to the left of the first passenger, in the image, the second region is set to the left of the first region. When the second occupant is positioned to the right of the first occupant, the second region may be set to the right of the first region in the image.

[0023] In the second embodiment, for example, if the second occupant is positioned to the left of the first occupant, and both the first and second occupants are viewing, for example, an image directly facing the first occupant (the first image), the second region to which correction parameters suitable for the second occupant are applied is positioned (set) to the left of the first region, corresponding to the seating position of each occupant. If the second occupant is seated to the right of the first occupant, the second region is positioned (set) to the right of the first region.

[0024] For an occupant viewing an image from an oblique angle, the portion of the image closer to them has a greater visual impact than the portion further away. In other words, the portion closer to the occupant can be considered the essential part of the image. Therefore, when an occupant views the image from the left at an oblique angle, the area to which the appropriate correction parameters for that occupant are applied (the second area) is also positioned to the left of the first area, corresponding to the occupant's seating position. This allows for the application of optimal correction parameters to the area that has a greater visual impact on the occupant viewing the image from an oblique angle, effectively improving display quality.

[0025] In a third embodiment dependent on the first embodiment, The control unit, The aforementioned image may be enlarged, and the enlarged image may be divided into the first and second regions.

[0026] In the third embodiment, the image is enlarged, and the first and second regions are set within the enlarged image. For example, when a secondary occupant (secondary observer) views the image from an oblique angle, the area of ​​the second region set within the image (preferably including the main range that the secondary occupant is presumed to see) may be small, or it may be difficult to secure the second region itself, making it difficult for the secondary occupant to see.

[0027] This situation is particularly likely to occur when there is a large lateral distance between the first and second crew members.

[0028] Therefore, when setting the first and second regions, the necessary area (at least the minimum area) of the second region can be reliably secured by, for example, enlarging the image (for example, by enlarging it by a certain magnification in the horizontal direction (or by a magnification set appropriately according to the viewing environment), or by enlarging it by a certain magnification in both the horizontal and vertical directions (or by a magnification set appropriately according to the viewing environment). Thus, the second occupant can reliably see the distortion-corrected image. Even when the image is relatively difficult to see, the visibility of the image is ensured, improving the practicality of the display device.

[0029] In a fourth embodiment dependent on any one of the first to third embodiments, The control unit, If, while the first and second areas are provided, it is detected that the second occupant is no longer visually observing the second area, the second area may be left in place for a predetermined period of time from the time of detection without being erased.

[0030] In the fourth aspect, when a common image is being viewed by multiple crew members, and the direction of the gaze (face orientation) of a crew member who was viewing the image from an oblique angle (a second crew member acting as a secondary viewer) changes, and the gaze concentration (gaze overlap) is resolved (in other words, when there is a change in the number of crew members viewing the image), the display state will be maintained for a predetermined period from the time the change is detected.

[0031] If the area to which correction parameters suitable for the secondary viewer occupant are applied (the second area) is immediately eliminated, then, for example, when the gaze shifts immediately afterward and returns to a state of focused gaze, the correction parameters will be changed repeatedly. This could cause the way the image appears to the occupant viewing it from the front (the first occupant as the primary viewer) to change abruptly in a short period of time, potentially leading to discomfort or annoyance.

[0032] Therefore, until a predetermined period has elapsed, the previous state will be maintained (in other words, the correction parameters suitable for the secondary viewer occupant will continue to be used). This will help to suppress any discomfort or other issues for the occupant viewing the image (the occupant acting as the primary viewer).

[0033] In a fifth embodiment dependent on the fourth embodiment, The control unit, During the predetermined period, the area of ​​the second region may be reduced in stages or continuously.

[0034] In the fourth embodiment, during the predetermined period described above, correction parameters suitable for the second occupant, who is the secondary viewer, are continued to be used, however, the area of ​​the region to which these correction parameters are applied (the second region) is gradually or continuously reduced (shrinked) within the predetermined period. As a result, during the predetermined period, the area of ​​the region to which the correction parameters suitable for the first occupant are applied (the first region) increases little by little, suppressing abrupt changes in how the image appears and suppressing any sense of discomfort.

[0035] In the sixth embodiment, the display device is A display control device according to any one of the first to fifth embodiments, An image data correction unit that corrects the image data of the image using the correction parameters so as to suppress distortion of the image as seen from at least one of the first occupant and the second occupant, A display unit that displays an image based on corrected image data, It has.

[0036] According to the sixth embodiment, if there is an occupant viewing the image from an oblique angle (an occupant acting as a secondary viewer), an image with distortion correction suitable for that occupant can be displayed on the display unit, thereby reducing discomfort for that occupant.

[0037] This makes it possible to realize a high-performance display device that can show images with minimal distortion to both the primary and secondary viewer crew members.

[0038] In a seventh embodiment dependent on the sixth embodiment, The aforementioned display device is The head-up display device may further include an optical system that projects the display light of the image displayed on the display unit onto a projection member provided in the vehicle, and has the function of allowing the first occupant and the second occupant to view a virtual image. Or, The display device may also have a function to display a real image on the display unit as a display panel, and to allow the first and second crew members to visually perceive the real image.

[0039] The seventh aspect clarifies that the display device may be a HUD device or a display device. A high-performance HUD device or display device can be realized that can display images with minimal distortion to both the first and second occupants.

[0040] Furthermore, as the display device, a liquid crystal display (e.g., a TFT liquid crystal display) or a self-emissive display device such as a micro-LED display device may be used.

[0041] In the eighth embodiment, the display control method is: A display control method for controlling the image display of a display device mounted in a vehicle, which allows the occupants of the vehicle to view an image, A first step of acquiring gaze direction information including at least one of the occupant's gaze direction and the orientation of their face, A second step is to obtain correction parameters to be used when correcting the image data of the image in order to suppress distortion of the image as seen from the occupant, based on the line of sight direction information. The process includes a third step of correcting the image data of the image using the acquired correction parameters, In the second step described above, When the crew member who views the aforementioned image from the front and is the primary observer is designated as the first crew member, and the crew member who views the aforementioned image from an oblique angle and is the secondary observer is designated as the second crew member, If only the first occupant is viewing the image, then a correction parameter suitable for the first occupant is obtained as the correction parameter. If only the second occupant is viewing the aforementioned image, then a correction parameter suitable for the second occupant is obtained as the correction parameter. If the aforementioned image is visible to the first and second crew members, The image is divided into a first area prioritizing the first occupant and a second area prioritizing the second occupant. As correction parameters corresponding to the first region, correction parameters suitable for the first occupant are obtained, As correction parameters corresponding to the second region, correction parameters suitable for the second occupant are obtained.

[0042] According to the eighth aspect, if there is an occupant viewing the image from an oblique angle (an occupant acting as a secondary viewer), an image with distortion correction suitable for that occupant can be displayed on the display unit, thereby reducing discomfort for that occupant.

[0043] This makes it possible to display images with minimal distortion to both the primary and secondary viewer crew members.

[0044] Those skilled in the art will readily understand that the embodiments of the present invention illustrated can be further modified without departing from the spirit of the invention. [Brief explanation of the drawing]

[0045] [Figure 1] Figure 1(A) shows a display system in which two display devices (HUD devices) are placed in a vehicle with two seats in the front row, illustrating a state where the gazes of the two occupants intersect. Figure 1(B) shows a state where the gazes are focused. [Figure 2]Figure 2(A) shows an example configuration of a HUD device equipped with a warping correction function, and Figure 2(B) shows the effect of the warping correction process. [Figure 3] Figure 3(A) shows an example of setting warping parameters when eye-line intersection occurs, Figure 3(B) shows an example of setting warping parameters when eye-line intersection occurs, Figures 3(C) to (F) show an example of setting regions (first and second regions) to which different warping parameters are applied within an enlarged image, and Figure 3(G) shows a major configuration example of a display device including a display control device. [Figure 4] Figure 4(A) shows an example of a display system with three HUD devices arranged in a vehicle with three seats in the front row, and Figure 4(B) shows a modified example in which one of the three HUD devices is used to display an image to an occupant seated in the rear seat. [Figure 5] Figure 5(A) shows an example of setting correction parameters when line of sight crossing occurs in the display system of Figure 4(A), and Figure 5(B) shows an example of setting correction parameters when line of sight convergence occurs in the display system of Figure 4(A). [Figure 6] Figures 6(A) to 6(C) show an example of display control when eye focus is relieved. [Figure 7] Figures 7(A) to 7(E) show other examples of display control when eye focus is eliminated. [Figure 8] Figure 8 shows an example of the main components of a display device (HUD device) and an example of the configuration of a display system. [Figure 9] Figure 9 is a flowchart showing an example of the display control procedure. [Modes for carrying out the invention]

[0046] The best embodiments described below are used to facilitate understanding of the present invention. Therefore, those skilled in the art should note that the present invention is not unduly limited by the embodiments described below.

[0047] (First Embodiment) Refer to Figure 1. Figure 1(A) shows a display system in which two display devices (HUD devices) are placed in a vehicle with two seats in the front row, illustrating a state where the gazes of the two occupants intersect. Figure 1(B) shows a state where the gazes are focused.

[0048] In Figure 1, the X direction represents the width direction (or left-right direction) of vehicle 1, the Y direction represents the height direction of vehicle 1, and the Z direction represents the front (front-back direction) of vehicle 1.

[0049] Vehicle 1 is a passenger car with a steering wheel 8 on the right side and two seats in the front row. Inside Vehicle 1, the driver (occupant) 3a is seated in the driver's seat 6, and the passenger (occupant) 3b is seated in the passenger seat 9.

[0050] Driver 3a and passenger 3b are both "occupants." In the following explanation, the terms "first occupant" and "second occupant" may be used. For example, an occupant who is directly facing (or nearly directly facing) the displayed image (display image), in other words, an occupant who is looking at the image from the front, may be called the "first occupant," and an occupant who is looking at the image from an oblique angle to the left or right may be called the "second occupant."

[0051] Furthermore, the meaning of "front" or "directly facing" should be interpreted broadly, and a range that can be considered "front" or "directly facing" may be defined, including some degree of variation.

[0052] Furthermore, the term "crew member" can also be referred to as a "viewer" who sees the image. The viewer who sees the image from the front is called the "primary viewer," and the viewer who sees the image from an oblique angle to the left or right is called the "secondary viewer."

[0053] The first crew member mentioned above is the "primary sight-seeing crew member," and the second crew member is the "secondary sight-seeing crew member."

[0054] In Figure 1(A), the driver 3a and passenger 3b are seated adjacent to each other in the left-right direction. Passenger 3b is positioned to the left of the driver 3a. There is no occupant in the rear seat 11.

[0055] Two display devices (in this case, HUD devices) 100a and 100b are arranged along the width direction (left-right direction: X direction) of the vehicle 1 in the front portion of the vehicle 1, which is the part on the windshield 2 side. HUD device 100a is installed (positioned) inside the dashboard and instrument panel (neither shown) in front of the driver's seat 6. HUD device 100b is installed (positioned) inside the dashboard and instrument panel (neither shown) in front of the passenger seat 9.

[0056] Furthermore, the display device is not limited to a HUD device; a liquid crystal display (e.g., a TFT liquid crystal display) or a self-emissive display device such as a micro-LED display may also be used.

[0057] Furthermore, Figure 1(A) shows a passenger placement detection camera 200 that detects the arrangement of passengers inside the vehicle. The passenger placement detection unit (not shown in Figure 1, reference numeral 202 in Figure 8) analyzes the images captured by this camera 200 to detect the arrangement of passengers inside the vehicle.

[0058] The HUD devices 100a and 100b project image display light onto the windshield 2, allowing the driver 7 and passenger 10 to see the image (virtual image).

[0059] In Figure 1(A), the virtual image V1 is an image (virtual image) directly facing the driver 3a. In other words, with respect to the image (virtual image) V1, the driver 3a is the primary observer among the occupants, and the passenger 3b is the secondary observer among the occupants.

[0060] Furthermore, the virtual image V2 is an image (virtual image) that is directly facing passenger 3b. In other words, with respect to image (virtual image) V2, passenger 3b is the primary observer, and driver 3a is the secondary observer.

[0061] The HUD devices 100a and 100b incorporate a display control device (processor: reference numeral 159 in Figure 8, not shown in Figure 1) that has a function to change correction parameters (warping parameters) for distortion correction.

[0062] As a method for correcting image distortion, for example, a "pre-distortion correction process (warping correction process)" can be employed, which involves pre-applying a distortion with characteristics opposite to the distortion that occurs when the image is displayed to the image data. In the following explanation, this may be simply referred to as "warping process."

[0063] By appropriately switching the warping parameters as correction parameters, the content of distortion correction performed by the warping process can be changed. Specific examples of display control in the display control device will be described later.

[0064] Furthermore, the vehicle 1 is equipped with eye or face direction detection cameras 300a and 300b. Camera 300a captures images of the driver's eyes (pupils) or face. Based on these captured images, for example, a gaze direction detection unit (not shown in Figure 1, reference numeral 130 in Figure 8) performs image processing to detect changes in the position of the viewpoint in the eye box, or changes in the direction of the face in the left-right direction, thereby detecting the direction of the gaze (gaze direction).

[0065] As autonomous driving technology advances, drivers 3a will be able to move their gaze relatively freely. In other words, the degree of freedom in the direction of their gaze will increase.

[0066] Therefore, driver 3 can, for example, see the image (virtual image) V2 displayed diagonally to the left and in front of them, which is directly facing passenger 10 (the direction of the line of sight in this situation is indicated by the dashed arrow in Figure 1(A)).

[0067] Furthermore, passenger 3b can also see the image (virtual image) V1 displayed diagonally to the right and in front of the driver 3a, which is directly facing the driver (the direction of the gaze in this situation is indicated by a dashed arrow in Figure 1(A)).

[0068] In Figure 1(A), a situation occurs where the lines of sight of the two occupants (driver 3a and passenger 3b) intersect as they view an image (virtual image) diagonally in front of them. In the following explanation, this may be referred to as "line of sight intersection."

[0069] In Figure 1(A), solid arrows are also shown for driver 3a and passenger 3b. These arrows indicate the line of sight of driver 3a and passenger 3b when they are looking at images (virtual images) V1 and V2 that are directly facing them.

[0070] Next, refer to Figure 1(B). The arrangement of the display devices in Figure 1(B) (in other words, the configuration of the display system) is the same as in Figure 1(A).

[0071] However, in Figure 1(B), both the driver 3a and the passenger 3b are looking at the image (virtual image) V1. The direction of the driver 3a's line of sight is indicated by a solid arrow, and the direction of the passenger 3b's line of sight is indicated by a dashed arrow.

[0072] In the following explanation, a situation in which multiple people simultaneously view the same image may be referred to as "gaze concentration (or gaze overlap)."

[0073] When performing warping processing, it is necessary to consider whether or not there are crew members viewing the image from an oblique angle, specifically taking into account the occurrence of "eye-line crossing" and "eye-line concentration," in order to use appropriate warping parameters (in other words, select, decide, or acquire them). This point will be discussed later.

[0074] Next, refer to Figure 2. Figure 2(A) shows an example configuration of a HUD device equipped with a warping correction function, and Figure 2(B) shows the effect of the warping correction process.

[0075] The HUD device 100 is mounted on a vehicle (which can be interpreted in a broad sense) 1. The HUD device 100 includes a display unit (for example, a light-transmitting screen or a liquid crystal display device) 101, a reflector (plane mirror) 103, a curved mirror (for example, a concave mirror, and the reflective surface may be a free-form surface) 105 as an optical element for projecting display light, an image generation unit 150, a control unit 160, and a ROM (storage device) 210 having an image conversion table 212 in which warping parameters WP as correction parameters are stored.

[0076] The control unit 160 can be constructed as a functional block included in the display control device (processor: reference numeral 159 in Figure 12). The control unit 160 can selectively retrieve warping parameters WP from the ROM 210. In other words, the control unit 160 has the function of appropriately changing the warping parameters WP to be used based on, for example, information on the presence or absence of occupants in the vehicle 1, information on the arrangement of occupants, and the gaze information (gaze direction information) of each occupant.

[0077] The image displayed on the display unit 101 is projected onto the projection area 5 of the windshield 2, which is the projection target, via the reflector 103 and the curved mirror 105. Reference numeral 4 indicates the projection area of ​​the image.

[0078] Furthermore, the HUD device 100 may be provided with multiple curved mirrors. In addition, a configuration including a functional optical element such as a refractive optical element such as a lens or a diffractive optical element may be adopted in addition to, or in place of, some (or all) of, the mirror (reflective optical element) of this embodiment.

[0079] A portion of the image display light is reflected by the windshield 2 and incident on the viewpoint (eye) A of the driver or other person located inside (or on) the pre-set eye box EB (which is actually three-dimensional, but for the sake of explanation it is depicted as a rectangle of a predetermined area), and by forming an image in front of the vehicle 1, a virtual image V is displayed on a virtual virtual image display surface PS corresponding to the display surface 102 of the display unit 101.

[0080] The image on the display unit 101 is distorted due to the shape of the curved mirror 105 and the shape of the windshield 2. In other words, distortion occurs due to the optical system of the HUD device 100 and the optical components including the windshield 2. To suppress this distortion, a distortion with the opposite characteristics is applied to the image in advance (warping process, or warping image correction process).

[0081] Figure 2(B) shows an example of a virtual image V as seen by the occupants through the windshield 2. In Figure 2(B), the virtual image V, which has a rectangular outline, is given, for example, 5 vertically and 5 horizontally, for a total of 25 reference points (reference pixel points or coordinate points) GD(i,j) (where i and j are variables that can take values ​​from 1 to 5). For each reference point (each coordinate point) in the image (original image), a distortion with characteristics inverse to the distortion that occurs in the virtual image V is pre-applied by warping processing. Therefore, the pre-applied distortion and the distortion that actually occurs cancel each other out, and ideally, a virtual image V without curvature, as shown in Figure 1(B), is displayed. The number of reference points GD(i,j) can be increased as appropriate by interpolation processing, etc.

[0082] Next, refer to Figure 3. Figure 3(A) shows an example of setting warping parameters when eye-line intersection occurs, Figure 3(B) shows an example of setting warping parameters when eye-line intersection occurs, Figures 3(C) to (F) show an example of setting regions (first and second regions) to which different warping parameters are applied within an enlarged image, and Figure 3(G) shows a major configuration example of a display device including a display control device.

[0083] In Figure 3, it is assumed that there are two occupants 7a and 7b inside vehicle 1 (either one may be the driver, or both may be passengers). Furthermore, the displayed images are assumed to be image G100a facing occupant 7a and image G100b facing occupant 7b. Note that images G100a and G100b may be virtual images displayed by a HUD device, or real images displayed by a display device.

[0084] In Figure 3(A), crew member 7a is looking at the image (which may be a virtual image) G100b diagonally forward to the right, and crew member 7b is looking at the image (which may be a virtual image) G100a diagonally forward to the left, resulting in a line-of-sight intersection. The symbols EBa and EBb indicate the eye boxes for crew members 7a and 7b, respectively.

[0085] The eyebox is essentially a virtual three-dimensional space, and the display device (display system) is designed on the premise that a person's eyes (viewpoint) can see the displayed image when they are positioned within the eyebox. In this embodiment, the displayed image consists of a first image viewed by the occupant from the front and a second image viewed by the occupant from an oblique angle. In principle, when the occupant's eyes are positioned within the eyebox, they can see both the first and second images.

[0086] However, if no countermeasures are taken, the degree of distortion when viewing an image from an angle will be greater than the degree of distortion when viewing the image from the front, resulting in a difficult-to-view image. If the distortion increases significantly, it may become difficult to obtain sufficient necessary information.

[0087] Therefore, in Figure 3(A), consideration is also given to the occupant who views the image from an oblique angle, and when correcting the image display control, correction parameters (warping parameters) suitable for the occupant as a secondary viewer are applied.

[0088] In Figure 3(A), for image G100a, only the secondary viewer, crew member 7b (crew member viewing image G100a from an oblique angle), is present. Therefore, when performing distortion correction processing (warping processing) in the display control of G100a, the warping parameter WP-b suitable for crew member 7b is used.

[0089] Regarding image G100b, only the crew member 7a, who is a secondary viewer (a crew member viewing image G100b from an oblique angle), is present. Therefore, when warping is performed in the display control of G100b, the warping parameter WP-a, which is suitable for crew member 7a, is used.

[0090] Here, we will explain the "parameters suitable for the crew." Corresponding to the multiple reference points explained earlier in Figure 2(B), the eyebox (which can be rephrased as a virtual plane located in front of the crew if the eyebox is simplified and drawn in a planar manner) is divided into multiple regions (unit regions: not shown).

[0091] If the crew member's viewpoint (either the right or left eye) is located within a certain unit area, then a correction parameter (warping parameter) corresponding to that unit area is selected (viewpoint position tracking warping process). This warping parameter selected in accordance with the unit area can be considered the "parameter suitable for the crew member" mentioned above.

[0092] However, this is just one example and is not limited to this. Since it may not be possible to set warping parameters for each unit area of ​​the eyebox, in such cases, a predetermined warping parameter corresponding to the crew member's eyebox (a parameter uniquely determined for the eyebox) may be referred to as the "parameter suitable for the crew member."

[0093] Next, refer to Figure 3(B). In Figure 3(B), a focus of gaze is observed. Crew members 7a and 7b are both looking at image G100b. Therefore, when correcting the image data of image G100b, it is necessary to apply two types of correction parameters: one suitable for crew member 7a and another suitable for crew member 7b.

[0094] Therefore, in the example shown in Figure 2(B), image G100b is divided into a first region S1 that prioritizes crew member 7b, who is the primary viewer (the viewer who sees image G100b from the front), and a second region S2 (shown with diagonal lines in the figure) that prioritizes crew member 7a, who is the secondary viewer (the viewer who sees image G100b from an oblique angle). In other words, the first region S1 and the second region S2 are set (arranged) within image G100b.

[0095] Corresponding to this division, the display image (original image) of the display surface 102 of the display unit 100, as shown earlier in Figure 2(A), is divided, and the image data of the image region corresponding to the above-mentioned region S1 is to which the warping parameter WP-b suitable for crew member 7b is applied, and the image data of the image region corresponding to the above-mentioned region S2 is to which the warping parameter WP-a suitable for crew member 7a is applied.

[0096] Thus, when there are occupants (in other words, secondary viewers) who view the displayed image from an oblique angle in the left-right direction (in other words, the width direction of the vehicle), correction parameters are selected to reduce the distortion of the image as seen by these secondary viewers, taking their perspective into consideration. Therefore, it is possible to provide a display with reduced distortion not only for the primary viewer (for example, the viewer directly facing the image) but also for secondary viewers.

[0097] Even when there are multiple occupants in the vehicle, the quality of the displayed image (at least the minimum required image quality) can be ensured for all occupants (viewers). Optimal display can be provided for all viewers, thereby reducing discomfort and inconvenience for each viewer, and ensuring that each occupant can obtain the necessary information.

[0098] In other words, the image quality of the displayed image is ensured for both the crew member viewing the image from the front (first crew member) and the crew member viewing the image from an angle (second crew member), reducing discomfort and annoyance, and improving image recognition.

[0099] Next, we will explain the process of enlarging an image when the viewer's gaze is focused. Refer to Figures 3(C) and (D).

[0100] In Figure 3(C), the roughly square image G100b (the region of this image G100b is designated as S3) is enlarged horizontally (in the vehicle's width direction) to form a horizontally elongated rectangular image G100b', and the region division described earlier is then performed within this enlarged image G100b'. As a result, the enlarged image G100b' is composed of region S3 and region S4.

[0101] In region S3, the warping parameter WP-b, suitable for crew member 7b, is applied. In region S4, the warping parameter WP-a, suitable for crew member 7a, is applied.

[0102] Image enlargement is not strictly necessary, but it is often preferable to perform it. For example, in the example in Figure 3(C), the position of crew member 7a's face is near the right edge of the eye box EBa, and even if crew member 7a moves their eyes to the right, it may be difficult for them to see the entire image G100b before enlargement. A similar difficulty can be expected if there is a large lateral distance between crew member 7a and crew member 7b.

[0103] By performing the image enlargement described above, if the crew member 7a's eyes are inside the eye box EBa, they will be able to see the entire image G100b' (at least the main part) more easily, even when viewing the image G100b' at an angle. In addition, the area of ​​the image region S4, which has been corrected with parameters suitable for crew member 7a, can be increased, further improving the image quality as seen by crew member 7a.

[0104] Furthermore, if focus concentration occurs and two (or more in a broader sense) correction parameters suitable for each crew member are applied, image magnification may be performed unconditionally, or certain conditions may be set, and image magnification may be performed only when those conditions are met.

[0105] For example, if the image is difficult to see for an occupant viewing it at an angle (such as at night or in bad weather), the image may be enlarged.

[0106] In this way, when setting multiple regions in a single image to which different correction parameters are applied, by, for example, enlarging the image (for example, by enlarging it by a constant magnification in the horizontal direction (or by a magnification set appropriately according to the viewing environment), or by enlarging it by a constant magnification in both the horizontal and vertical directions (or by a magnification set appropriately according to the viewing environment), it is possible to ensure that the necessary area (at least a minimum area) of the region to which correction parameters suitable for occupants viewing the image from an oblique angle are applied is ensured. Therefore, occupants viewing the image from an oblique angle can be sure to see an image with distortion corrected. Even when the image is relatively difficult to see, the visibility of the image is ensured, thus improving the practicality of the display device.

[0107] Next, refer to Figure 3(D). In the example in Figure 3(D), a roughly square image G100b (let's call the region of this image G100b S3) is enlarged in the left-right direction (the width direction of the vehicle: horizontal direction) and the height direction of the vehicle (the height direction: vertical direction) to create an enlarged image G100b'' with a similar outline, and the region division described earlier is performed within this enlarged image G100b''. As a result, the enlarged image G100b'' is composed of region S3, region S5, and region S6.

[0108] In Figure 3(D), warping parameters WP-b suitable for crew member 7b are applied to regions S3 and S5, and warping parameters WP-a suitable for crew member 7a are applied to region 6.

[0109] Furthermore, in the examples in Figures 3(B), (C), and (D), the area to which correction parameters suitable for crew member 7a viewing the image from an oblique angle are applied (the shaded area: the second area) is set (positioned) to the left of the area to which correction parameters suitable for crew member 7b viewing the image from the front are applied (the white area: the first area).

[0110] This corresponds to the position of crew member 7b to the left of crew member 7a (in other words, the seating position of each crew member). If crew member 7b is positioned to the right of crew member 7a, the second area is set (positioned) to the right of the first area.

[0111] For an occupant viewing an image from an oblique angle, the portion of the image closer to them has a greater visual impact than the portion further away. In other words, the portion closer to the occupant can be considered the essential part of the image. Therefore, when an occupant views the image from the left at an oblique angle, the area to which the appropriate correction parameters for that occupant are applied (the second area) is also positioned to the left of the first area, corresponding to the occupant's seating position. This allows for the application of optimal correction parameters to the area that has a greater visual impact on an occupant viewing the image from an oblique angle, effectively improving display quality.

[0112] Furthermore, the ratio of the area to which correction parameters suitable for occupants viewing the image from an oblique angle are applied (the area with diagonal lines: the second area) to the area to which correction parameters suitable for occupants viewing the image from the front are applied (the area with white space: the first area) can be appropriately changed depending on the driving conditions and surrounding environment. For example, while the ratio is normally 1:2, if it is determined that the image is particularly difficult to see for occupants viewing from an oblique angle, adaptive display control may be implemented, such as expanding the area of ​​the second area to make the area ratio 1:1.

[0113] Furthermore, the outlines of the first and second regions may be quadrilaterals, as shown in the examples in Figures 3(B) to (D). However, this is not the only option; for example, as shown in Figures 3(E) and (F), the outlines of each region can also be determined by dividing the image diagonally into two.

[0114] In Figure 3(E), a horizontally elongated rectangular image is divided diagonally into two equal parts. Warping parameter WP-a is applied to the lower left region (the area with diagonal lines) S8, and warping parameter WP-b is applied to the upper right region (the area with white outline) S7.

[0115] In Figure 3(F), a horizontally elongated rectangular image is divided diagonally into two equal parts. The warping parameter WP-a is applied to the upper left region (the area with diagonal lines) S10, and the warping parameter WP-b is applied to the lower right region (the area with white outline) S9.

[0116] Next, refer to Figure 3(G). The configuration of the display device shown in Figure 3(G) is the same as (basically identical to) the configuration shown in the lower right of Figure 2. In Figure 3(G), the same reference numerals are used for parts that are common to both Figure 2 and Figure 3(G).

[0117] However, Figure 3(G) clearly shows that the control unit 160 is a component of the display control device 159. Figure 3(G) also clarifies that the display control device 159 includes an information acquisition unit 135 that acquires gaze information (gaze direction information). Based on the acquired gaze information (gaze direction information), the control unit 160 appropriately performs display control as shown in Figures 3(A) to (F).

[0118] (Second embodiment) Next, refer to Figure 4. Figure 4(A) shows an example of a display system with three HUD devices arranged in a vehicle with three seats in the front row, and Figure 4(B) shows a modified example in which one of the three HUD devices is used to display an image to an occupant seated in the rear seat. In the example in Figure 4, the steering wheel 8 is located on the left side of the vehicle.

[0119] In Figure 4(A), three display devices (in this case, HUD devices) 110a, 110b, and 110c are arranged along the width direction (left-right direction: X direction) of the vehicle 1 in the front portion of the vehicle 1, which is the part on the windshield 2 side.

[0120] A HUD device 110a is positioned in a location corresponding to the driver's seat 16 (facing the driver directly), a HUD device 110b is positioned in a location corresponding to the adjacent passenger seat (center passenger seat) 18, and a HUD device 110c is positioned in a location corresponding to the adjacent passenger seat (rightmost passenger seat) 20.

[0121] Furthermore, cameras 310a to 310c are provided to detect the direction of each person's eyes or face, corresponding to the driver and each passenger.

[0122] Driver 3e is seated in the driver's seat 16, passenger 3f is seated in the passenger seat 18, and passenger 3g is seated in the passenger seat 20. There is no passenger in the rear seat 21.

[0123] Furthermore, the presence (seated) of driver 3e, the presence (seated) of passengers 3f and 3g, and the absence of occupants in the rear seat 21 can be determined by image analysis of the images captured by the camera (occupant placement detection camera) 200.

[0124] Each of the HUD devices (first to third HUD devices) 110a to 110c projects an image display light onto the windshield 2, allowing the driver 3e and passengers 3f and 3g to see the image (virtual image).

[0125] The HUD device 110a is a display device that has the driver 3e as the primary viewer and the passengers 3f and 3g as secondary viewers.

[0126] HUD device 110b is a display device that has passenger 3f as the primary viewer and the driver 3e or passenger 3g as secondary viewers. HUD device 110c is a display device that has passenger 3g as the primary viewer and passenger 3f or driver 3e as secondary viewers.

[0127] Each HUD device 110a to 110c (the control unit 160 of the display control device) has information that can identify the primary viewer of each HUD device pre-programmed and set.

[0128] For example, for display device 110a, one of the three display devices 110a to 110c, which is installed in the position corresponding to driver 3e, if "1" is entered on the initial setup screen, it is set that driver 3e is the primary observer.

[0129] Similarly, for the display device 110b located in the position corresponding to the central passenger seat 18, entering "2" on the initial setup screen sets the primary viewer to be the passenger 3f located in the front row, next to the driver's seat 16 (in other words, seated in the central passenger seat 18). Similarly, for the display device 110c, entering "3" on the initial setup screen sets the primary viewer to be the passenger 19 seated in the front row, on the far right passenger seat 20.

[0130] In this way, after setting all the display devices, if information such as the presence or absence of occupants in the vehicle 1, occupant placement information, and gaze information (gaze direction information) is input to each display device 110a to 110c, the control unit 160 of each display device 110a to 110c will determine whether or not the correction parameters have been changed, and the correction parameters will be automatically optimized. Therefore, a high-performance display system can be realized that is easy to use and can display images with little distortion for both the driver and occupants.

[0131] In Figure 4(B), two display devices (in this case, HUD devices) 100a and 100b are positioned in a passenger car with two seats in the front row (this is the same as in Figures 1(A) and (B)), and an additional display device 100c is placed between the driver's seat 6 and the passenger seat 9. In addition, a camera 300c is added to detect the direction of the eyes or face of the occupant 3c seated in the center of the rear seat 11.

[0132] In the embodiment described above, the viewer of the image was defined as an occupant seated in the front row of the vehicle 1. However, as shown in the example in Figure 4(B), an occupant (passenger) 3d seated in the rear seat 11, for example, in the central position (more broadly, in a position where the display by the display device 100d can be seen), may also be added as a viewer. The display control operation is basically the same as in Figure 4(A).

[0133] The example in Figure 4(A) will be explained in detail below. Refer to Figure 5. Figure 5(A) shows an example of setting correction parameters when line of sight crossing occurs in the display system of Figure 4(A), and Figure 5(B) shows an example of setting correction parameters when line of sight convergence occurs in the display system of Figure 4(A).

[0134] First, refer to Figure 5(A). In Figure 5(A), three eye boxes EB3e, EB3f, and EB3g are set up on the near side. Each of the eye boxes EB3e, EB3f, and EB3g corresponds to the driver 3e, passenger (occupant) 3f, and passenger (occupant) 3g, respectively.

[0135] In front of vehicle 1, the virtual image display surface PS is divided into three sections horizontally, and three rectangular image display areas are provided to correspond to each occupant 3e to 3g. Virtual images (pictures) V10 to V12 are displayed in each image display area.

[0136] The arrows in the diagram indicate the direction of gaze for the driver (occupant) 3e and passengers (occupants) 3f and 3g. The driver (occupant) 3e is looking at the virtual image V10 directly in front of him, the central passenger (occupant) 3f is looking at the virtual image V12 located at the far right from an oblique angle, and the far right passenger (occupant) 3g is looking at the central virtual image V11 from an oblique angle. Therefore, there is a line of sight intersection between passengers (occupants) 3f and 3g.

[0137] Therefore, the same display control as in the example in Figure 3(A) described earlier can be implemented for the virtual images V11 and V12. In other words, when correcting the distortion of the image data corresponding to the central virtual image V11, the warping parameter WP-3g suitable for the passenger (crew member) 3g on the far right is applied, and when correcting the distortion of the image data corresponding to the virtual image V12 on the far right, the warping parameter WP-3f suitable for the central passenger (crew member) 3f is applied.

[0138] Next, refer to Figure 5(B). In Figure 5(B), the driver (occupant) 3e is looking at the virtual image V10 on the far left, the passenger (occupant) 3f in the center is looking at the virtual image V12 located on the far right from an oblique angle, and the passenger (occupant) 3g on the far right is looking at the virtual image V12 on the far right from the front. Therefore, with respect to the virtual image V12, passengers (occupants) 3f and 3g are focusing their gaze.

[0139] In this case, no crew member is looking at the central virtual image V11. However, considering the normal situation where each crew member is facing forward, the central crew member 3f, who is directly facing the virtual image V11, will be the primary observer. Taking this into consideration, warping parameters suitable for the central crew member 3f are applied to correct the image data corresponding to the central virtual image V11.

[0140] Furthermore, the virtual image V12 at the far right can be controlled in the same way as in the examples shown in Figures 3(B) to (F) described earlier. In Figure 5(B), regions S10 and S11 are set in the virtual image V12. Region S11 is positioned to the left of region S10.

[0141] For distortion correction of image data corresponding to region S10, the warping parameter WP-3g, suitable for passenger (crew member) 3g who views the virtual image V12 from the front, is applied. For distortion correction of image data corresponding to region S11, the warping parameter WP-3f, suitable for passenger (crew member) 3f who views the virtual image V12 from an oblique angle, is applied.

[0142] In this way, even when there are multiple occupants in the vehicle, the quality of the displayed image (at least the minimum required image quality) can be ensured for all occupants (viewers). Optimal display can be provided for all viewers, thereby reducing discomfort and inconvenience for each viewer, and allowing each occupant to obtain the necessary information.

[0143] In other words, the image quality of the displayed image is ensured for both the crew member viewing the image from the front (first crew member) and the crew member viewing the image from an angle (second crew member), reducing discomfort and annoyance, and improving image recognition.

[0144] Next, refer to Figure 6. Figures 6(A) to 6(C) show an example of display control when eye focus is eliminated.

[0145] Figure 6(A) is a reproduction of Figure 5(C) shown earlier. Next, in Figure 6(B), the direction of the gaze of passenger 3g on the far right is changed, and the state changes from a state of concentrated gaze to a state where each occupant sees an image (virtual image) directly in front of them. In other words, there are no occupants who are viewing the image from an oblique angle, and the state of concentrated gaze is resolved. However, even if a change in the state is detected, the correction parameters of the displayed image are not immediately changed, and the state before the detection is maintained for a predetermined period. The display mode of the image (virtual image) in Figure 6(B) is the same as in Figure 6(A), and there is no change.

[0146] This is because if the region (second region) S11, to which correction parameters suitable for the occupant viewing the image from an oblique angle (the occupant as a secondary viewer) were applied is immediately eliminated, then, for example, immediately afterward, when that occupant's gaze shifts and returns to a focused state, the correction parameters will be changed (switched) repeatedly. This could cause the way the image appears to the occupant viewing it from the front (the occupant as the primary viewer) to change abruptly in a short period of time, potentially leading to discomfort or annoyance.

[0147] If the predetermined period has elapsed and the state of gaze concentration has been resolved, then, as shown in Figure 6(C), region S11 disappears, and only the region to which correction parameters suitable for the occupant 3g viewing the image from the front are applied remains.

[0148] Thus, when a common image is being viewed by multiple crew members (overlapping gazes), and the direction of the gaze (face orientation) of a crew member who was viewing the image from an oblique angle (a second crew member acting as a secondary viewer) changes, and the overlapping gaze is resolved (in other words, when there is a change in the number of crew members viewing the image), the display state will be maintained for a predetermined period from the time the change is detected.

[0149] If the area to which the correction parameters suitable for the secondary viewer occupant are applied (the second area) is immediately eliminated, then, for example, when the gaze shifts immediately afterward and returns to a state of focused gaze, the correction parameters will be changed repeatedly. This could cause the way the image appears to the occupant viewing that image (the first occupant as the primary viewer) changes abruptly in a short period of time, potentially leading to discomfort or annoyance.

[0150] Therefore, until a predetermined period has elapsed, the previous state will be maintained (in other words, the correction parameters suitable for the secondary viewer occupant will continue to be used). This will help to suppress any discomfort or annoyance the occupant viewing the image (the primary viewer occupant) may experience.

[0151] Next, refer to Figure 7. Figures 7(A) to 7(E) show other examples of display control when gaze concentration is eliminated. In the example in Figure 7, the area to which correction parameters suitable for occupants who were viewing the image from an oblique angle are applied is gradually or continuously reduced during the predetermined period described above.

[0152] Figure 7(A) is a reproduction of Figure 6(B) explained earlier. As explained earlier in Figure 6(B), even when it is detected that the gaze concentration state has been resolved (in other words, that there are no longer any occupants viewing the image from an oblique angle), the region (the second region) to which correction parameters suitable for occupants viewing the image from an oblique angle are applied remains for a predetermined period of time.

[0153] Here, as shown in Figures 7(B) to 7(D), the area of ​​the region (second region) S11 is gradually or continuously reduced over time during the predetermined period. In Figure 7(C), the area of ​​region S11 is smaller than in Figure 7(B), and in Figure 7(D), the area of ​​region S11 is even smaller.

[0154] Then, after a predetermined period of time has elapsed, the display will be in the state shown in Figure 7(E). Note that Figure 7(E) is a reproduction of Figure 6(C) from earlier.

[0155] Thus, in the example shown in Figure 7, the area (second area) to which correction parameters suitable for occupants viewing the image from an oblique angle are applied during the predetermined period described above is gradually or continuously reduced (shrinked).

[0156] As a result, over a predetermined period, the area of ​​the region (the first region) to which correction parameters suitable for the occupant viewing the image from the front (the first occupant as the primary viewer) are applied gradually increases. Therefore, abrupt changes in how the image appears are suppressed, and discomfort or annoyance is reduced.

[0157] Next, refer to Figure 8. Figure 8 shows an example of the main components of a display device (HUD device) and an example of the display system configuration. The main components of the display device were previously shown in Figures 2 and 3(G), but Figure 8 shows a more detailed configuration. In the example in Figure 8, three HUD devices are used as the display device (corresponding to the example in Figure 4(A)).

[0158] In Figure 8, the display system includes a communication unit 112, an ECU (electronic control unit) 120, eye or face orientation detection cameras 310a to 10c, gaze direction detection unit 130, occupant placement detection camera (in-vehicle imaging camera) 200, occupant placement detection unit 202 for detecting occupants (including the driver), and first to third HUD devices 110a, 110b, and 110c. Each HUD device has the same configuration. The main components of the third HUD device 110c will be described below.

[0159] The third HUD device 110c includes a display unit 101, a display control unit 145, an image generation unit 150 (including an image data correction unit 155), at least one processor (display control device) 159 (including a control unit 160, which has a warping correction control unit 163), a database 170 in which navigation data and display image data are stored, and a ROM 210 as a storage device.

[0160] The ROM 210 has an image conversion table 212. The image conversion table 212 has warping parameters WP as correction parameters for distortion correction. As explained earlier, the warping parameters WP include parameters for occupants who view the image from the front (in other words, main parameters) and parameters for occupants who view the image from an oblique angle in the left-right direction (in other words, sub-parameters).

[0161] The processor (display control device) 159 (specifically its control unit 160) has a function to appropriately change the warping parameters. The image data correction unit 155 corrects the image data of the display image using the warping parameters to suppress distortion of the image as seen by the occupants. The display unit 101 displays an image based on the corrected image data.

[0162] With this configuration, if there are occupants viewing the image from an oblique angle, appropriate correction parameters are selected taking those occupants into consideration, and an image with reduced distortion due to these correction parameters can be displayed on the display unit 101. As a result, discomfort and other issues are reduced, and the display quality of the image is improved. This realizes a high-performance display device and display system that can display an image with minimal distortion to all occupants.

[0163] (Third embodiment) Next, refer to Figure 9. Figure 9 is a flowchart showing an example of a display control procedure.

[0164] Step S1 detects the presence and arrangement of occupants (in other words, passengers including the driver and other passengers).

[0165] In step S2, the direction of each crew member's (each passenger's) gaze is detected.

[0166] In step S3, each display device determines whether or not there is an occupant viewing the image from an oblique angle. If the result is N, the process proceeds to step S4.

[0167] In step S4, the image is corrected using warping parameters suitable for occupants viewing the image from the front, and the corrected image is displayed.

[0168] If the result in step S3 is Y, the process proceeds to step S5. In step S5, the warping parameters are determined considering the crew viewing the image from an oblique angle, the image data is corrected, and the corrected image is displayed. In other words, for crew members viewing the image from an oblique angle, the image is displayed (presented) that has been corrected using warping parameters appropriate for that crew member.

[0169] In step S6, it is determined whether the situation of viewing the image from an oblique angle has been resolved. In other words, it is determined whether there are no occupants viewing the image from an oblique angle. If the result is N, the determination process continues; if the result is Y, the process proceeds to step S7.

[0170] In step S7, the warping parameters are updated to suit the state after the oblique image viewing has been resolved, the image is corrected using the updated parameters, and the corrected image is displayed.

[0171] However, in step S7, when processing is performed for the case where the gaze concentration is relieved, processing may be performed to maintain the previous state for a predetermined period. Also, during that predetermined period, processing may be performed to gradually or continuously reduce the application area of ​​parameters suitable for occupants viewing the image from an oblique angle.

[0172] In step S8, it is determined whether or not to terminate the image display. If the result is N, the image display control is terminated. If the result is Y, the process proceeds to step S2.

[0173] As described above, according to the present invention, when an image can be viewed by both a primary viewer and a secondary viewer, it is possible to provide a display with reduced distortion not only for the primary viewer but also for the secondary viewer.

[0174] The present invention can be used in either a monocular type HUD device, which emits the same image display light to each eye, or a parallax type HUD device, which emits images with parallax to each eye.

[0175] Furthermore, various display devices such as liquid crystal displays, micro-LED displays, and other types of display devices can be used as the display device.

[0176] Furthermore, in this specification, the term "vehicle" can be interpreted broadly as a means of transportation. Similarly, terms related to navigation (e.g., signs, etc.) shall also be interpreted broadly, taking into consideration, for example, broad navigation information useful for the operation of a vehicle. In addition, HUD devices shall include those used as simulators (e.g., aircraft simulators or simulators used as game devices).

[0177] The present invention is not limited to the exemplary embodiments described above, and those skilled in the art will be able to easily modify the exemplary embodiments described above to the extent included in the claims. [Explanation of Symbols]

[0178] 1...Vehicle (own vehicle), 2...Projected element (reflective and translucent element, windshield, etc.), 3a...Occupant (driver), 3b...Passenger (occupant), 3g~3e...Occupant (including driver and passenger), 4...Image projection area, 6...Driver's seat, 7a, 7b...Occupant (including driver and passenger), 8...Steering wheel, 9...Passenger seat, 11...Rear seat, 100 (100a~100c)...HUD device (display device in a broad sense), 110 (110a~110c)...HUD device (display device in a broad sense), 101...Display unit (LCD panel, etc.), 102...Image display surface of the display unit, 112...Communication unit, 120...ECU (Electronic Control Unit), 130...Direction of gaze Detection unit, 135... Information acquisition unit, 145... Display control unit, 150... Image generation unit, 155... Image data correction unit, 159... Processor (display control device), 160... Control unit, 163... Warping correction control unit, 170... Database, 200... Crew position detection camera, 202... Crew position detection unit, 210... Storage device (ROM), 212... Image conversion table, 300 (300a~300c), 310 (310a~310c)... Eye or face orientation detection camera, WP... Warping parameter, G100a, G100b, G100b', G100b''... Image (virtual or real image), EB (EBa, EBb)... Eye box, V... Virtual image.

Claims

1. A display control device that controls the image display of a display device mounted in a vehicle, which allows the occupants of the vehicle to view an image, An information acquisition unit that acquires gaze direction information including at least one of the occupant's gaze direction and the direction of their face, A control unit having a function to change correction parameters used when correcting image data of the image in order to suppress distortion of the image as seen from the occupant, based on the line of sight direction information, It has, When the crew member who views the aforementioned image from the front and is the primary observer is designated as the first crew member, and the crew member who views the aforementioned image from an oblique angle and is the secondary observer is designated as the second crew member, The control unit, If only the first occupant is viewing the image, the correction parameter used shall be a correction parameter suitable for the first occupant. If only the second occupant is viewing the aforementioned image, then a correction parameter suitable for the second occupant shall be used as the correction parameter. If the aforementioned image is visible to the first and second crew members, The image is divided into a first area that prioritizes the first occupant and a second area that prioritizes the second occupant. As correction parameters corresponding to the first region, correction parameters suitable for the first occupant are used. As correction parameters corresponding to the second region, correction parameters suitable for the second occupant are used. The second region is set relative to the first region on the side corresponding to the seating position of the second occupant. The first region and the second region are, The image is divided by a boundary along the left-right direction, or by a diagonal line that divides the image in two. Display control device.

2. A display control device that controls the image display of a display device mounted in a vehicle, which allows the occupants of the vehicle to view an image, An information acquisition unit that acquires gaze direction information including at least one of the occupant's gaze direction and the direction of their face, A control unit having a function to change correction parameters used when correcting image data of the image in order to suppress distortion of the image as seen from the occupant, based on the line of sight direction information, It has, When the crew member who views the aforementioned image from the front and is the primary observer is designated as the first crew member, and the crew member who views the aforementioned image from an oblique angle and is the secondary observer is designated as the second crew member, The control unit, If only the first occupant is viewing the image, the correction parameter used shall be a correction parameter suitable for the first occupant. If only the second occupant is viewing the aforementioned image, then a correction parameter suitable for the second occupant shall be used as the correction parameter. If the aforementioned image is visible to the first and second crew members, The image is divided into a first area that prioritizes the first occupant and a second area that prioritizes the second occupant. As correction parameters corresponding to the first region, correction parameters suitable for the first occupant are used. As correction parameters corresponding to the second region, correction parameters suitable for the second occupant are used. When the width direction of the vehicle is defined as the left-right direction, The control unit, When setting the first and second regions, When the second occupant is positioned to the left of the first occupant, the second region is set to the left of the first region in the image. When the second occupant is positioned to the right of the first occupant, the second region is set to the right of the first region in the image. Display control device.

3. The control unit, The aforementioned image is enlarged, and the enlarged image is divided into the first and second regions. The display control device according to claim 1.

4. The control unit, When the first and second regions are provided, and it is detected that the second occupant is no longer visually observing the second region, the second region is left in place for a predetermined period of time from the moment it is detected that the second occupant is no longer visually observing the second region. A display control device according to any one of claims 1 to 3.

5. The control unit, During the predetermined period, the area of ​​the second region is reduced in stages or continuously. The display control device according to claim 4.

6. A display control device according to any one of claims 1 to 5, An image data correction unit that corrects the image data of the image using the correction parameters so as to suppress distortion of the image as seen from at least one of the first occupant and the second occupant, A display unit that displays an image based on corrected image data, A display device.

7. The aforementioned display device is The head-up display device further includes an optical system that projects the display light of the image displayed on the display unit onto a projection member provided in the vehicle, and has the function of allowing the first occupant and the second occupant to view a virtual image. Or, This display device has the function of displaying a real image on the display unit, which serves as a display panel, and allowing the first and second occupants to visually perceive the real image. The display device according to claim 6.

8. A display control method for controlling the image display of a display device mounted in a vehicle, which allows the occupants of the vehicle to view an image, A first step of acquiring gaze direction information including at least one of the occupant's gaze direction and the direction of their face, A second step is to obtain correction parameters to be used when correcting the image data of the image in order to suppress distortion of the image as seen from the occupant, based on the line of sight direction information. The process includes a third step of correcting the image data of the image using the acquired correction parameters, In the second step described above, When the crew member who views the aforementioned image from the front and is the primary observer is designated as the first crew member, and the crew member who views the aforementioned image from an oblique angle and is the secondary observer is designated as the second crew member, If only the first occupant is viewing the image, then a correction parameter suitable for the first occupant is obtained as the correction parameter. If only the second occupant is viewing the aforementioned image, then a correction parameter suitable for the second occupant is obtained as the correction parameter. If the aforementioned image is visible to the first and second crew members, The image is divided into a first area that prioritizes the first occupant and a second area that prioritizes the second occupant. As correction parameters corresponding to the first region, correction parameters suitable for the first occupant are obtained, As correction parameters corresponding to the second region, correction parameters suitable for the second occupant are obtained, The second region is set relative to the first region on the side corresponding to the seating position of the second occupant. The first region and the second region are, The image is divided by a boundary along the left-right direction, or by a diagonal line that divides the image in two. Display control method.

Images