Image display device and image display method

Abstract

To provide an image display device which can prevent reduction of the concentration of a user due to display of an image.SOLUTION: The image display device includes: an image display unit 110 for displaying a display image in the screen of a display device located in a vehicle; a line-of-sight detection unit 120 for detecting the direction of the line of sight of a user in the vehicle; an acceptance degree estimation unit 130 for estimating an acceptance degree showing the degree with which the user in the vehicle accepts display of the display image in the screen. The image display unit 110 displays the display image in the screen so that a vision blocking degree showing the degree of blocking the vision of the user when the user in the vehicle is looking at the direction of the line of sight is lower when the acceptance degree is lower than when the acceptance degree is higher.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display method. [Background technology]

[0002] Conventionally, advertising systems comprising an advertising server and a vehicle have been known (see, for example, Patent Document 1). In the advertising system described in Patent Document 1, the advertising server includes an advertising information storage means for storing advertising information, which is the content of the advertisement to be presented, in association with advertising space information including the user's load status to which the advertisement can be presented, and an advertising information transmission means for transmitting the advertising information and advertising space information to the vehicle. The vehicle includes multiple sensors, an advertising information receiving means for receiving the advertising information and advertising space information from the advertising server, a load determination means for determining the user's load from the information acquired from the multiple sensors, and an advertising presentation means for identifying advertising space information corresponding to the user's load and presenting advertising information associated with the advertising space information. The advertising space is subject to restrictions such as the format of provision in consideration of the driver's load status. When the driver's load is high, the advertising presentation means reduces the amount of information presented by imposing restrictions on the number of characters, character size, color scheme, etc. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-45169 Summary of the Invention [Problem to be solved by the invention]

[0004] In an advertising system, even if the amount of information presented is reduced when the burden on the user is heavy, the user's concentration may decrease if the presented information is displayed in a manner that obstructs the user's view.

[0005] The problem to be solved by the present invention is to provide an image display device and an image display method that can prevent a decrease in a user's concentration due to image display. [Means for solving the problem]

[0006] The present invention detects the gaze direction of a user in a vehicle, estimates an acceptance level indicating a degree to which the user in the vehicle accepts the display of an image on a screen, While the in-car user is looking at the displayed image, The above problem is solved by displaying the display image on the screen so that the visibility obstruction level, which indicates the degree of obstruction to the visibility when the user in the vehicle looks in the line of sight, is lower when the acceptance level is low compared to when the acceptance level is high. [Effects of the Invention]

[0007] According to the present invention, it is possible to prevent a decrease in the user's concentration due to image display. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram of a vehicle according to this embodiment. [Figure 2] FIG. 2 is a conceptual diagram of an image displayed on the screen of the AR device. [Figure 3] Figure 3 is a conceptual diagram showing the driver's field of vision. [Figure 4] FIG. 4 is a flowchart showing an example of the procedure of the image display method in the image display device of this embodiment. [Figure 5] FIG. 5 is a block diagram of a vehicle according to another embodiment of the present invention. [Figure 6] FIG. 6 is a flowchart showing an example of the procedure of an image display method in an image display device according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] First Embodiment An embodiment of an image display device according to the present invention will be described with reference to the accompanying drawings. FIG. 1 is a block diagram illustrating an example of the configuration of a vehicle including an image display device according to the present embodiment. As illustrated in FIG. 1, the vehicle 1 includes an in-vehicle device 2, an in-vehicle camera 3, an AR device 4, and a controller 100. Note that the vehicle 1 is not limited to the configuration illustrated in FIG. 1 and may also include, for example, driving devices such as an engine and a motor, auxiliary devices such as a navigation system, and a battery. The image display device according to the present embodiment is a device including the controller 100, and displays a virtual image such as an avatar to a user in the vehicle, providing the user with a virtual space in which the user feels as if they are riding with the avatar. The image display device also detects the user's line of sight and restricts the display of the avatar to prevent the display of the avatar from obstructing the user's view. In the following description, the user in the vehicle will be described as a driver. The embodiment will also be described assuming that the display image displayed in the virtual space includes a virtual image such as an avatar.

[0010] The in-vehicle devices 2 are devices used to detect the driving state of the vehicle, and are, for example, steering sensors, accelerator position sensors, brake pressure sensors, direction indicators, sonar, and other objective sensors.

[0011] The in-vehicle camera 3 is a device that captures images of the inside of the vehicle, capturing images of the user's face and eye movements so that the driver's line of sight can be detected at least. Note that although the in-vehicle camera 3 is also one of the in-vehicle devices, for the sake of explanation, in this embodiment it will be described as a separate device.

[0012] The AR device 4 is a display device located inside the vehicle, worn on the driver's head, and displays a virtual space to the driver. The image of the virtual space is an image in which a human-like avatar is superimposed on the real image seen by the driver, and is an image that virtually extends the real space by superimposing virtual visual information on the real scenery outside and / or inside the vehicle. The avatar is represented by a user or an animated character, for example, an image of a person representing a remote user. The AR device 4 also outputs posture information indicated by the coordinate position, tilt, etc. of the AR device 4 to the controller 100.

[0013] FIG. 2 shows an image viewed by the driver via the AR device 4. When the driver is wearing the AR device 4 and facing forward while the vehicle is moving, the AR device 4 displays an actual image seen from the driver's front, without displaying an avatar. On the other hand, when the driver is wearing the AR device 4 and facing the passenger seat while the vehicle is stopped, for example, the AR device 4 displays a virtual image (augmented image) including the avatar 5 in addition to the actual scenery inside and outside the vehicle (see FIG. 2). The AR device 4 may also be connected to a server via in-vehicle communication under the control of the controller 100, and the server is connected to a remote user's operation terminal in a communicable state. The remote user can move the avatar 5 by operating the operation terminal, and the driver can also see the movement of the avatar 5 in FIG. 2. The avatar 5 may represent a remote user or a character resembling, for example, a tour guide.

[0014] The controller 100 acquires vehicle information from the in-vehicle device 2, acquires captured images from the in-vehicle camera 3, and generates a display image to be displayed on the display screen of the AR device 4. When an avatar (virtual image) is included in the display screen, the display image is an image in which the virtual image is superimposed on a real image. When displaying a virtual image on the display screen of the AR device 4, the controller 100 displays the display image on the display screen of the AR device 4 based on the vehicle information and / or the captured image so that the virtual image does not obstruct the driver's field of vision. The controller 100 includes at least an image display unit 110, a gaze detection unit 120, and an acceptance estimation unit 130 as functional blocks. Each functional block executes its function through cooperation between software and hardware. In this embodiment, the functions of the controller 100 are divided into three blocks and the functions of each functional block are described. However, the functions of the controller 100 do not necessarily have to be divided into three blocks and may be divided into two or less functional blocks, or four or more functional blocks.

[0015] The image display unit 110 displays a display image on the screen of the AR device 4. The image display unit 110 includes a virtual image generation unit 111 and a visibility obstruction degree estimation unit 112. The virtual image generation unit 111 generates a virtual image to be displayed on the screen of the AR device 4. The virtual image generation unit 111 acquires a command to display the virtual image based on the driver's operation, etc. The virtual image generation unit 111 transmits and receives signals to and from the AR device 4 and identifies the line of sight and / or field of view of the driver via the AR device 4. When the virtual image generation unit 111 displays an avatar representing a remote user as a virtual image, it acquires motion information from outside the vehicle to create the movement of the avatar. Then, the virtual image generation unit 111 generates a virtual image according to the identified line of sight and / or field of view. When adding motion to the virtual image, the virtual image generation unit 111 generates the virtual image using the motion information acquired from outside the vehicle. Note that the operation information does not necessarily need to be obtained from outside the vehicle. For example, an animation that gives the avatar movement through a conversation between the driver and the avatar may be added in advance, and the avatar may then move in response to the conversation with the driver.

[0016] The visibility obstruction degree estimation unit 112 acquires information indicating the driver's line of sight from the line of sight detection unit 120 (described later), and estimates the visibility obstruction degree based on the acquired information. The visibility obstruction degree is a parameter (index) that indicates the degree to which the driver's view is obstructed when looking in the line of sight direction. In other words, when a virtual image is superimposed on a real image (an image of the actual scenery inside and outside the vehicle) and displayed to a user who is viewing the real image, the visibility obstruction degree represents the degree to which the user's view is obstructed or the degree to which the user's view is restricted by the virtual image being displayed in and / or around the user's field of view.

[0017] Here, the degree of visibility obstruction will be explained with reference to Fig. 3. Fig. 3 is a conceptual diagram of the visual field range centered on the driver's gaze point. In Fig. 3, point O is the driver's gaze point. For example, as an example of a driving scene, when a vehicle is traveling just before an intersection and a pedestrian is walking on the crosswalk in front of the vehicle (hereinafter referred to as a "scene with pedestrian"), the driver will focus their gaze on the pedestrian. In this case, the position of the pedestrian in the driver's visual field becomes the driver's gaze point.

[0018] In FIG. 3, region P indicates the range of viewing angles from 0 to 50 degrees, region Q indicates the range of viewing angles from 50 to 100 degrees, and region R indicates the range of viewing angles from 100 to 120 degrees. The viewing angles defining regions P to R can be set to any angle. Generally, the human viewing angle is approximately 100 degrees, with region P corresponding to the effective field of vision and region Q corresponding to the peripheral field of vision. For example, in the driving scenario described above, if the driver is looking ahead at the vehicle via AR device 4 and a virtual image (avatar) is displayed within region P, it will interfere with driving. Furthermore, if a virtual image (avatar) is displayed within region Q, it may decrease the driver's concentration on driving or increase fatigue while driving. In the scenario described above where a pedestrian is present, if the driver is facing forward and the avatar is displayed in the passenger seat, the avatar will be displayed in the driver's peripheral field of vision (region Q), which may decrease the driver's concentration on driving. That is, if a virtual image is displayed within the range of area P or area Q, it will obstruct the driver's field of vision, and the degree of obstruction to the field of vision will increase.

[0019] On the other hand, since region R is outside the driver's field of view, even if a virtual image is displayed in region R, the virtual image is unlikely to obstruct the driver's field of view, resulting in a low degree of field of view obstruction. In this manner, the degree of field of view obstruction may be a value corresponding to a range defined by a field of view angle centered on the driver's gaze point. For example, the degree of field of view obstruction may be estimated so that the degree of field of view obstruction is high in a range close to the gaze point and low in a range far from the gaze point. Note that the degree of field of view obstruction may not be limited to the field of view angle, but may also be estimated based on, for example, the distance from the gaze point within the field of view. Furthermore, the degree of field of view obstruction may be estimated based on the shape, size, color, transparency, etc. of the virtual image, rather than the position of the virtual image. For example, when a virtual image is displayed within region Q of FIG. 3, the transparency of the virtual image can be increased or the size of the virtual image can be reduced to prevent the virtual image from obstructing the driver's field of view. When a virtual image is displayed in this manner, the degree of field of view obstruction is reduced.

[0020] The degree of visibility obstruction may also be estimated by combining multiple factors that obstruct the driver's visibility, such as the position of the virtual image, the shape, size, color, and transparency of the virtual image. For example, the degree of visibility obstruction may be a parameter that combines the transparency of the virtual image and the viewing angle in the up, down, left, and right directions from the driver's line of sight. Alternatively, for example, the degree of visibility obstruction may be a parameter that combines the transparency of the virtual image and the position of the virtual image in the driver's field of view.

[0021] The image display unit 110 generates a display image by superimposing a virtual image generated by the virtual image generation unit 111 on a real image viewed by the driver according to the driver's line of sight. For example, when a virtual image (avatar) is displayed at the passenger seat position to realize a virtual space in which the driver is riding with the avatar, if the driver is facing the front of the vehicle, the image display unit 110 generates a real image as a display image. Also, if the driver faces the passenger seat, the image display unit 110 generates a display image by superimposing a virtual image on a real image. Furthermore, if the acceptance is low, the image display unit 110 generates a display image that reduces the degree of visibility obstruction compared to when the acceptance is high. The acceptance will be described later.

[0022] That is, the image display unit 110 performs image processing on the virtual image to reduce the degree of visibility obstruction. The image processing includes image generation and display format determination. Specifically, the image display unit 110 performs image processing such as moving the position of the virtual image away from the line of sight, increasing the transparency of the virtual image, reducing the size of the virtual image, making the virtual image less noticeable, changing the virtual image to a less noticeable color, or not displaying the virtual image at all. Note that the size of the virtual image can be reduced by, for example, maintaining the same shape (reducing) or by changing the shape to a different one and then reducing the size. The image display unit 110 then generates a display image by overlaying the virtual image after image processing on the real image, and displays the generated display image on the screen of the AR device 4.

[0023] The gaze detection unit 120 detects the gaze direction of the in-vehicle user (driver). Specifically, the gaze detection unit 120 acquires an image captured by the in-vehicle camera 3, identifies the driver's eyes from the acquired image, and detects the driver's gaze direction from the driver's viewing position. The gaze detection unit 120 may also measure the distance between the centers of both eyeballs and estimate the position of a gaze target located at the gaze point based on the distance. The gaze detection unit 120 can then estimate whether the gaze target is located outside or inside the vehicle based on whether the distance from the driver's position to the gaze target is within the vehicle interior. When the driver is wearing an AR device 4, the gaze detection unit 120 may transmit and receive signals to and from the AR device 4 and detect the gaze direction of the driver via the AR device 4.

[0024] The acceptance estimation unit 130 estimates the acceptance. The acceptance represents the degree to which the driver accepts the display of the display image on the screen of the AR device 4. For example, even if a virtual image is displayed on the screen of the AR device 4, the driver may not accept the display of the virtual image when the driver is driving, talking to other users in the car, the driver is highly tired, or the driver is concentrating on an activity other than the conversation with the avatar. In other words, the display of the virtual image may interfere with the driver's actions. The acceptance estimation unit 130 estimates the ease with which the display image including the virtual image is accepted as the acceptance. The higher the acceptance value, the more likely the driver is to accept the display of the display image on the screen of the AR device 4.

[0025] The acceptance level estimation unit 130 acquires user information and / or vehicle information, and estimates the acceptance level based on the acquired information. User information is information indicating the user's state, and is indicated by the user's gaze direction, position of the point of gaze, facial expression, movement (movement of the arms, head, etc.), voice, and biological information (brain waves, heart rate, etc.). The acceptance level estimation unit 130 acquires user information by detecting the user's gaze direction, position of the point of gaze, facial expression, and movement from images captured by the in-vehicle camera 3. The acceptance level estimation unit 130 acquires voice information from a speaker in the vehicle. The acceptance level estimation unit 130 acquires biological information using a pulse wave sensor and a heart rate sensor worn by the driver. Note that the user information does not need to include all of the information listed above, such as the user's gaze direction and position of the point of gaze, but it is sufficient to include at least one of the listed information.

[0026] The acceptance level estimation unit 130 estimates the acceptance level based on the acquired user information. The acceptance level estimation unit 130 identifies whether the driver is looking outside or inside the vehicle based on the user's gaze direction or the position of the gaze point included in the user information. If the driver is looking outside the vehicle, it is likely that the driver is driving, and even if a virtual image is displayed while driving, the driver is likely to not accept it. Therefore, when the acceptance level estimation unit 130 identifies that the driver is looking outside the vehicle, it lowers the acceptance level compared to when the driver is looking inside the vehicle. When the gaze detection unit 120 estimates the position of the viewed object, the acceptance level estimation unit 130 may estimate the acceptance level depending on whether the viewed object is located outside or inside the vehicle. That is, when the viewed object is located outside the vehicle, the acceptance level estimation unit 130 estimates a lower acceptance level compared to when the viewed object is located inside the vehicle.

[0027] The acceptance level estimation unit 130 also determines whether the driver is concentrating on something other than the conversation with the avatar based on the user's facial expression, movements, and voice included in the user information. For example, if the driver's face is smiling or the driver is looking back before the avatar is displayed, the driver may be conversing with another user (e.g., talking on the phone or with a user in the backseat). Even if the avatar is displayed in this state, the driver may not pay attention to the avatar, or displaying the avatar may interfere with the conversation between the driver and the other user. In such cases, the acceptance level estimation unit 130 lowers the acceptance level. Note that the driver's conversation with another user may also be determined by analyzing voice information included in the user information.

[0028] The acceptance level estimation unit 130 also determines whether the driver is in a tense state based on biometric information included in the user information. For example, if the driver is in a tense state, such as when the driver is driving under heavy load, the driver may not focus on the avatar even if it is displayed, or the display of the avatar may reduce the driver's concentration. Therefore, when the acceptance level estimation unit 130 determines that the driver is in a tense state based on the biometric information, it lowers the acceptance level.

[0029] The acceptance level estimation unit 130 acquires vehicle information from the in-vehicle device 2 and estimates the acceptance level based on the acquired vehicle information. The acceptance level estimation unit 30 identifies the vehicle's driving state from detection information from a steering sensor, accelerator position sensor, brake pressure sensor, or objective sensor, or information on the operating state of a turn signal, which is included in the vehicle information. When the vehicle is driving, the driver is likely to be driving, and even if a virtual image is displayed while driving, the driver is likely to not accept it. Therefore, when the vehicle is driving, the acceptance level estimation unit 130 lowers the acceptance level compared to when the vehicle is stopped.

[0030] Then, when the acceptance level is low, the image display unit 110 generates a display image that reduces the degree of visibility obstruction compared to when the acceptance level is high, and displays the image on the screen of the AR device 4. A display image that reduces the degree of visibility obstruction is generated, for example, by the following method. The image display unit 110 sets a visibility obstruction threshold according to the acceptance level estimated by the acceptance level estimation unit 130. The visibility obstruction threshold is the upper limit of the visibility obstruction level allowed by the display of a virtual image. For example, when the driver's load is low, such as when the vehicle is stopped, the acceptance level is high, so the visibility obstruction threshold is set to a high value. Next, when generating a virtual image, the image display unit 110 matches the display of the virtual image with the visibility obstruction level and determines the generation and display form of the virtual image so that the visibility obstruction level is lower than the visibility obstruction threshold. If the visibility obstruction threshold is set to a high value, a display screen that obstructs the user's visibility to a large extent can be displayed, so the virtual image generation unit 111 generates a virtual image with low transparency, and the image display unit 110 overlays the virtual image on the real image so that the virtual image is displayed near the point of gaze.

[0031] On the other hand, if the visibility obstruction threshold is set to a low value, a display screen that obstructs the user's visibility to a high degree cannot be displayed, and therefore the virtual image generation unit 111 generates a virtual image with high transparency. The image display unit 110 reduces the visibility obstruction degree when the virtual image is displayed by moving the display position of the generated virtual image away from the gaze point. The image display unit 110 then overlays the virtual image on the real image at a position such that the visibility obstruction degree when the virtual image is displayed is less than the visibility obstruction threshold. As a result, when the acceptance level is low, the image display unit 110 generates a display image with a lower visibility obstruction degree than when the acceptance level is high, and displays the display image on the screen of the AR device 4. Note that the visibility obstruction threshold does not necessarily need to be set, and the method of generating a display image with a lower or higher visibility obstruction degree is not limited to the above and may be other methods.

[0032] Next, a display method of a display image executed by the controller 100 will be described with reference to Fig. 4. Fig. 4 is a flowchart showing a processing flow of image display control by the controller 100. The controller 100 starts the following control flow when it identifies a command to start image display for an in-vehicle user wearing the AR device 4 based on an operation by the in-vehicle user or a control command received from outside the vehicle.

[0033] In step S1, the virtual image generation unit 111 generates a virtual image. This virtual image is an image that has not been subjected to image processing to reduce the degree of visibility obstruction. In step S2, the gaze detection unit 120 detects the driver's gaze direction based on the image captured by the in-vehicle camera 3 and / or the attitude information of the AR device 4.

[0034] In step S3, the acceptance level estimation unit 130 estimates the driver's acceptance level. In step S4, the visibility hindrance level estimation unit 112 estimates the visibility hindrance level. First, the visibility hindrance level estimation unit 112 acquires information indicating the driver's line of sight from the gaze detection unit 120. The visibility hindrance level estimation unit 112 estimates the visibility hindrance level when a virtual image is displayed in the field of view range and / or the periphery of the field of view range specified from the driver's line of sight. The visibility hindrance level is estimated according to the display format of the virtual image. For example, the visibility hindrance level is estimated to be high for a display format in which a virtual image is displayed in a position close to the point of gaze and / or a virtual image with low transparency is displayed. Furthermore, for example, the visibility hindrance level is estimated to be low for a display format in which a virtual image is displayed in a position far from the point of gaze and / or a virtual image with high transparency is displayed.

[0035] In step S5, the image display unit 110 generates a display image to be displayed on the screen of the AR device 4. First, the image display unit 110 determines a display form of the virtual image according to the estimated acceptance level based on the estimated acceptance level and the estimated degree of visibility obstruction. When the acceptance level is high, the image display unit 110 determines a display form of the virtual image that will result in a high degree of visibility obstruction. When the acceptance level is low, the image display unit 110 determines a display form of the virtual image that will result in a low degree of visibility obstruction. The image display unit 110 performs image processing on the virtual image to match the determined display form. Furthermore, the image display unit 110 generates a display image by overlaying the virtual image after image processing on the real image.

[0036] In step S6, the image display unit 110 transmits a signal including the generated display image to the AR device 4. As a result, the image display unit 110 displays the display image on the screen of the AR device 4 so that the degree of visibility obstruction is lower when the acceptance level is low compared to when the acceptance level is high.

[0037] As described above, in this embodiment, controller 100 detects the gaze direction of the in-vehicle user, estimates the acceptance level, which indicates the degree to which the in-vehicle user accepts the display of a display image on the screen of a display device located in the vehicle, and displays the display image on the screen so that the visibility obstruction level, which indicates the degree to which the visibility of the in-vehicle user when looking in the gaze direction, is lower when the acceptance level is low compared to when the acceptance level is high. In this way, the image display is changed to reduce the visibility obstruction level in consideration of the vehicle user's acceptance level and gaze direction, thereby preventing the vehicle user's concentration on driving from being reduced due to the image display in the vehicle.

[0038] In this embodiment, the acceptance level estimation unit 130 acquires user information indicating the state of the user in the vehicle and estimates the acceptance level based on the user information, thereby making it possible to estimate the acceptance level from the user's movements and voice.

[0039] In this embodiment, the acceptance level estimation unit 130 acquires vehicle information from the in-vehicle device 2 of the vehicle 1 and estimates the acceptance level based on the acquired vehicle information. This makes it possible to estimate the acceptance level of the vehicle user based on the vehicle information.

[0040] In this embodiment, the gaze detection unit 120 estimates the position of a visual target in the gaze direction of the in-vehicle user, and the acceptance estimation unit 130 estimates a lower acceptance level when the visual target is located outside the vehicle compared to when the visual target is located inside the vehicle. This makes it possible to identify the vehicle's running state from the gaze direction of the in-vehicle user and estimate the acceptance level.

[0041] Furthermore, in this embodiment, when the acceptance level is low, the image display unit 110 displays the image on the screen in a smaller size than when the acceptance level is high, thereby ensuring the visibility of the user inside the vehicle.

[0042] Furthermore, in this embodiment, when the acceptance level is low, the image display unit 110 displays the display image on the screen with a higher transparency than when the acceptance level is high, thereby ensuring visibility for the user inside the vehicle.

[0043] Furthermore, in this embodiment, when the acceptance level is low, the image display unit 110 displays the virtual image included in the display image on the screen at a position farther from the gaze point of the in-vehicle user than when the acceptance level is high, thereby ensuring the field of view of the in-vehicle user.

[0044] As a modification of this embodiment, when the visual target is located outside the vehicle, the image display unit 110 may display the display image on the screen so that the degree of visibility obstruction is low, regardless of the level of acceptance, or may not display the display image on the screen. When the visual target is located outside the vehicle, it is highly likely that the vehicle is being driven. In such a case, the display image is displayed on the screen so that the degree of visibility obstruction is low, or the display of the virtual image is forcibly prohibited. This makes it possible to forcibly ensure the field of view of the in-vehicle user, regardless of the in-vehicle user's acceptance.

[0045] As a modification of this embodiment, the image display unit 110 may display the display image on the screen so that the visibility obstruction level is lower than a predetermined value, or may not display the display image on the screen. The controller 100 controls the display image of the AR device 4 based on an operation from the in-vehicle user to forcibly ensure the field of view. For example, when the in-vehicle user wants to concentrate on driving, the in-vehicle user may turn off the avatar display, or may perform a limiting operation to limit the avatar display so that the avatar display does not obstruct the field of view. The limiting operation may be performed, for example, by operating a switch in the vehicle, operating a button on the AR device 4, or by voice input using voice recognition processing.

[0046] When the controller 100 receives a control command based on the restricting operation, the controller 100 performs image processing on the virtual image so that the degree of visibility obstruction caused by displaying the virtual image is lower than a predetermined value. The predetermined value is a lower limit of the visibility obstruction degree so that the display of the virtual image does not obstruct the visibility of the in-vehicle user regardless of the user's acceptance level, for example, by forcibly moving the virtual image away from the periphery of the field of view or by significantly increasing the transparency of the virtual image. Furthermore, when the controller 100 receives a control command based on the restricting operation, the controller 100 forcibly prohibits the display of the virtual image and causes the AR device 4 to display only the real image. This allows the controller 100 to forcibly accept a command to forcibly ensure visibility regardless of the user's acceptance level, thereby forcibly ensuring the visibility of the in-vehicle user.

[0047] As a modification of this embodiment, the acceptance level estimation unit 130 may estimate the acceptance level according to the type of visual object. For example, if the visual object is a pedestrian, it is highly likely that the user is driving. Alternatively, if the visual object is a screen for an in-vehicle navigation system or if the user is looking at scenery through a side window, it is highly unlikely that the user is driving. The acceptance level estimation unit 130 identifies the type of object the user is viewing based on images captured by an in-vehicle camera and / or an exterior camera and the visual object estimated by the gaze detection unit 120. If the identified type of visual object is related to vehicle driving (e.g., another vehicle, a sign, an obstacle, etc.), the acceptance level estimation unit 130 lowers the acceptance level. This allows the user in the vehicle to determine whether the user is gazing at a specific object based on the user's visual information and lower the acceptance level, thereby ensuring the user's field of vision.

[0048] In this embodiment, the acceptance level may be expressed as levels such as level 1, level 2, and level 3 depending on the task the in-vehicle user is currently performing (hereinafter referred to as the current task). The current task represents the driver's current activity, such as driving a vehicle, talking to someone, or watching a video. The current task is identified based on vehicle information and / or user information. The acceptance level represents the degree to which the in-vehicle user is willing to accept a task other than the current task. For example, if the current task is driving a vehicle, the driver is unlikely to accept another task, such as talking to an avatar. In such a case, the acceptance level estimation unit 130 sets the acceptance level to "1." On the other hand, if the current task is watching a video, the driver is more likely to accept another task, such as talking to an avatar, than when driving a vehicle. In such a case, the acceptance level estimation unit 130 sets the acceptance level to "3." The image display unit 110 displays the display image on the screen so that the degree of visibility obstruction is lower when the acceptance level is low compared to when the acceptance level is high. This allows the image display to be changed to reduce the degree of visibility obstruction in consideration of the content of the current task of the in-vehicle user, thereby preventing the vehicle user's concentration from being reduced by the image display inside the vehicle.

[0049] In this embodiment, the display image is not limited to the screen of the AR device 4, but may be displayed on a display located inside the vehicle, such as an in-vehicle display for a navigation system or a display of a user terminal owned by a user inside the vehicle.

[0050] In this embodiment, the controller 100 performs image processing on the virtual image to display the display image on the screen so as to reduce the degree of visibility obstruction. However, the display image for which the degree of visibility obstruction is reduced does not necessarily need to include the virtual image. Image processing may be performed on the real image to display the display image on the screen so as to reduce the degree of visibility obstruction. For example, when displaying an image on an in-vehicle display, if the acceptance is low, such as while driving a vehicle, the display screen of the in-vehicle display may be included in the field of view depending on the line of sight of the in-vehicle user. In such a case, the controller 100 performs image processing on the real image to reduce the degree of visibility obstruction, such as by changing the color of the display screen to a less noticeable color. In this way, the controller 100 may perform image processing on the real image to display the display image on the screen so as to reduce the degree of visibility obstruction.

[0051] Second Embodiment Next, an image display device according to a second embodiment will be described. The second embodiment differs in that the controller 100 includes a driving-related information acquisition unit 140 and in the control of the image display unit 110. Note that, apart from the differences from the image display device according to the first embodiment in the points described below, the second embodiment has the same configuration as the first embodiment and operates or functions in the same way as the first embodiment, and the description of the first embodiment will be used as appropriate.

[0052] The image display device according to this embodiment displays a virtual image including driving-related information on the screen of the AR device 4 according to the vehicle's driving scene and the driver's state, even when the acceptance level is low. For example, in a driving scene in which a vehicle is about to enter an intersection and turn right, if the driver's gaze point is located at the position of an oncoming vehicle and a pedestrian on a crosswalk beyond the right turn is not within the driver's field of view, the controller 100 displays a virtual image of a pedestrian on the screen of the AR device 4 to alert the driver to the presence of the pedestrian. In other words, when the driver is concentrating on driving, the acceptance level estimated by the acceptance level estimation unit 130 is low. However, if the driver is not aware of driving-related information such as pedestrians, it is better to proactively display an image including driving-related information. In this embodiment, the controller 100 determines whether the driver is aware of driving-related information. If the controller 100 determines that the driver is not aware of driving-related information, the controller 100 displays the driving-related information in the display image on the screen of the AR device 4, even when the acceptance level is low.

[0053] The controller 100 includes a driving-related information acquisition unit 140 in addition to the image display unit 110 and the like. The driving-related information acquisition unit 140 acquires driving-related information around the vehicle using a sensor that detects the outside of the vehicle, such as a distance measurement sensor or an exterior camera included in the in-vehicle device 2. The driving-related information is information necessary for driving the vehicle, such as information about other vehicles, pedestrians, signs, and obstacles. The driving-related information indicates the shape and size of an object or the position of the object, and is information related to driving other than entertainment or navigation information, such as object information around the vehicle detected by a sensor, information about traffic devices such as traffic lights, and sign information. For example, if the object indicated by the driving-related information is a pedestrian, the driving-related information includes at least the pedestrian's position. The driving-related information acquisition unit 140 may acquire driving-related information according to the driving scene of the vehicle. For example, when the driver operates a turn signal to make a right turn, the driving-related information acquisition unit 140 uses the turn signal operation as a trigger to detect the condition of the road ahead of the right turn using a sensor and acquires driving-related information from the detected data. The driving-related information acquisition unit 140 may acquire driving-related information not only when the vehicle turns right or left, but also when the vehicle changes lanes or other driving situations.

[0054] When the driving-related information is acquired, the image display unit 110 acquires information indicating the driver's line of sight from the gaze detection unit 120 and determines whether the driver is aware of the driving-related information. Specifically, the image display unit 110 identifies an object indicated in the driving-related information and the position of the object, and determines the driver's field of view from the driver's line of sight. If the identified object is within the driver's field of view, the image display unit 110 determines that the driver is aware of the driving-related information. On the other hand, if the identified object is not within the driver's field of view, the image display unit 110 determines that the driver is not aware of the driving-related information. Alternatively, the image display unit 110 may determine whether the driver is aware of the driving-related information depending on whether the driver's gaze point is focused on the object indicated in the driving-related information.

[0055] If it is determined that the driver does not recognize the driving-related information, the virtual image generation unit 111 generates a virtual image representing the driving-related information. For example, if the object represented by the driving-related information is a pedestrian, the virtual image generation unit 111 generates an avatar resembling a person as a virtual image. The image display unit 110 generates a display screen for the AR device 4 so that the virtual image representing the driving-related information is displayed on the screen of the AR device 4. At this time, for example, when a virtual image for realizing a virtual space in which the driver feels as if they are riding with an avatar (a virtual image not related to driving) is displayed, the degree of visibility obstruction may be adjusted according to the acceptance level, as in the first embodiment, and the image may be displayed on the display screen of the AR device 4. When the acceptance level is particularly low, the controller 100 may not display the virtual image not related to driving on the display screen of the AR device 4, but may display a virtual image related to driving, which is information not recognized by the driver, on the display screen of the AR device 4. The display position of the virtual image related to driving may be at least within the driver's field of view (area P or area Q in FIG. 3 ), and may be at or near the driver's point of gaze. Furthermore, when a virtual image showing driving-related information is displayed on the screen of the AR device 4, the virtual image generation unit 111 may emphasize the virtual image. The image display unit 110 may emphasize the virtual image by adjusting the display format, such as color and size, and / or the display position of the virtual image. This allows, for example, when the driver's acceptance is low, driving-related information such as pedestrians that are not within the driver's field of view to be displayed on the display screen of the AR device 4.

[0056] Next, a display method of a display image executed by the controller 100 will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the processing flow of the image display control by the controller 100.

[0057] The control processing of steps S11 and S12 is the same as steps S1 and S2 in the first embodiment, and therefore description thereof will be omitted. In step S13, the driving-related information acquisition unit 140 acquires driving-related information using a sensor. In step S14, the image display unit 110 determines whether the driver is aware of the driving-related information based on information on the driver's line of sight detected by the line of sight detection unit 120 and the acquired driving-related information. If it is determined that the driver is aware of the driving-related information, the controller 100 executes the control processing of step S16.

[0058] If the driver is not aware of the driving-related information, in step S15, the virtual image generation unit 111 generates a virtual image (driving-related image) that shows the driving-related information. The control processes of steps S16 to S19 are the same as steps S3 to S6 in the first embodiment, and therefore, description thereof will be omitted. However, if a virtual image related to driving is generated in the control process of step S15, in step S18, the image display unit 110 generates a display image for the AR device 4 including the virtual image related to driving, and in step S18, the image display unit 110 transmits a signal including the generated display image to the AR device 4.

[0059] As described above, in this embodiment, the controller 100 determines whether the driver is aware of driving-related information, and if it determines that the driver is not aware of the driving-related information, the controller 100 displays the driving-related information in the display image on the screen even if the driver's acceptance level is low. This makes it possible to proactively display driving-related information to alert the driver after determining whether the driver is aware of the driving-related information.

[0060] It should be noted that the above-described embodiments have been described to facilitate understanding of the present invention, and are not intended to limit the present invention. Therefore, each element disclosed in the above-described embodiments is intended to include all design modifications and equivalents that fall within the technical scope of the present invention. [Explanation of symbols]

[0061] 1...Vehicle 2…In-vehicle equipment 3...In-car camera 4…AR equipment 100...Controller 110...Image display unit 120...Gaze detection unit 130...Acceptance estimation unit

Claims

1. An image display unit that displays a display image on a screen of a display device located inside a vehicle; a gaze detection unit that detects the gaze direction of a user inside the vehicle; an acceptance level estimation unit that estimates an acceptance level indicating a degree to which the in-vehicle user accepts display of the display image on the screen, the image display unit displays the display image on the screen so that a visibility obstruction degree, which indicates a degree of obstruction to a view of the user in the vehicle when the user looks in the line of sight, is lower when the acceptance degree is low compared to when the acceptance degree is high, while the user in the vehicle is looking at the display image; The acceptance level estimation unit is an image display device that acquires user information indicating a state of the user in the vehicle and estimates the acceptance level based on the user information.

2. 2. The image display device according to claim 1, The acceptance level estimation unit is an image display device that acquires vehicle information from an in-vehicle device and estimates the acceptance level based on the vehicle information.

3. An image display unit that displays a display image on a screen of a display device located inside the vehicle; a gaze detection unit that detects the gaze direction of a user inside the vehicle; an acceptance level estimation unit that estimates an acceptance level indicating a degree to which the in-vehicle user accepts display of the display image on the screen, the image display unit displays the display image on the screen so that a visibility obstruction degree, which indicates a degree of obstruction to a view of the user in the vehicle when the user looks in the line of sight, is lower when the acceptance degree is low compared to when the acceptance degree is high, while the user in the vehicle is looking at the display image; the line-of-sight detection unit estimates a position of a visual target ahead of the line-of-sight direction, The image display device wherein the acceptance estimating unit estimates the acceptance to be lower when the visual object is located outside the vehicle compared to when the visual object is located inside the vehicle.

4. An image display unit that displays a display image on a screen of a display device located inside the vehicle; a gaze detection unit that detects the gaze direction of a user inside the vehicle; an acceptance level estimation unit that estimates an acceptance level indicating a degree to which the in-vehicle user accepts display of the display image on the screen, the image display unit displays the display image on the screen so that a visibility obstruction degree, which indicates a degree of obstruction to a view of the user in the vehicle when the user looks in the line of sight, is lower when the acceptance degree is low compared to when the acceptance degree is high, while the user in the vehicle is looking at the display image; the line-of-sight detection unit estimates a position of a visual target ahead of the line-of-sight direction, When the visual object is located outside the vehicle, the image display unit displays the display image on the screen so that the degree of visibility obstruction is low, regardless of the magnitude of the acceptance level, or does not display the display image on the screen.

5. The image display device according to any one of claims 1 to 4, When the acceptance level is low, the image display unit reduces the size of the display image and displays it on the screen, compared to when the acceptance level is high.

6. An image display unit that displays a display image on a screen of a display device located inside the vehicle; a gaze detection unit that detects the gaze direction of a user inside the vehicle; an acceptance level estimation unit that estimates an acceptance level indicating a degree to which the in-vehicle user accepts display of the display image on the screen, the image display unit displays the display image on the screen so that a visibility obstruction degree, which indicates a degree of obstruction to a view of the user in the vehicle when the user looks in the line of sight, is lower when the acceptance degree is low compared to when the acceptance degree is high, while the user in the vehicle is looking at the display image; When the acceptance level is low, the image display device displays the display image on the screen with a higher transparency than when the acceptance level is high.

7. An image display unit that displays a display image on a screen of a display device located inside the vehicle; a gaze detection unit that detects the gaze direction of a user inside the vehicle; an acceptance level estimation unit that estimates an acceptance level indicating a degree to which the in-vehicle user accepts display of the display image on the screen, the image display unit displays the display image on the screen so that a visibility obstruction degree, which indicates a degree of obstruction to a view of the user in the vehicle when the user looks in the line of sight, is lower when the acceptance degree is low compared to when the acceptance degree is high, while the user in the vehicle is looking at the display image; When the acceptance level is low, the image display device displays the virtual image included in the display image on the screen at a position farther from the gaze point of the in-vehicle user than when the acceptance level is high.

8. The image display device according to any one of claims 1 to 7, The image display unit is an image display device that, in response to an instruction from the user in the vehicle, displays the display image on the screen so that the degree of visibility obstruction is lower than a predetermined value, or does not display the display image on the screen.

9. The image display device according to any one of claims 1 to 8, The image display unit generates a virtual image and displays the display image including the virtual image on the screen.

10. The image display device according to any one of claims 1 to 9, The image display unit Determine whether the driver is aware of driving-related information related to driving, When it is determined that the driver is not aware of the driving-related information, the image display device displays the driving-related information on the screen, including the driving-related information in the display image, even if the acceptance level is low.

11. 1. An image display method executed by a processor to display a display image on a screen of a display device located inside a vehicle, comprising: The processor: Detects the gaze direction of the user inside the vehicle, acquiring user information indicating the state of the in-vehicle user; estimating an acceptance level indicating a degree to which the in-vehicle user accepts the display of the display image on the screen based on the user information; An image display method in which, when the user inside the vehicle is looking at the display image, the display image is displayed on the screen so that when the acceptance level is low, the visibility obstruction level, which indicates the degree to which the user inside the vehicle obstructs the view when looking in the direction of their line of sight, is lower than when the acceptance level is high.

Images