Display control method and display control device

Abstract

To provide a display control method and a display control device with which it is possible to correct and control, in accordance with the motion of the head of an occupant driving a vehicle, the display position of a virtual image that is displayed by being superimposed on a foreground visible from the occupant.SOLUTION: A display control method and a display control device acquire vehicle vibration information that indicates the vibration of a vehicle, and acquire viewpoint displacement information that indicates the displacement in the vehicle's static system of the viewpoint position of an occupant driving the vehicle. The phase of vibration at a prescribed frequency is calculated as a first phase on the basis of the vehicle vibration information, and the phase of displacement at a prescribed frequency is calculated as a second phase on the basis of the viewpoint displacement information. When a phase difference between the first and second phases is greater than or equal to a prescribed threshold, the display position of a virtual image being displayed in a display region that passes the surrounding of the vehicle through, is corrected on the basis of the displacement at the prescribed frequency.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a display control method and a display control device.

Background Art

[0002] Patent Document 1 discloses a technique for acquiring vehicle state information and correcting a shift in the display position of a virtual image with respect to the foreground when controlling the display position of the virtual image superimposed on the foreground visible to the occupant. According to this technique, based on the vibration generated in the vehicle, it is determined whether the road on which the vehicle is traveling or is scheduled to travel is a rough road, and when it is determined that the road is a rough road, the correction control of the display position of the virtual image is suppressed.

Prior Art Documents

Patent Documents

[0003]

Patent Document Ⅰ

Summary of the Invention

Problems to be Solved by the Invention

[0004] According to the technique described in Patent Document 1, the switching of the correction control of the display position of the virtual image is performed based on whether the road on which the vehicle is traveling or is scheduled to travel is a rough road, and the relationship between the movement of the head of the occupant driving the vehicle and the vibration of the vehicle is not considered. Therefore, there is a problem that the correction control of the display position of the virtual image cannot be performed in accordance with the movement of the head of the occupant.

[0005] The present invention has been made in view of the above problems. An object thereof is to provide a display control method and a display control device capable of performing correction control of the display position of a virtual image superimposed on the foreground visible to the occupant in accordance with the movement of the head of the occupant driving the vehicle.

Means for Solving the Problems

[0006] To solve the above-mentioned problems, a display control method and display control device according to one aspect of the present invention acquire vehicle vibration information indicating vehicle vibration and viewpoint displacement information indicating the displacement of the viewpoint position of an occupant riding in the vehicle in the vehicle's stationary system. Based on the vehicle vibration information, the phase of vibration at a predetermined frequency is calculated as the first phase, and based on the viewpoint displacement information, the phase of displacement at a predetermined frequency is calculated as the second phase. Then, if the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold, the display position of the virtual image to be displayed in a display area that transmits light around the vehicle is corrected based on the displacement at the predetermined frequency. [Effects of the Invention]

[0007] According to the present invention, it is possible to correct and control the display position of a virtual image superimposed on the foreground visible to the occupant, in accordance with the head movements of the occupant driving the vehicle. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing the configuration of a display control device according to one embodiment of the present invention. [Figure 2] Figure 2 is a flowchart showing the processing of a display control device according to one embodiment of the present invention. [Figure 3A] Figure 3A shows an example of a phase difference when correcting the display position based on the displacement of the occupant's viewpoint. [Figure 3B] Figure 3B shows an example of the phase difference when no correction is made to the display position based on the displacement of the occupant's viewpoint position. [Figure 4] Figure 4 shows an example of the shift in the position of the virtual image due to the displacement of the crew member's viewpoint. [Modes for carrying out the invention]

[0009] Next, embodiments of the present invention will be described in detail with reference to the drawings. In the description, identical components are denoted by the same reference numerals, and redundant explanations are omitted.

[0010] [Display control device configuration] Referring to Figure 1, an example of the configuration of the display control device according to this embodiment will be described. The display control device is mounted on a vehicle as an example. As shown in Figure 1, the display control device comprises a display unit 21, a detection unit 30, and a controller 100.

[0011] The display unit 21 is connected to the controller 100 and has a display area 23 that is transparent to the surroundings of the vehicle so that it can be seen from the driver's seat of the vehicle or from the seats of other vehicles. The display area 23 displays an AR (Augmented Reality) guide (image) generated by the controller 100. Therefore, the AR guide is displayed in the display area 23 superimposed on the scenery around the vehicle.

[0012] The AR guide presented by the display unit 21 shows, for example, the guidance route to the vehicle's destination, a road map, the vehicle's current location, points of interest (POIs) between the vehicle and the destination that are of interest to the user, and the locations of surrounding vehicles.

[0013] For example, the display unit 21 is a head-up display device. The display area 23 is a predetermined area on the vehicle's windshield (WS), and an AR guide may be projected from a projector PJ onto the display area 23 to present the AR guide to the user. As a result, the user will see the AR guide superimposed on the scenery around the vehicle as seen through the windshield WS.

[0014] The detection unit 30 acquires viewpoint displacement information indicating the displacement of the viewpoint position of an occupant riding in the vehicle while the vehicle is stationary. More specifically, the detection unit 30 acquires an image of the occupant's head captured by a camera installed inside the vehicle (in-vehicle camera), and based on this image, acquires the occupant's viewpoint position (eye position) and the direction of their gaze (direction of their eyes). The detection unit 30 may also be used in conjunction with a driver monitoring system (DMS) or an occupant monitoring system (OMS).

[0015] A driver monitoring system is a system that monitors the driving behavior of occupants using in-vehicle cameras, with the aim of preventing dangerous driving and accidents. For example, a driver monitoring system may perform facial recognition, dangerous driving detection, distracted driving detection, drowsy driving detection, and age, gender, and facial expression determination. Similarly, an occupant monitoring system is a system that monitors the condition of occupants using in-vehicle cameras.

[0016] Viewpoint displacement information, which indicates the displacement of the viewpoint position of occupants in a vehicle, shows the amount of deviation of the occupants' viewpoint position from a fixed position (reference position) in the vehicle's stationary frame. Viewpoint displacement information consists of information on the displacement of the viewpoint position along the longitudinal direction of the vehicle (the direction of travel when the vehicle is moving straight), information on the displacement of the viewpoint position along the vertical direction of the vehicle (the height direction of the vehicle), and information on the displacement of the viewpoint position along the left-right direction perpendicular to the longitudinal and vertical directions.

[0017] Furthermore, the detection unit 30 acquires vehicle vibration information indicating the vibration of the vehicle. For example, the detection unit 30 may be an on-board sensor that detects the state of the vehicle. The detection unit 30 may also detect the vehicle's movement speed (movement speed in the longitudinal direction, lateral direction, and turning speed), the steering angle of the vehicle's wheels, and the rate of change of the steering angle. In particular, the detection unit 30 may acquire the time variation of the vehicle's pitch angle, time variation of the roll angle, and time variation of the yaw angle as vehicle vibration information.

[0018] Here, the in-vehicle sensors may include sensors that measure the absolute position of the vehicle, i.e., the position, attitude, and speed of the vehicle relative to a predetermined reference point, using position detection sensors that measure the absolute position of the vehicle, such as GPS (Global Positioning System) or odometry.

[0019] The controller 100 is a general-purpose computer including a CPU (Central Processing Unit), a memory, a storage device, an input / output unit, etc. The controller 100 may be connected to a navigation device not shown in the figure. For example, the navigation device executes vehicle route guidance.

[0020] A computer program for functioning as a display control device is installed in the controller 100. By executing the computer program, the controller 100 functions as a plurality of information processing circuits (110, 150, 170, 140) included in the display control device. Note that the computer program may be stored in a computer-readable recording medium.

[0021] In this embodiment, an example of realizing a plurality of information processing circuits (110, 150, 170, 140) by software is shown. However, it is also possible to prepare dedicated hardware for executing each of the information processes shown below to constitute the information processing circuits (110, 150, 170, 140). Further, the plurality of information processing circuits (110, 150, 170, 140) may be constituted by individual hardware. Furthermore, the information processing circuits (110, 150, 170, 140) may be shared with a navigation device or a control unit used for vehicle control.

[0022] As shown in FIG. 1, the controller 100 includes an image generation unit 110, a spectrum conversion unit 150, a position correction unit 170, and an output unit 140 as a plurality of information processing circuits (110, 150, 170, 140).

[0023] The image generation unit 110 generates an AR guide (virtual image) that overlaps with the planned route of the vehicle, which is visible to the occupant in the display area 23 as seen from the driver's seat. The image generation unit 110 sets the display position of the AR guide assuming that the viewpoint of the occupant seated in the driver's seat is at a fixed position (reference position) in the stationary system of the vehicle. This display position is corrected based on viewpoint displacement information by the position correction unit 170, which will be described later.

[0024] The spectral conversion unit 150 decomposes the vehicle vibration into at least one component based on the vehicle vibration information. For example, the spectral conversion unit 150 performs a Fourier transform on the vehicle vibration to decompose it into components for each frequency. Alternatively, the spectral conversion unit 150 may calculate the phase of each component obtained by the decomposition of the vehicle vibration as the first phase.

[0025] Furthermore, the spectral transformation unit 150 decomposes the displacement of the crew's viewpoint position into at least one component based on the viewpoint displacement information. For example, the spectral transformation unit 150 performs a Fourier transform on the displacement of the crew's viewpoint position and decomposes it into components for each frequency. Alternatively, the spectral transformation unit 150 may calculate the phase of each component obtained by the decomposition of the viewpoint position displacement as the second phase.

[0026] In addition, the spectral conversion unit 150 may calculate a specific frequency in which the component obtained by decomposing the vehicle vibration (or the vehicle vibration) has the maximum amplitude based on the vehicle vibration information, and set this specific frequency as a predetermined frequency. Alternatively, the spectral conversion unit 150 may calculate a specific frequency in which the component obtained by decomposing the displacement of the viewpoint position (or the displacement of the viewpoint position) has the maximum amplitude based on the viewpoint displacement information, and set this specific frequency as a predetermined frequency. The predetermined frequency will be used in the position correction unit 170, which will be described later.

[0027] The position correction unit 170 compares the vibration component of the vehicle and the displacement component of the occupant's viewpoint position for each frequency. The position correction unit 170 then determines whether the phase difference between the first phase and the second phase at a predetermined frequency is greater than or equal to a predetermined threshold.

[0028] The position correction unit 170 corrects the display position of the virtual image displayed in the display area 23 based on the component of the displacement of the occupant's viewpoint position at a frequency (predetermined frequency) where it is determined that the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold. In other words, when the position correction unit 170 determines that the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold, it corrects the display position of the virtual image displayed in the display area 23 based on the displacement of the occupant's viewpoint position at that frequency.

[0029] For example, the position correction unit 170 may set the correction amount to be larger the larger the amplitude of the displacement component of the occupant's viewpoint position, and change the display position by the amount of the correction. This suppresses the display of the AR guide, which is superimposed on the scenery around the vehicle, at a position that is significantly different from the scenery around the vehicle.

[0030] In addition, the position correction unit 170 may correct the display position of the virtual image displayed in the display area 23 based on the vibration of the vehicle. This prevents the position of the AR guide, which is displayed superimposed on the scenery around the vehicle, from being displayed in a position that is misaligned with the scenery around the vehicle due to changes in the occupant's viewpoint position caused by the vibration of the vehicle.

[0031] The point that the display position of the AR guide changes due to the displacement of the crew's viewpoint will be explained with reference to Figure 4. Figure 4 is a diagram showing an example of the shift in the position of the virtual image due to the displacement of the crew's viewpoint. Light projected from the projector PJ is reflected, for example, by a reflection point DP on the windshield WS and reaches the crew's viewpoint, resulting in the crew seeing the AR guide.

[0032] When the crew's viewpoint is at position PS1, based on the light reflected at reflection point DP, the crew perceives the virtual image corresponding to the AR guide as being at position MG1. However, if the crew's viewpoint changes to position PS2, based on the light reflected at reflection point DP1, the crew will perceive the virtual image corresponding to the AR guide as being at position MG2. In other words, when the crew's viewpoint changes from position PS1 to position PS2, the position of the virtual image corresponding to the AR guide changes from position MG1 to position MG2.

[0033] Positions MG1 and MG2 are offset by a distance LG along the longitudinal direction of the vehicle. Depending on the magnitude of distance LG, this can cause a sense of incongruity in the virtual image superimposed on the surrounding scenery. Therefore, it becomes necessary to prevent the position of the virtual image corresponding to the AR guide from changing from position MG1 to position MG2. By adjusting the direction of the light projected from projector PJ to change reflection point DP1 to reflection point DP2 (i.e., changing the display position of the AR guide), the position of the virtual image corresponding to the AR guide can be maintained at position MG2 even if the occupant's viewpoint changes.

[0034] Next, the relationship between the phase component of the occupant's viewpoint position and the phase component of the vehicle's vibration will be explained using Figures 3A and 3B.

[0035] Changes in the occupant's viewpoint position can include both changes that lag behind vehicle vibrations (Figure 3A) and changes that lag sufficiently behind vehicle vibrations (Figure 3B). Here, if a correction based on changes in the occupant's viewpoint position that lags sufficiently behind vehicle vibrations is performed simultaneously with a correction based on vehicle vibrations, the amount of correction becomes excessive.

[0036] Therefore, if the display position is corrected based on the components obtained by the decomposition of the viewpoint position displacement, regardless of the phase of each component (second component) obtained by the decomposition of the viewpoint position displacement, the correction of the display position may become excessive. This can result in an unnatural feeling towards the corrected virtual image. To suppress such unnatural feelings associated with correction, the position correction unit 170 corrects the display position based on the components of the occupant's viewpoint position displacement at a frequency (predetermined frequency) where the phase difference between the first phase and the second phase is determined to be greater than or equal to a predetermined threshold.

[0037] Figure 3A shows an example of a phase difference when correcting the display position based on the displacement of the occupant's viewpoint. In Figure 3A, the component TS2 of the displacement of the occupant's viewpoint follows the component TS1 of the vehicle vibration with a phase difference ΔΦ of half a wavelength or more. In this case, the position correction unit 170 corrects the display position based on the component of the displacement of the occupant's viewpoint at a predetermined frequency.

[0038] On the other hand, Figure 3B shows an example of a phase difference when no correction is made to the display position based on the displacement of the occupant's viewpoint. In Figure 3B, the component TS2 of the displacement of the occupant's viewpoint follows the component TS1 of the vehicle vibration sufficiently with a phase difference ΔΦ of less than half a wavelength. In this case, the position correction unit 170 does not perform correction of the display position based on the component of the displacement of the occupant's viewpoint at a predetermined frequency in order to suppress the unnatural feeling associated with the correction.

[0039] The position correction unit 170 may also correct the display position when the amplitude of the vehicle vibration component or the amplitude of the displacement component of the occupant's viewpoint position is greater than a predetermined amplitude threshold. As a result, if the AR guide's display position does not deviate significantly from the surrounding scenery of the vehicle, the display position correction is not performed, and the processing load associated with the display position correction is reduced.

[0040] In addition, the position correction unit 170 may set the predetermined threshold used when comparing the first phase and the second phase to a phase of half a wavelength of a predetermined frequency.

[0041] As a result, the position correction unit 170 can determine that, at a predetermined frequency, the displacement component of the occupant's viewpoint position follows the vehicle's vibration component sufficiently quickly when the phase difference between the first phase and the second phase is less than half a wavelength. Furthermore, the position correction unit 170 can determine that, at a predetermined frequency, the displacement component of the occupant's viewpoint position follows the vehicle's vibration component with a delay when the phase difference between the first phase and the second phase is greater than or equal to half a wavelength.

[0042] The output unit 140 outputs the AR guide generated by the image generation unit 110. The output AR guide is displayed in the display area 23 at a display position corrected by the position correction unit 170.

[0043] [Processing procedure for display control device] Next, the processing procedure of the display control device according to this embodiment will be described with reference to the flowchart in Figure 2. Figure 2 is a flowchart showing the processing of the display control device according to one embodiment of the present invention.

[0044] In step S101, the detection unit 30 acquires vehicle vibration information indicating vehicle vibration.

[0045] In step S103, the detection unit 30 acquires viewpoint displacement information indicating the displacement of the viewpoint position of the occupants riding in the vehicle within the vehicle's stationary system.

[0046] In step S105, the spectral conversion unit 150 calculates the phase of each component obtained by the decomposition of the vehicle's vibrations as the first phase.

[0047] In step S107, the spectral conversion unit 150 calculates the phase of each component obtained by the decomposition of the viewpoint position as the second phase.

[0048] In step S109, the position correction unit 170 determines whether the phase difference between the first phase and the second phase at a predetermined frequency is greater than or equal to a predetermined threshold.

[0049] If it is determined that the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold (if the answer is YES in step S109), the position correction unit 170 calculates the amount of correction for the display position in step S111.

[0050] If it is determined that the phase difference between the first phase and the second phase is less than a predetermined threshold (if NO in step S109), in step S113, the position correction unit 170 sets the correction amount of the display position to 0.

[0051] In step S115, the image generation unit 110 generates an AR guide (image). Subsequently, the position correction unit 170 corrects the display position of the virtual image to be displayed in the display area.

[0052] In step S117, the output unit 140 outputs an AR guide with the display position corrected. The output AR guide is presented to the user via the display area 23. After that, the display control device terminates its processing.

[0053] [Effects of the Embodiment] As described in detail above, the display control method and display control device according to this embodiment acquire vehicle vibration information indicating vehicle vibration and viewpoint displacement information indicating the displacement of the viewpoint position of an occupant riding in the vehicle in the vehicle's stationary system. Based on the vehicle vibration information, the phase of vibration at a predetermined frequency is calculated as the first phase, and based on the viewpoint displacement information, the phase of displacement at a predetermined frequency is calculated as the second phase. Then, if the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold, the display position of the virtual image to be displayed in the display area that transmits light around the vehicle is corrected based on the displacement at the predetermined frequency.

[0054] This allows for correction and control of the display position of the virtual image superimposed on the foreground visible to the occupant, in accordance with the head movements of the occupant driving the vehicle.

[0055] Changes in the position of the occupant's viewpoint can include both changes that occur in a sufficiently fast response to vehicle vibrations and changes that follow the vehicle vibrations with a delay. Here, correction based on changes in the position of the occupant's viewpoint that follow the vehicle vibrations with a delay is in the opposite direction to correction based on vehicle vibrations. According to the display control method and display control device of this embodiment, the display position is corrected only when the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold, thereby suppressing the sense of unnaturalness regarding the corrected virtual image. As a result, user convenience is improved.

[0056] Furthermore, the display control method and display control device according to this embodiment may correct the display position based on vehicle vibrations. This makes it possible to suppress discrepancies between the surrounding scenery of the vehicle and the position of the AR guide caused by vehicle vibrations.

[0057] Furthermore, the display control method and display control device according to this embodiment may calculate a specific frequency having the maximum amplitude of vibration based on vehicle vibration information, and set this specific frequency as a predetermined frequency. This allows for a determination of whether or not to correct the display position by focusing on the component that causes the greatest discrepancy between the scenery around the vehicle and the position of the AR guide. As a result, it becomes possible to correct the display position with higher accuracy. Moreover, it is possible to suppress the sense of unnaturalness regarding the corrected virtual image.

[0058] Furthermore, the display control method and display control device according to this embodiment may calculate a specific frequency having the maximum amplitude of displacement based on viewpoint displacement information, and set this specific frequency as a predetermined frequency. This allows for a determination of whether or not to correct the display position by focusing on the component that causes the greatest discrepancy between the scenery around the vehicle and the position of the AR guide. As a result, it becomes possible to correct the display position with higher accuracy. Moreover, it is possible to suppress the sense of unnaturalness towards the corrected virtual image.

[0059] Furthermore, the display control method and display control device according to this embodiment may set a predetermined threshold value to a phase of half a wavelength of a predetermined frequency. This allows it to be determined that, at a predetermined frequency, the displacement component of the occupant's viewpoint position follows the vehicle's vibration component sufficiently quickly when the phase difference between the first phase and the second phase is smaller than half a wavelength. Conversely, when the phase difference between the first phase and the second phase is greater than or equal to half a wavelength, it can be determined that, at a predetermined frequency, the displacement component of the occupant's viewpoint position follows the vehicle's vibration component with a delay.

[0060] Each of the functions described in the embodiments above may be implemented by one or more processing circuits. These processing circuits may include programmed processors, electrical circuits, and even devices such as application-specific integrated circuits (ASICs) and circuit components arranged to perform the described functions.

[0061] Although the present invention has been described above in accordance with the embodiments, it will be obvious to those skilled in the art that the present invention is not limited to these descriptions and that various modifications and improvements are possible. The discussion and drawings that constitute part of this disclosure should not be understood as limiting the present invention. Various alternative embodiments, examples, and operational techniques will become apparent to those skilled in the art from this disclosure.

[0062] Of course, the present invention includes various embodiments and the like that are not described herein. Therefore, the technical scope of the present invention is defined solely by the inventive features of the claims that are reasonable based on the above description. [Explanation of symbols]

[0063] 21 Display section 23 Display area 30 Detection unit 100 controllers 110 Image generation unit 140 Output section 150 Spectrum Conversion Section 170 Position correction section

Claims

1. A display control method for controlling a device comprising a display unit, a detection unit, and a controller, The aforementioned display unit has a display area that allows the surroundings of the vehicle to pass through. The detection unit is Vehicle vibration information indicating the vibration of the aforementioned vehicle is acquired, Obtain viewpoint displacement information indicating the displacement of the viewpoint position of an occupant riding in the vehicle in the vehicle's stationary system. The aforementioned controller, Based on the vehicle vibration information, the phase of the vibration at a predetermined frequency is calculated as the first phase. Based on the viewpoint displacement information, the phase of the displacement at the predetermined frequency is calculated as the second phase. Determine whether the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold. If it is determined that the value is greater than or equal to the predetermined threshold, the display position of the virtual image to be displayed in the display area is corrected based on the displacement at the predetermined frequency. A display control method characterized by the following.

2. A display control method according to claim 1, The aforementioned controller, The display position is corrected based on the aforementioned vibration. A display control method characterized by the following.

3. A display control method according to claim 1, The aforementioned controller, Based on the vehicle vibration information, a specific frequency having the maximum amplitude of the vibration is calculated. The specified frequency is set as the predetermined frequency. A display control method characterized by the following.

4. A display control method according to claim 1, The aforementioned controller, Based on the viewpoint displacement information, a specific frequency is calculated in which the displacement has the largest amplitude. The specified frequency is set as the predetermined frequency. A display control method characterized by the following.

5. A display control method according to any one of claims 1 to 4, The aforementioned controller, The phase of half a wavelength of the predetermined frequency is set as the predetermined threshold. A display control method characterized by the following.

6. A display control device comprising a display unit, a detection unit, and a controller, The aforementioned display unit has a display area that allows the surroundings of the vehicle to pass through. The detection unit is Vehicle vibration information indicating the vibration of the aforementioned vehicle is acquired, Obtain viewpoint displacement information indicating the displacement of the viewpoint position of an occupant riding in the vehicle in the vehicle's stationary system. The aforementioned controller, Based on the vehicle vibration information, the phase of the vibration at a predetermined frequency is calculated as the first phase. Based on the viewpoint displacement information, the phase of the displacement at the predetermined frequency is calculated as the second phase. Determine whether the phase difference between the first phase and the second phase is greater than or equal to a predetermined threshold. If it is determined that the value is greater than or equal to the predetermined threshold, the display position of the virtual image to be displayed in the display area is corrected based on the displacement at the predetermined frequency. A display control device characterized by the following.

Images