Display control device, head-mounted display, and display control method

Abstract

To reduce a load of image processing while maintaining quality of a user experience, in an image display technique using a head-mounted display.SOLUTION: A stereo camera in a head-mounted display captures a left-viewpoint and a right-viewpoint captured images 24a, 24b, and the like, at a frame rate of 1 / Δt, as shown in (a). As shown in (c), a display control device uses one of the left-viewpoint and right-viewpoint captured images to generate display images for left and right eyes (e.g., display images 28a, 28b) alternately in each frame at the same rate as when capturing, and causes the display images to be each displayed in a corresponding region of a display panel while making the other region hidden.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This invention relates to a display control device for controlling image display, a head-mounted display for displaying an image, and a display control method performed thereby.

Background Art

[0002] Image display systems that can view a target space from a free viewpoint have become widespread. For example, a system has been developed in which a panoramic video is displayed on a head-mounted display, and an image corresponding to the line-of-sight direction of a user wearing the head-mounted display is displayed. By using a head-mounted display, it is possible to enhance the sense of immersion in the video and improve the operability of applications such as games.

[0003] In addition, a technique for realizing mixed reality by providing a video camera that shoots the real space on a head-mounted display and superimposing computer graphics on the captured image has also been put into practical use. For example, Patent Document 1 discloses a technique for suppressing the amount of data to be processed by generating a mixed reality image using frames decimated from the captured moving image and displaying the image at a long cycle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When displaying images within a field of view corresponding to the user's gaze direction, low-latency display that follows the user's movements is required. On the other hand, attempting to achieve higher image quality by increasing image resolution and frame rate increases the data size of the images that need to be processed. As a result, the processing load increases and communication bandwidth becomes congested, making it easier for delays to occur before display. Increased processing load also affects power consumption, and in the case of head-mounted displays in particular, it can shorten the battery life and worsen wearing comfort due to heat generation.

[0006] This invention has been made in view of these problems, and its purpose is to provide a technology that can reduce the image processing load while maintaining the quality of the user experience in image display technology using a head-mounted display. [Means for solving the problem]

[0007] To solve the above problems, one aspect of the present invention relates to a display control device. This display control device is a display control device that displays left-eye and right-eye display images constituting frames of a moving image in the left and right regions of a display panel, respectively, and is characterized by comprising: an image data generation unit that alternately generates either the left-eye or right-eye display image for each frame; and an output control unit that controls the display panel so that one of the display images is displayed in the corresponding region of the left and right regions, while the other region is hidden.

[0008] Another aspect of the present invention relates to a head-mounted display. This head-mounted display is characterized by comprising the display control device, a stereo camera for capturing stereo moving images to be displayed as display images, and the display panel.

[0009] A further aspect of the present invention relates to a display control method. This display control method is characterized in that a display control device that displays display images for the left eye and the right eye constituting frames of a moving image in the left and right regions of a display panel, respectively, includes the steps of alternately generating one of the display images for the left eye or the right eye for each frame, and controlling the display panel so that one of the display images is displayed in the corresponding region of the left and right regions, while the other region is hidden.

[0010] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, computer programs, data structures, recording media, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0011] According to the present invention, in image display technology using a head-mounted display, it is possible to reduce the image processing load while maintaining the quality of the user experience. [Brief explanation of the drawing]

[0012] [Figure 1] This figure shows an example of the appearance of the head-mounted display according to this embodiment. [Figure 2] This figure shows an example configuration of the image display system of this embodiment. [Figure 3] This diagram schematically shows the data path in the image display system of this embodiment. [Figure 4] This figure illustrates the process of generating a display image from a captured image in this embodiment. [Figure 5] This figure illustrates a configuration in which images for the left eye and the right eye are displayed alternately in this embodiment. [Figure 6] This figure shows the internal circuit configuration of the content processing device in this embodiment. [Figure 7] This figure shows the internal circuit configuration of the head-mounted display in this embodiment. [Figure 8]This is a diagram showing the configuration of the functional blocks of the display control device in the present embodiment. [Figure 9] This is a diagram for explaining the cross-fading method by the mode transition control unit in the present embodiment. [Figure 10] This is a diagram schematically showing the transition of the display image when the luminance change shown in (b) of FIG. 9 is given. [Figure 11] This is a diagram exemplifying a general processing procedure when generating a display image from a captured image. [Figure 12] This is a diagram showing the processing procedure when the correction unit generates a display image from the captured image in the present embodiment. [Figure 13] This is a diagram schematically showing a state in which the additional image drawing unit superimposes an additional image on the display image in the present embodiment. [Figure 14] This is a diagram for explaining a method for the additional image drawing unit to efficiently draw an additional image in the present embodiment.

Embodiments for Carrying Out the Invention

[0013] FIG. 1 shows an external appearance example of the head-mounted display 100. In this example, the head-mounted display 100 is composed of an output mechanism unit 102 and a mounting mechanism unit 104. The mounting mechanism unit 104 includes a mounting band 106 that wraps around the head when worn by the user to achieve fixation of the device. The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head-mounted display 100, and a display panel is provided inside so as to face the eyes during wearing.

[0014] Inside the housing 108, there is further provided an eyepiece lens that is located between the display panel and the user's eyes when the head-mounted display 100 is worn and magnifies the image. The head-mounted display 100 may further include a speaker or earphone at a position corresponding to the user's ears when worn. The head-mounted display 100 may also incorporate a motion sensor to detect the translational and rotational movements of the head of the user wearing the head-mounted display 100, and thus the position and orientation at each moment.

[0015] The head-mounted display 100 further includes a stereo camera 110 on the front surface of the housing 108. In the present embodiment, by displaying the moving image captured by the stereo camera 110 with a small delay, a mode is provided that shows the state of the real space in the direction the user is facing as it is. Hereinafter, such a mode will be referred to as a "see-through mode". For example, the head-mounted display 100 automatically enters the see-through mode during a period when no content image is being displayed.

[0016] Thereby, the user can check the surrounding situation without removing the head-mounted display 100 before the start, after the end, or during the interruption of the content. The see-through mode may also be started or ended triggered by an explicit operation by the user. Thereby, even during the appreciation of the content, the display can be temporarily switched to the image of the real space at an arbitrary timing, and necessary operations such as dealing with sudden events in the real world can be performed. In the illustrated example, the stereo camera 110 is provided below the front surface of the housing 108, but its arrangement is not particularly limited. Also, a camera other than the stereo camera 110 may be provided.

[0017] Images captured by the stereo camera 110 can also be used as content images. For example, by compositing virtual objects onto the captured images with their position, orientation, and movement matching real objects in the camera's field of view, Augmented Reality (AR) and Mixed Reality (MR) can be realized. Furthermore, regardless of whether the captured images are included in the display or not, the captured images can be analyzed, and the results can be used to determine the position, orientation, and movement of objects to be rendered.

[0018] For example, by applying stereo matching to the captured image, corresponding points of the subject's image can be extracted, and the distance to the subject can be obtained using the principle of triangulation. Alternatively, the position and orientation of the head-mounted display 100, and by extension the user's head, relative to the surrounding space can be obtained using well-known techniques such as Visual SLAM (Simultaneous Localization and Mapping). Through these processes, the virtual world can be rendered and displayed with a field of view corresponding to the user's viewpoint and gaze direction.

[0019] Figure 2 shows an example configuration of the image display system in this embodiment. In the image display system 10, the head-mounted display 100 is connected to the content processing device 200 via wireless communication or an interface for connecting peripheral devices such as USB Type-C. The content processing device 200 may also be connected to a server via a network. In that case, the server may provide the content processing device 200 with online applications such as games that multiple users can participate in via the network.

[0020] The content processing device 200 is essentially an information processing device that processes content, generates display images, and transmits them to the head-mounted display 100 for display. Typically, the content processing device 200 identifies the viewpoint position and direction of gaze based on the head position and posture of the user wearing the head-mounted display 100, and generates display images within the corresponding field of view. For example, the content processing device 200 generates images representing the virtual world that serves as the setting for an electronic game while progressing through the game, thereby realizing virtual reality (VR).

[0021] In this embodiment, the content processed by the content processing device 200 is not particularly limited; as described above, it may implement AR or MR, or it may be content for which display images have been created in advance, such as a movie. In the following description, images other than real-time images of the real space displayed in see-through mode will be referred to as "content images" for comparison.

[0022] Figure 3 schematically shows the data path in the image display system 10 of this embodiment. The head-mounted display 100 includes a stereo camera 110 and a display panel 122 as described above. The display panel 122 is a panel having a general display mechanism such as a liquid crystal display or an organic EL display. In this embodiment, the display panel 122 displays images for the left eye and right eye, which constitute the frames of the moving image, in the left and right regions that are directly facing the user's left and right eyes, respectively.

[0023] By creating a stereo image with parallax corresponding to the distance between the left and right eyes, the displayed object can be shown in three dimensions. The display panel 122 may consist of two panels, one for the left eye and one for the right eye, placed side by side, or it may be a single panel that displays an image created by connecting the left and right eye images horizontally.

[0024] The head-mounted display 100 further includes an image processing integrated circuit 120. The image processing integrated circuit 120 is, for example, a system-on-a-chip equipped with various functional modules, including a CPU. In addition, the head-mounted display 100 may also be equipped with motion sensors such as a gyroscope, accelerometer, and angular acceleration sensor, as well as main memory such as DRAM (Dynamic Random Access Memory), an audio circuit for providing sound to the user, and a peripheral device interface circuit for connecting peripheral devices, but these are omitted from the illustration here.

[0025] The diagram shows two data paths with arrows in the case where images captured by the stereo camera 110 are included in the display. When realizing AR or MR, images captured by the stereo camera 110 are generally taken in by the content processing unit, where they are combined with virtual objects to generate a display image. In the illustrated image display system 10, the content processing unit is the content processing unit 200, so as shown by arrow B, the images captured by the stereo camera 110 are first transmitted to the content processing unit 200 via the image processing integrated circuit 120.

[0026] The virtual object is then synthesized and returned to the head-mounted display 100, where it is displayed on the display panel 122. On the other hand, in see-through mode, as shown by arrow A, the image captured by the stereo camera 110 can be corrected by the image processing integrated circuit 120 to an image suitable for display and then displayed on the display panel 122. The path of arrow A has a significantly shorter data transmission path compared to the path of arrow B, thus reducing the time from image capture to display and lowering the power consumption required for transmission.

[0027] However, the purpose of this embodiment is not to limit the data path for the see-through mode to arrow A. In other words, the path indicated by arrow B may be used, and the image captured by the stereo camera 110 may be transmitted to the content processing device 200. The content processing device 200 may then correct the image for display and send it back to the head-mounted display 100 for display.

[0028] Figure 4 is a diagram illustrating the process of generating a display image from a captured image. Assume that in real space, a table with an object on it is in front of the user. The stereo camera 110 captures it, obtaining a captured image 16a from the left viewpoint and a captured image 16b from the right viewpoint. Due to the parallax of the stereo camera 110, there is a horizontal shift in the position of the image of the same subject in the captured images 16a and 16b.

[0029] Furthermore, the camera lens causes distortion in the image of the subject in the captured images 16a and 16b. Generally, such lens distortion is corrected to generate distortion-free images 18a (left view) and 18b (right view). If the pixels at position coordinates (x,y) in the original captured images 16a and 16b are corrected to position coordinates (x+Δx,y+Δy) in the corrected images 18a and 18b, then the displacement vector (Δx,Δy) representing the amount of correction can be expressed by the following general formula.

[0030]

number

[0031] Here, r is the distance from the optical axis of the lens to the target pixel in the image plane, and (Cx,Cy) is the position of the optical axis of the lens. Also, k1, k2, k3, ... are lens distortion coefficients and depend on the lens design. There is no particular upper limit to the order. When displaying captured images on a flat panel display or performing image analysis, a general image corrected in this way is used. On the other hand, in the head-mounted display 100, in order for distortion-free images 18a and 18b to be visible when viewed through the eyepiece, it is necessary to apply a distortion that is the opposite of the distortion caused by the eyepiece.

[0032] For example, in the case of a lens where the four sides of an image appear concave in a pincushion shape, the image is curved in a barrel shape. In other words, the distortion-free images 18a and 18b are distorted to correspond to the eyepiece and adjusted to the size of the display panel 122, thereby generating the final display image 22 consisting of an image for the left eye and an image for the right eye (S12). The relationship between the images of each subject in the left-eye and right-eye display images 22 and the images of the subjects in the uncorrected distortion-free images 18a and 18b is equivalent to the relationship between an image with camera lens distortion and an image with distortion correction.

[0033] Therefore, the inverse vector of the displacement vector (Δx, Δy) in Equation 1 can be used to generate a distorted image in the displayed image 22. However, the distortion coefficient is a value specific to the eyepiece. Note that this embodiment does not intend to limit the formula used for correction to Equation 1. Also, as shown in Figure 1, if the stereo camera 110 in the head-mounted display 100 is located at a position offset from the user's line of sight, it is necessary to correct the captured image to an image corresponding to the user's field of view. A commonly used transformation matrix can be used for correction.

[0034] Other necessary corrections include correction for chromatic aberration in the eyepiece and gamma correction according to the characteristics of the display panel 122. Furthermore, it may be necessary to blur the edges of the field of view in the displayed image 22 or to superimpose necessary additional images such as the UI (User Interface). Thus, even when simply displaying a captured image in see-through mode, various processing is actually required, which can cause delays from capture to display.

[0035] Even high-performance systems capable of achieving high frame rates and high resolution displays may not fully utilize their capabilities if, depending on the image processing, image generation cannot keep up with the display cycle, resulting in so-called frame drops. In the case of head-mounted displays in particular, the delay in display in response to the user's head movements is easily apparent, which can actually degrade the quality of the user experience or even cause physical discomfort such as motion sickness.

[0036] Therefore, in this embodiment, the head-mounted display 100 is made to fully utilize its inherent performance by reducing the processing load while maintaining the quality of the user experience. Specifically, the head-mounted display 100 generates and displays either the left eye image or the right eye image alternately for each frame. Furthermore, as will be described later, by pre-calculating an integrated correction amount which is the sum of the correction amounts required for multiple types of correction, the correction process can be completed in one step, as in S14.

[0037] Furthermore, when displaying additional images such as UI elements overlaid on the display image, efficiency is improved by directly drawing the distorted additional image onto the distorted plane of the display image. By implementing at least one of these processes, it is possible to reduce the image processing load while still allowing the smooth motion made possible by the high frame rate to be visually apparent. Although this embodiment is not limited to see-through mode, an example of its application to see-through mode will be described below as a representative example.

[0038] Figure 5 illustrates the configuration in this embodiment where images for the left eye and the right eye are displayed alternately. (a) shows the image captured by the stereo camera 110, and (b) and (c) show the transition of the displayed image by the display panel 122, schematically with respect to the vertical time axis. In this example, the frame rates of the stereo camera 110 and the display panel 122 are set to a common value (=1 / Δt), but this embodiment is not intended to be limited to this.

[0039] As shown in (a), the captured image from the left viewpoint (e.g., captured image 24a) and the captured image from the right viewpoint (e.g., captured image 24b) are acquired at the timings t, t+Δt, t+2Δt, t+3Δt, ... In other words, the captured images correspond to frames of stereo video acquired with a period of Δt. The frame rate is, for example, 120fps. If all of these frames are corrected into display images and displayed at the same 120fps, as mentioned above, depending on the processing, it is possible that significant delays may occur or the display may become unstable.

[0040] One possible solution is to skip frames in the captured images and update the images at a rate of 60fps, thereby increasing processing time. However, in this case, the head-mounted display 100 would not be able to utilize its inherent 120fps display performance. For example, even if the user suddenly changes the direction of their face, the field of view from the previous moment remains fixed, which can easily cause discomfort. Also, simply halving the frame rate would compromise the smoothness of the moving images that could otherwise be displayed.

[0041] (b) shows one solution to this problem, which involves alternating between periods when the image is displayed and periods when it is hidden. Specifically, if the delay time due to the correction process from capture to display is Tt, then the images 24a and 24b captured at time t are displayed as the images 26a and 26b at time T. The image captured at the next time t+Δt is hidden. Furthermore, the image captured at the next time t+2Δt is displayed as the image at time T+2Δt, and the image captured at the next time t+3Δt is hidden.

[0042] In this way, a period of image hiding is allowed, and the display panel 122 refreshes the display at its original frame rate. Humans have the characteristic of being able to interpolate missing frames in their brains based on the changes in the displayed image up to that point. Therefore, compared to continuously displaying the downsampled images with twice the period 2Δt as described above, deliberately introducing a period of hiding makes the image appear smoother and less likely to cause discomfort even with sudden movements of the field of view. On the other hand, in this case, for example, at time t when the captured images 24a and 24b are obtained, it is necessary to start the correction processing of those images simultaneously, which may temporarily increase the load and potentially increase the delay time Tt.

[0043] Based on this, in this embodiment, as shown in (c), captured images from the right and left viewpoints are alternately thinned out, and the remaining captured images are used to alternately generate and display a display image for the left eye (e.g., display image 28a) and a display image for the right eye (e.g., display image 28b). In the example shown in the figure, of the captured images 24a and 24b at time t, the display image 28a for the left eye is generated and displayed from the captured image 24a from the left viewpoint, while the display image for the right eye is hidden. For the image captured at the next time t+Δt, the display image 28b for the right eye is generated and displayed from the captured image from the right viewpoint, while the display image for the left eye is hidden.

[0044] Thereafter, the display and hiding of the left-eye and right-eye display images are switched in the same manner. This averages out the number of images to be processed over time compared to case (b), so that half the data needs to be processed at all times. As a result, the delay time T'-t from capture to display can be reduced by approximately half. Also, similar to (b), during periods when the image is hidden, the brain interpolates based on the changes in the displayed image up to that point, allowing the display panel 122 to exhibit smooth motion at its inherent frame rate.

[0045] On the other hand, by using images for the left and right eyes that were captured at different times for display, the latest state can be consistently represented with high temporal resolution using either one of the images. This makes interpolation in the brain more accurate and easier compared to case (b). In the above explanation, "not showing" means that a black image may be displayed on one of the display panels or the relevant area, or the illumination of the display panel itself may be temporarily suspended. In the former case, control is easier, and in the latter case, the power consumption of the display panel can be reduced further.

[0046] Figure 6 shows the internal circuit configuration of the content processing unit 200. The content processing unit 200 includes a CPU (Central Processing Unit) 222, a GPU (Graphics Processing Unit) 224, and main memory 226. These components are interconnected via a bus 230. An input / output interface 228 is further connected to the bus 230. A communication unit 232, a storage unit 234, an output unit 236, an input unit 238, and a recording medium drive unit 240 are connected to the input / output interface 228.

[0047] The communication unit 232 includes peripheral device interfaces such as USB and IEEE1394, and network interfaces such as wired LAN or wireless LAN. The storage unit 234 includes a hard disk drive and non-volatile memory. The output unit 236 outputs data to the head-mounted display 100. The input unit 238 receives data input from the head-mounted display 100 and receives user operation information from a controller (not shown). The recording medium drive unit 240 drives removable recording media such as magnetic disks, optical disks, or semiconductor memory.

[0048] The CPU 222 controls the entire content processing device 200 by executing the operating system stored in the memory unit 234. The CPU 222 also executes various programs (e.g., game applications) read from the memory unit 234 or a removable recording medium and loaded into the main memory 226, or downloaded via the communication unit 232. The GPU 224 has both geometry engine and rendering processor functions, performing drawing processing according to drawing commands from the CPU 222 and outputting the drawing results to the output unit 236. The main memory 226 is composed of RAM (Random Access Memory) and stores the programs and data necessary for processing.

[0049] Figure 7 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU 136, main memory 138, display unit 124, and audio output unit 126. These units are interconnected via a bus 128. An input / output interface 130 is further connected to the bus 128. The input / output interface 130 is connected to a communication unit 132, which includes a wireless communication interface, a motion sensor 134, and a stereo camera 110.

[0050] The CPU 136 processes information acquired from various parts of the head-mounted display 100 via the bus 128, and also supplies display images and audio data acquired from the content processing device 200 to the display unit 124 and the audio output unit 126. The main memory 138 stores the programs and data necessary for processing by the CPU 136.

[0051] The display unit 124 includes the display panel 122 shown in Figure 3 and displays an image in front of the eyes of the user wearing the head-mounted display 100. The display unit 124 further includes an eyepiece lens positioned between the display panel 122 and the user's eyes when the head-mounted display 100 is worn. The audio output unit 126 consists of a speaker or earphone positioned to correspond to the user's ears when the head-mounted display 100 is worn, and provides the user with audio.

[0052] The communication unit 132 is an interface for sending and receiving data with the content processing device 200, and communication is achieved using known wireless communication technologies such as Bluetooth®. The motion sensor 134 includes a gyro sensor, an accelerometer, an angular acceleration sensor, etc., and acquires the tilt, acceleration, angular velocity, etc. of the head-mounted display 100. The stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints, as shown in Figure 1.

[0053] Figure 8 shows the configuration of the functional blocks of the display control device in this embodiment. Note that, as shown in the data path indicated by arrow A in Figure 3, when image capture and display are completed within the head-mounted display 100 in see-through mode, the display control device 150 is provided in the head-mounted display 100. As shown in the data path indicated by arrow B, when the content processing device 200 handles at least a portion of the display image generation process, the display control device 150 is divided and provided in both the content processing device 200 and the head-mounted display 100.

[0054] Furthermore, the functional blocks illustrated can be realized in hardware terms with the circuit configurations shown in Figures 6 and 7, and in software terms with programs that perform various functions such as data input, data retention, image processing, and communication, which are loaded from a recording medium or the like into main memory. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways using hardware alone, software alone, or a combination of both, and are not limited to any one of these.

[0055] The display control device 150 includes an image acquisition unit 152 that acquires images from the stereo camera 110, a buffer memory 154 that stores the data of the captured images, a content image acquisition unit 156 that acquires images of the content, an image data generation unit 160 that generates a display image, an additional image data storage unit 158 ​​that stores data of additional images to be superimposed on the display image, a correction rule storage unit 170 that stores correction rules for the display image, and an output control unit 172 that controls the output of images to the display panel 122.

[0056] The image acquisition unit 152 sequentially acquires frame data of left-view and right-view moving images captured by the stereo camera 110 at a predetermined rate. In see-through mode, as described above, the left-view and right-view images are used alternately for display, so the image acquisition unit 152 may select and acquire the necessary data at the time of image acquisition.

[0057] In particular, when the image acquisition unit 152 is provided in the content processing device 200, the communication bandwidth required for data transmission and reception with the head-mounted display 100 can be saved by selecting the necessary data. As a modification, the stereo camera 110 itself may alternately capture images from the left viewpoint and images from the right viewpoint for each frame. In this case, the image acquisition unit 152 does not need to select the images necessary for display. Furthermore, the communication bandwidth required for data transmission can be saved.

[0058] The image acquisition unit 152 sequentially stores the acquired image data in the buffer memory 154. The content image acquisition unit 156 acquires data for content images to be displayed during periods other than see-through mode. For example, the content image acquisition unit 156 draws computer graphics images representing the virtual world of an electronic game being run as the content application. Alternatively, the content image acquisition unit 156 may acquire data for computer graphics images or content such as movies from an external device.

[0059] The content image acquisition unit 156 sequentially stores the acquired content image data in the buffer memory 154. The image data generation unit 160 sequentially generates display image frames using the captured image or content image data stored in the buffer memory 154. In detail, the image data generation unit 160 includes a display target control unit 162, a correction unit 164, an additional image drawing unit 166, and a mode transition control unit 168.

[0060] The display target control unit 162 determines whether to display the left-eye image and the right-eye image alternately (hereinafter referred to as the "single-sided display state") or to display both simultaneously (hereinafter referred to as the "simultaneous display state"). If the former is to be used, it controls the switching of the images to be displayed. For example, the display target control unit 162 always maintains the simultaneous display state for content images, and the single-sided display state in see-through mode. Therefore, the display target control unit 162 obtains information from the content image acquisition unit 156 about whether or not content images are being supplied, and implements the see-through mode in the single-sided display state during periods when content images are not being supplied.

[0061] The correction unit 164 acquires the data of the image to be displayed from the buffer memory 154 and applies the necessary corrections for display. In see-through mode, as shown in Figure 4, the correction unit 164 applies corrections to the captured image to remove distortion caused by the lens of the stereo camera 110 and to add distortion for the eyepiece lens of the head-mounted display 100. The corrections performed by the correction unit 164 are not limited to these and may include any of the commonly performed corrections described above.

[0062] However, in this embodiment, by performing these corrections at once, the correction process is accelerated, and the memory area required to unfold the corrected image is minimized. For this reason, the correction amounts (pixel displacement amount and displacement direction) required for each correction, such as the displacement vector (Δx, Δy) described above, are calculated, and the final correction amount obtained by summing them up is derived as the integrated correction amount. Since the integrated correction amount is information for each pixel, it is stored in the correction rule storage unit 170 as a map associated with the pixel position information.

[0063] However, as will be described later, in practice, a color sample from the captured image is taken for each pixel of the displayed image, so information that identifies the corresponding position on the captured image from the pixels of the displayed image may be prepared as an integrated correction amount. If the captured image acquisition unit 152 acquires captured images from both the left and right viewpoints, the correction unit 164 selects one of the images to be used for display from the data stored in the buffer memory 154 and then performs the correction. The corrected images are sequentially stored in the frame memory 174 of the output control unit 172.

[0064] In the single-sided display state, the correction unit 164 either stores a black fill image in the image area of ​​the side to be hidden, or stores information indicating that the illumination of the display panel 122 should be temporarily suspended in that area. The additional image drawing unit 166 draws additional images to be superimposed on the displayed image. Additional images are images that represent additional information such as UI, warnings, various menus, and operating instructions, and are superimposed on the content image or captured image. The additional image drawing unit 166 draws additional images as needed, such as when explicitly called by the user, during the see-through mode period, or when a warning is necessary.

[0065] During the period in which the additional image is superimposed, the correction unit 164 excludes from the drawing target any areas of the original display image that will be obscured by the additional image. This eliminates unnecessary correction processing and enables faster display. The additional image drawing unit 166 also directly draws the distorted additional image onto the plane of the display image, which has distortion for the eyepiece, in order to conform to that distortion. This improves compatibility with modes in which multiple corrections are performed on the display image at once, and enables low-latency display even during the period in which the additional image is superimposed. Furthermore, it eliminates the need for memory space to unfold the distortion-free additional image.

[0066] The mode transition control unit 168 controls the image transition when switching between the content image display mode and the see-through mode. Specifically, the mode transition control unit 168 adjusts the brightness of both the content image and the captured image so that they crossfade. Crossfading is a technique used when switching between two types of images, in which the brightness of the image before the switch is gradually reduced until it disappears, while the brightness of the image after the switch is gradually increased until it appears.

[0067] However, when the content images are displayed simultaneously and the see-through mode is set to a single-sided display state, the mode transition control unit 168 introduces a time difference in the brightness changes of the left and right images displayed in the simultaneous display state. Specifically, the mode transition control unit 168 first changes the brightness of one of the display images for the left eye and the right eye displayed in the simultaneous display state alternately frame by frame, before the other, and replaces it with the image displayed in the single-sided display state. Then, in accordance with the brightness change of the image displayed in the single-sided display state, it applies a brightness change to the remaining image displayed in the simultaneous display state, creating the appearance of a crossfade.

[0068] This eliminates the period during which the left and right images and the image to be displayed in the single-sided display state must all be processed simultaneously, thereby suppressing the increased processing load during the transition process while ensuring that the crossfade is visible. Further details will be described later. The output control unit 172 reads the data of the display images for the left eye and right eye, drawn by the correction unit 164 and the additional image drawing unit 166, from the frame memory 174 and outputs them to the left and right areas of the display panel 122, respectively.

[0069] Figure 9 is a diagram illustrating the crossfade technique performed by the mode transition control unit 168. (a) shows an example of the brightness change of images before and after switching in a typical crossfade, with the horizontal direction representing time and the vertical direction representing brightness. The solid line 30 represents the brightness change of the image before switching, and the dashed line 32 represents the brightness change of the image after switching. Generally, as shown in the figure, the brightness of the image after switching is increased in parallel with the decrease in brightness of the image before switching, so that the disappearance of the former and the appearance of the latter are represented by a smooth change.

[0070] In this case, during transition period A, the images before and after the switch are displayed on top of each other, so they need to be processed simultaneously. Applying this to this embodiment, for example, when switching from a simultaneous display state to a single-eye display state, the brightness change of the solid line 30 is applied to both the left-eye image and the right-eye image. As a result, during transition period A, it becomes necessary to process three types of images simultaneously, including the image of the other side to be displayed in the single-eye display state, and the processing load temporarily increases.

[0071] (b) illustrates the brightness change of the crossfade in this embodiment. This example also assumes switching from a simultaneous display state to a single-sided display state, where the solid line 34a represents the brightness change of one of the simultaneously displayed images, the dashed line 34b represents the brightness change of the other image, and the dashed line 36 represents the brightness change of the image to be displayed in the single-sided display state. However, in practice, the brightness changes shown are applied alternately to the left-eye image and the right-eye image frame by frame. In the single-sided display state after the switch, as described above, the image that does not receive the change indicated by the dashed line 36 is hidden.

[0072] The mode transition control unit 168 first decreases the brightness of one of the two images displayed simultaneously on the left and right at the crossfade start time B (solid line 34a). Then, it waits until time C when that image is hidden, and then makes the image to be displayed in single-sided display mode appear and gradually increases its brightness (dashed line 36). At the same time, the brightness of the other image that was displayed simultaneously is also decreased from time C (dotted line 34b). The crossfade is completed at time D when that image is hidden and the image to be displayed in single-sided display mode reaches its original brightness.

[0073] In other words, the mode transition control unit 168 staggers the time periods for decreasing the brightness of the simultaneously displayed images on the left and right sides, and makes the image to be displayed in the single-sided display state appear in the area that was previously hidden. As a result, at any point during the transition period, only two types of images need to be processed, thus suppressing an increase in processing load. Furthermore, since the brightness of the remaining simultaneously displayed image is maintained until time C, as shown by the dashed line 34b, the user perceives that the images before and after the switch are cross-faded during the period from time C to D.

[0074] In this embodiment, a state where only one of the left or right images is displayed is permitted. Similarly, the illustrated crossfade utilizes a visual effect where, even if one of the simultaneously displayed images is hidden, the brightness of the remaining image is maintained, making it appear as if it is being displayed normally. This allows for smooth transitions without processing three types of images simultaneously. A similar effect can be achieved when switching from a single-image display state to a simultaneous display state by reversing the time axis.

[0075] It is desirable that the brightness changes shown in the diagram be achieved by the actual light emission of the display panel. If the pixel values ​​are changed linearly, the actual light emission may change nonlinearly due to the characteristics of the display panel, and the transition may not appear smooth. Therefore, a conversion rule for the pixel values ​​in the data that causes the brightness on the display to change linearly with respect to time is determined in advance and stored in the correction rule storage unit 170. The brightness on the display is suitably controlled by changing the pixel values ​​of the display image over time, for example, nonlinearly, according to the conversion rule in the correction rule storage unit 170.

[0076] Figure 10 schematically shows the changes in the displayed image when the brightness change shown in Figure 9(b) is applied. In displayed images 40a to 40e, the upper row represents odd-numbered frames, and the lower row represents even-numbered frames. In the simultaneous display state, as shown in displayed image 40a, the left-eye image and the right-eye image are displayed at their original brightness regardless of the order of the frames. In the example shown in the figure, a virtual world in which a character resides is represented.

[0077] When the mode transition control unit 168 starts crossfading at time B, the brightness of the left-eye image and the right-eye image are alternately selected and subjected to brightness reduction, as shown in display image 40b. Since the images to be targeted will later be replaced by the images displayed in the single-sided display state, the selection order is synchronized with the selection order of the display targets in the single-sided display state. Then, at time C, when the images that have been subjected to brightness reduction have been completely hidden, as shown in display image 40c, the mode transition control unit 168 starts brightness reduction of the remaining images.

[0078] Simultaneously, the mode transition control unit 168 starts displaying the image shown in the single-sided display state from a low brightness and gradually increases the brightness. This results in a state where the image in the simultaneous display state and the image in the single-sided display state are mixed on the left and right sides at a moderate brightness, as shown in display image 40d. Subsequently, the former is hidden, and after time D when the latter reaches its original brightness, the display becomes a complete single-sided display state, as shown in display image 40e.

[0079] Next, as outlined in Figure 4, the effect of the correction unit 164 performing multiple types of corrections simultaneously will be explained in more detail. Figure 11 illustrates a general processing procedure when generating a display image from a captured image. In Figures 11 and 12, the change in only one of the left or right images is shown as the display target for the checkerboard pattern. In the example in Figure 11, the captured image 46 has distortion due to the lens of the stereo camera 110. First, the distortion is removed from the captured image 46, and the resulting intermediate image 48 is loaded into the buffer memory 50 (S20).

[0080] Then, by applying a correction to the intermediate image 48 that compensates for the distortion caused by the eyepiece of the head-mounted display 100, the display image 52 is generated and output to the display panel 122 (S22). In this procedure, the degree of distortion in the captured image 46 is greater closer to the edges of the image. In an attempt to maintain the resolution near the center of the captured image 46, the peripheral areas are stretched, and as a result, the intermediate image 48 becomes a non-rectangular data that is larger in size than the captured image 46. The wider the field of view of the stereo camera 110, the larger the size of the intermediate image 48.

[0081] The buffer memory 50 needs to be prepared to correspond to the rectangle encompassing the intermediate image 48, resulting in a large amount of wasted space without image data, as shown in black in the figure. Furthermore, this procedure requires two samplings. Specifically, in S20, a pixel displacement vector (Δx, Δy) is used to remove distortion caused by the camera lens, and the color of a position on the captured image 46 shifted by (-Δx, -Δy) from each pixel of the intermediate image 48 is sampled.

[0082] In S22, a pixel displacement vector (Δx', Δy') is used to remove distortion caused by the eyepiece, and the color of a position on the intermediate image 48, shifted by (Δx', Δy') from each pixel of the displayed image 52, is sampled. In many cases, sampling requires interpolating the surrounding pixel values ​​to determine the color, so reading the surrounding pixel values ​​and performing interpolation calculations is required for each pixel in the corrected image at each stage, which increases the processing load.

[0083] Furthermore, the two-stage interpolation process results in two opportunities for the MTF (Modulation Transfer Function) to degrade, which can easily impair the resolution characteristics of the final displayed image 52. Also, with this method, the wider the field of view of the displayed image, the more significantly the load of data reading, writing, and calculations on the buffer memory 50 increases.

[0084] Figure 12 shows the processing procedure when the correction unit 164 generates a display image from a captured image in this embodiment. As described above, the correction unit 164 generates a display image 52 from the captured image 46 without going through an intermediate image (S30). Assuming the displacement vector shown in Figure 11, the correction unit 164 determines the pixel values ​​of the display image 52 by sampling the color of a position on the captured image 46 that is shifted by (Δx'-Δx, Δy'-Δy) from each pixel of the display image 52.

[0085] In practice, the correction amount (Δx'-Δx, Δy'-Δy) is derived as an integrated correction amount by summing up displacements for various corrections, including not only the removal of distortion caused by the camera lens and the addition of distortion for the eyepiece, but also the angle of view correction as described above. The integrated correction amount is stored in the correction rule storage unit 170 as a map corresponding to each pixel in the image plane of the displayed image 52. Taking into account the chromatic aberration of the eyepiece, the integrated correction amount is derived individually for the three primary colors: red, green, and blue. As a result, in the displayed image 52, the image is shifted in the red, green, and blue planes, and an image without shift is visible when viewed through the eyepiece.

[0086] This procedure, compared to the method shown in Figure 11, eliminates the need for buffer memory 50, saving storage space and reducing the load on calculations related to data reading, writing, and interpolation. As a result, even with a wide-angle display image 52, the increase in required resources can be suppressed, enabling low-latency display. Furthermore, the size of the image processing integrated circuit 120 and other components built into the head-mounted display 100 can be made more compact, reducing power consumption and heat generation, thus improving the wearing comfort of the head-mounted display 100. In addition, reducing the number of sampling cycles makes it easier to maintain the resolution characteristics of the displayed image.

[0087] Figure 13 schematically shows the state in which the additional image rendering unit 166 superimposes an additional image onto the displayed image. In this example, an additional image is superimposed that shows the operation method for switching the display to the content image represented in VR during the period when the see-through mode is implemented in one-sided display mode. Note that in the figure, only one of the images for the left eye and one for the right eye is shown, but the other image is not displayed in the one-sided display state.

[0088] In both (a) and (b), additional images 54a and 54b, which are horizontally elongated black rectangles with blurred edges and text displayed in white, are superimposed on the display image 56 generated from the captured image. By superimposing the additional images 54a and 54b in this way, the correction unit 164 can omit drawing the areas of the display image 56 that are obscured by the overlapping additional images 54a and 54b. This eliminates unnecessary processing for reading data from the buffer memory 154 storing the captured image and for sampling.

[0089] On the other hand, the positions in which the additional images 54a and 54b are superimposed differ between (a) and (b). As shown in Figure 12, in this embodiment, the display image is generated from the captured image without going through a distortion-free intermediate image. For this reason, the additional image drawing unit 166 directly draws the distorted additional images 54a and 54b onto the display image 56 which has distortion for the eyepiece. As is clear from Figure 13, the degree of distortion of the additional images 54a and 54b differs depending on the superimposing position, which also results in a difference in the processing load on the additional image drawing unit 166.

[0090] Specifically, as shown in (a), the further away from the center of the display image 56 (the position of the optical axis of the eyepiece), the greater the degree of distortion due to distortion and chromatic aberration in the additional image 54a, and the heavier the rendering load. Therefore, the additional image rendering unit 166 superimposes the additional image only within a predetermined range from the center of the display image 56. For example, as shown in (b), by aligning the center of the display image 56 with the center of the additional image 54b, it becomes possible to render a nearly rectangular image with less load, regardless of the primary colors. By limiting the superimposing area in this way, it is possible to avoid generating extra load for rendering the additional image.

[0091] Figure 14 is a diagram illustrating a method by which the additional image drawing unit 166 efficiently draws additional images. (a) illustrates a typical procedure for superimposing additional images. In this case, first, a distortion-free additional image 60 is drawn into a buffer memory and superimposed onto a distortion-free display image 62. This display image 62 corresponds to the intermediate image 48 in Figure 11 and is stored separately in a buffer memory. Then, the entire distortion-free display image 62 with the superimposed additional image is distorted for the eyepiece lens to generate the display image 64.

[0092] In this processing procedure, the rectangular additional image 60 can be simply resized and pasted onto the distortion-free display image 62 without any deformation. Therefore, it is sufficient to represent the additional image 60 with two polygons whose vertices are the four corners of the image plane. On the other hand, this requires extra buffer memory to store the distortion-free additional image 60 and sampling processing from it.

[0093] (b) shows how the additional image drawing unit 166 of this embodiment directly draws a distorted additional image onto the display image. In this case, the image plane of the additional image is divided into a number of polygons such that at least one vertex is located in the internal region. The left side of the figure illustrates the model data 66 of the additional image, which represents the distribution of vertices of the polygons thus divided. The model data 66 is stored in the additional image data storage unit 158.

[0094] Furthermore, the additional image data storage unit 158 ​​also stores information representing the relationship between each vertex of the model data 66 and its position coordinate when superimposed on the distorted display image 68, as shown by arrow 70, for example. This allows the distortion of the additional image to be controlled with a granularity corresponding to the spacing between vertices. In other words, the additional image drawing unit 166 can directly impart distortion according to the position coordinates of the vertices by drawing the additional image on the plane of the distorted display image 68 in polygon units.

[0095] The more vertices are increased and the smaller the spacing between them, the more accurately the distortion can be represented. Also, as shown in the figure, when blurring the edges of an added image, the direction of the blur gradient becomes uneven due to the distortion of the added image itself. By preparing model data 66 containing a large number of vertices as shown in the figure, it becomes possible to set the blur on a vertex-by-vertex basis, making it easy to achieve natural processing according to the distortion. In this embodiment, the processing of images set at the vertices of polygons is not limited to blurring.

[0096] According to the embodiment described above, in the head-mounted display, for each frame of the moving image to be displayed, either the image for the left eye or the image for the right eye is displayed alternately. The image on the side that is not displayed is filled in by the user's brain based on the latest image of the other side and the movement of the image up to that point, so that it is perceived as if both images were being displayed simultaneously. As a result, the amount of data that needs to be processed is significantly reduced, while images can be recognized with the same smoothness as the original performance of the head-mounted display.

[0097] This allows the system to display real-world images with minimal delay in response to the user's head movements, especially in see-through mode, which instantly displays images taken in front of the user. This reduces the likelihood of discomfort and minimizes the possibility of motion sickness. In see-through mode, the system corrects the image directly to the display without going through intermediate images, based on an integrated correction amount which is the sum of multiple correction amounts to be applied to the captured image. This minimizes the required buffer memory and reduces the load on memory for reading, writing, and sampling calculations.

[0098] Furthermore, when switching between a mode that displays both images simultaneously and a mode that displays one image at a time alternately, a crossfade is used to make the transition appear smooth. In this process, the timing of the brightness changes for the left and right images, which are the objects of simultaneous display, is staggered, allowing the crossfade to be perceived without increasing the number of images that need to be processed simultaneously.

[0099] Furthermore, additional images representing extraneous information such as UI elements are superimposed on the display image. In this process, areas of the display image that are obscured by the additional images are excluded from the rendering target, thus reducing unnecessary processing. Additionally, by finely dividing the plane of the additional image into polygons and creating a large number of vertices, the distorted additional image is drawn directly onto the distorted display image plane. In this case as well, the required buffer memory can be minimized, and the load on memory for reading, writing, and sampling calculations can be reduced.

[0100] By at least one of the above embodiments, the visual changes are minimized, and the processing load and data transmission volume are significantly reduced compared to displaying both left and right images simultaneously. As a result, the possibility of image generation not keeping up with the frame display cycle is reduced, and high-quality images can be displayed stably. In addition, power consumption can be reduced, extending the battery life and suppressing heat generation. Furthermore, since operation is possible even with a relatively small built-in integrated circuit, the device can be made lighter, improving the wearing comfort of the head-mounted display.

[0101] The present invention has been described above based on embodiments. The embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their components and processing processes, and that such modifications also fall within the scope of the present invention. [Explanation of Symbols]

[0102] 10 Image display system, 100 Head-mounted display, 110 Stereo camera, 120 Integrated circuit for image processing, 122 Display panel, 150 Display control device, 152 Captured image acquisition unit, 154 Buffer memory, 156 Content image acquisition unit, 158 Additional image data storage unit, 160 Image data generation unit, 162 Display target control unit, 164 Correction unit, 166 Additional image drawing unit, 168 Mode transition control unit, 170 Correction rule storage unit, 172 Output control unit, 174 Frame memory, 200 Content processing device.

Claims

1. A display control device that displays left-eye and right-eye display images, which constitute the frames of a moving image, in the left and right regions of a display panel, respectively. An image data generation unit that alternately generates either the left-eye or right-eye display image for each frame of the moving image, The display panel includes an output control unit that controls the generated display image to be displayed alternately in corresponding areas of the left and right regions according to the frame period of the moving image, while the other region is hidden. A display control device characterized by being equipped with

2. The system further includes an image acquisition unit that acquires stereo video data captured by a stereo camera from left and right viewpoints. The display control device according to claim 1, characterized in that the image data generation unit alternately generates the left-eye and right-eye display images by alternately using images captured from the left and right viewpoints that constitute the frames of the stereo motion image.

3. The image acquisition unit alternately acquires images from the left and right viewpoints that constitute the frames of the stereo motion image. The display control device according to claim 2, characterized in that the image data generation unit generates the display image using the acquired captured image.

4. The system further includes a correction rule storage unit that stores, for each pixel, the amount of correction that integrates multiple types of corrections necessary to generate the display image from the captured image. The display control device according to claim 2 or 3, characterized in that the image data generation unit generates the display image without going through an intermediate image by correcting the captured image with the amount of the integrated correction.

5. The image data generation unit generates the one display image for each frame in a one-sided display state in which one display image is displayed in the corresponding area of ​​the left and right regions of the display panel, while the other region is hidden, and generates the left and right display images for each frame in a simultaneous display state in which the left and right eye display images are displayed simultaneously in the left and right regions of the display panel. The image data generation unit includes a display target control unit that determines the switching between the one-sided display state and the simultaneous display state, The display control device according to any one of claims 2 to 4, wherein the image data generation unit provides a crossfade process that simultaneously decreases the brightness of the display image before switching and increases the brightness of the display image after switching, in response to the decision to switch by the display target control unit, and in switching from the simultaneous display state to the single-sided display state, it alternately changes the brightness of one of the display images for the left eye and the right eye displayed in the simultaneous display state on a frame-by-frame basis, before the other, and replaces it with the image displayed in the single-sided display state.

6. The display control device according to claim 5, characterized in that the image data generation unit switches from the simultaneous display state to the one-sided display state when switching the display from a content image, which is an image other than a real-time image in real space, to the captured image.

7. The display control device according to claim 5 or 6, characterized in that the image data generation unit changes the pixel values ​​in the data of the display image over time based on a rule that linearly changes the brightness on the display with respect to time when switching between the simultaneous display state and the single-sided display state.

8. The display control device according to any one of claims 1 to 7, characterized in that the image data generation unit directly draws an additional image representing additional information, which is distorted in the opposite direction to the distortion of the eyepiece lens during viewing, within a predetermined range from the center of the display image, after applying position-appropriate distortion.

9. The system further includes an additional image data storage unit that stores data of a plane composed of multiple polygons having at least one vertex in its internal region, as model data for the additional image. The display control device according to claim 8, characterized in that the image data generation unit draws the additional image having distortion for each polygon based on the position coordinates of the vertices set on the plane of the display image.

10. The display control device according to claim 8 or 9, characterized in that the image data generation unit excludes the area of ​​the display image that is obscured by the additional image from the object of drawing.

11. A display control device according to any one of claims 1 to 10, A stereo camera that captures stereo moving images to be displayed as the aforementioned display image, The aforementioned display panel, A head-mounted display characterized by having the following features.

12. The head-mounted display according to claim 11, characterized in that the stereo camera alternately captures images from the left and right viewpoints that constitute the frames of the stereo motion image, frame by frame, and supplies them to the display control device.

13. A display control device that displays the left-eye and right-eye display images that make up the frames of a moving image in the left and right areas of the display panel, respectively, The steps include: generating either the left-eye or right-eye display image alternately for each frame of the moving image; The display panel is controlled to alternately display the generated display image in corresponding areas of the left and right regions according to the frame period of the moving image, while hiding the other region. A display control method characterized by including the following.

14. A computer that displays the left-eye and right-eye display images that make up the frames of a moving image in the left and right areas of the display panel, respectively. A function to alternately generate either the left-eye or right-eye display image for each frame of the moving image, The display panel has a function to control the display so that the generated display image is alternately displayed in the corresponding left and right regions according to the frame period of the moving image, while the other region is hidden. A computer program characterized by achieving this.

Images