Image display device and image display method

Abstract

To cause a high-definition and wide-field image to be visually recognized easily without a sense of discomfort.SOLUTION: In an image plane 200, a region 212a including a central portion is represented at a higher resolution than that of a region 214a outside the region 212a. The region 212a is represented with a laser scanning system that projects an image by reflecting a beam from the laser beam source 220 using a mirror 222 and carrying out two-dimensional scanning. The region 212a uses when a gazing point 202a is at the center of the image as a reference and controls the amplitude of an angle of the mirror 222 so that a region 212b is extended according to movement from the center.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image display device and an image display method that allow a user to view an image. [Background technology]

[0002] Image display systems that allow users to view a target space from any viewpoint are becoming widespread. For example, a system has been developed that displays a panoramic image on a head-mounted display, and when a user wearing the head-mounted display rotates his or her head, a panoramic image is displayed according to the line of sight. Using a head-mounted display can enhance the immersive feeling of the image and improve the operability of applications such as games. A walk-through system has also been developed that allows a user wearing a head-mounted display to virtually walk around a space displayed as an image by physically moving around.

[0003] To improve the quality of the video experience, it is necessary to express images with a wide field of view in high definition. However, the larger the resolution and the viewing angle, the larger the data size of the image to be processed, and the longer it takes to process and transfer, which makes it more likely that delays will occur before the image is displayed. To address this issue, a technology has been proposed that takes advantage of the visual characteristics of humans, whose eyesight decreases from the center to the edge of the field of view, by setting a difference in image resolution between the central area and its outer area, thereby reducing unnecessary processing while maintaining the visual image quality (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent No. 10,140,695 Summary of the Invention [Problem to be solved by the invention]

[0005] In the field of image display technology, not limited to head mounted displays, it is always a common challenge to be able to display high-resolution images with a wide field of view and low latency. For example, in the case of the technology of Patent Document 1, by providing separate displays for the central area and other areas and providing a clear difference in display resolution, it becomes easy to appropriately distribute processing resources. On the other hand, when images with different resolutions that are individually displayed are synthesized, the boundary line may look unnatural and the user may feel uncomfortable. To solve this problem, it is possible to smoothly connect the resolutions on the data, but separate image processing is required. Especially when a high-resolution area is linked to the user's gaze point, it is thought that a significant processing time will occur due to cutting out the changing area and adjusting the resolution.

[0006] The present invention has been made in view of the above problems, and has as its object to provide a technique that enables high-definition, wide-field-of-view images to be easily viewed without any sense of discomfort. [Means for solving the problem]

[0007] One aspect of the present invention relates to an image display device, which is characterized by including a central image generating unit that generates a central image representing a central portion of a display image in an image plane, a peripheral image generating unit that generates a peripheral image representing an area outside the central image in the display image, a central image output unit that displays the central image by a laser scanning method in which a laser beam representing pixels is two-dimensionally scanned by mirror reflection to project an image, a peripheral image output unit that displays the peripheral image, and an image synthesis unit that synthesizes the central image and the peripheral image to be visually recognized.

[0008] Another aspect of the present invention relates to an image display method, which includes the steps of generating a central image representing a central portion of a display image in an image plane, generating peripheral images representing an outer region of the central image from the display image, displaying the central image by a central image output unit of a laser scanning type that projects an image by two-dimensionally scanning laser light representing pixels by mirror reflection, displaying the peripheral image by a peripheral image output unit, and synthesizing the central image and the peripheral image for visual recognition.

[0009] Any combination of the above components, and any transformation of the present invention into a method, device, system, computer program, or recording medium having a computer program recorded thereon, are also effective aspects of the present invention. Effect of the Invention

[0010] According to the present invention, a high-definition, wide-field image can be easily viewed without any sense of discomfort. [Brief description of the drawings]

[0011] [Figure 1] 1 is a diagram for explaining the relationship between human visual characteristics and a display format according to an embodiment of the present invention. [Diagram 2] 1A and 1B are diagrams for explaining the relationship between a laser scanning display mechanism and an area on an image, which is employed in the present embodiment. [Diagram 3] 1 is a diagram showing an example of the appearance of a head mounted display according to an embodiment of the present invention; [Figure 4] FIG. 2 is a diagram showing an internal circuit configuration of the head mounted display according to the present embodiment. [Diagram 5] FIG. 2 is a diagram showing a functional block configuration of the head mounted display according to the present embodiment. [Figure 6] 10A and 10B are diagrams illustrating changes in a central image and a peripheral image in response to a movement of a gaze point in the present embodiment. [Figure 7]11A and 11B are diagrams for explaining changes in resolution relative to the size of a central image in the present embodiment. [Figure 8] 10 is a flowchart showing a processing procedure for displaying an image by the head mounted display of the present embodiment. [Figure 9] 1A and 1B are diagrams illustrating an example of the structure of an image synthesis unit that synthesizes a central image and a peripheral image and visually recognizes the synthesized image in the head mounted display of the present embodiment. [Figure 10] 11 is a diagram showing another example of the structure of the image synthesis unit that synthesizes a central image and a peripheral image and visually recognizes the synthesized image in the head mounted display of the present embodiment. FIG. [Figure 11] 11 is a diagram showing another example of the structure of the image synthesis unit that synthesizes a central image and a peripheral image and visually recognizes the synthesized image in the head mounted display of the present embodiment. FIG. [Figure 12] 11 is a diagram showing another example of the structure of the image synthesis unit that synthesizes a central image and a peripheral image and visually recognizes the synthesized image in the head mounted display of the present embodiment. FIG. [Figure 13] 11 is a diagram showing another example of the structure of the image synthesis unit that synthesizes a central image and a peripheral image and visually recognizes the synthesized image in the head mounted display of the present embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] FIG. 1 is a diagram for explaining the relationship between human visual characteristics and the display format of this embodiment. First, assume that the user's gaze point 202, indicated by a circle, is present near the center of an image plane 200 shown in (a). In terms of the visual characteristics of a typical person, an area 204 that corresponds to within 5° of the line of sight from the pupil to the gaze point as the central axis is called the discrimination visual field, and visual functions such as visual acuity are excellent. An area 206 that corresponds to within approximately 30° in the horizontal direction and approximately 20° in the vertical direction is called the effective visual field, and information can be instantly received only by eye movement.

[0013] Furthermore, the area 208 corresponding to within 60-90° in the horizontal direction and 45-70° in the vertical direction is the stable fixation field, while the area 210 corresponding to within 100-200° in the horizontal direction and 85-130° in the vertical direction is the auxiliary field, and the ability to distinguish information decreases as the area moves away from the fixation point 202. In light of this characteristic, in this embodiment, as shown in (b), the basis is to display an area 212 of a predetermined range including the fixation point 202 with a higher resolution than an area 214 outside it. To achieve this, the areas 212 and 214 are displayed using different display mechanisms, and are ultimately viewed in a synthesized state.

[0014] In this embodiment, a laser scanning method is adopted as a display mechanism for the area 212 including the gaze point 202. The laser scanning method is a technology for forming an image on a projection surface by two-dimensionally scanning laser light corresponding to pixels using a deflection mirror. For example, a technology for converging laser light on a user's pupil and projecting an image on the retina is being applied mainly to wearable displays (see, for example, International Publication No. 2009 / 066465). In addition, a small projector that projects an image on an external screen or the like has also been put to practical use (see, for example, Japanese Patent Application Laid-Open No. 2017-83657).

[0015] 2 is a diagram for explaining the relationship between the laser scanning display mechanism and the area on the image, which is adopted in this embodiment. The upper parts of (a) and (b) are schematic diagrams of the laser scanning display mechanism as seen from the side. The laser light source 220 outputs laser light containing red, blue, and green components. The laser light is reflected by the mirror 222 and projected onto the projection surface (image plane 200).

[0016] By oscillating the mirror 222 around two axes, the laser light is scanned two-dimensionally on the projection surface, and an image is formed in which the laser light output at each time is used as pixels. In this example, the mirror 222 is oscillated symmetrically with respect to the posture in which the reflected laser light reaches the center of the image plane 200. As a result, the center of the area 212a expressed by the laser scanning method coincides with the center of the image plane. However, this is not intended to limit the present embodiment.

[0017] The figure also assumes a head-mounted display, with the user's eye 224 shown on the opposite side of a projection surface such as a transparent screen. However, as mentioned above, direct imaging on the retina is also possible, and a screen is not essential. For example, a MEMS (Micro Electro Mechanical Systems) mirror is used as the mirror 222. A MEMS mirror is a small, low-power device that can precisely control angular changes around two axes by electromagnetic drive. However, there are no particular limitations on the mirror drive method.

[0018] In this embodiment, the size of the area represented by the laser scanning method is expanded as the gaze point moves away from the center of the image plane 200 by changing the amplitude of the angle of the mirror 222 in response to the movement of the gaze point. First, in the case shown in (a), as shown in FIG. 1, the gaze point 202a is at the center of the image plane 200. The amplitude of the angle of the mirror 222 in this case, and therefore the amplitude of the scanning angle of the laser light, is set to a reference value θ std The reference value θ std may be a predetermined minimum value.

[0019] On the other hand, in the case shown in (b), the gaze point 202b moves to the lower left direction of the image plane 200. At this time, the amplitude θ of the angle of the mirror 222 is set as θ std Specifically, the angle amplitude θ of the mirror 222 is increased to expand the area 212b so that the edge of the area 212b represented by the laser scanning method is kept at a certain distance from the gaze point 202b. For example, when the area 212a in the reference state shown in (a) is expanded vertically by n times, the angle amplitude θ can be determined as follows. tanθ=n×tan(θ std ) Naturally, the amplitude of the angle of the mirror 222 in the lateral direction can also be determined in a similar manner.

[0020] In the display method using laser scanning, when the frame rate is fixed, the scanning angle and the resolution are inversely proportional to each other. That is, when the area 212a is expanded by n times, the number of pixels per unit angle in the expansion direction becomes 1 / n. On the other hand, the resolution of the outer areas 214a and 214b displayed by another display mechanism can be constant. According to such control, first, the fixation point is always inside the areas 212a and 212b displayed by the laser scanning method, so that the boundary part where the resolution difference occurs is difficult to be fixed. Also, the closer the fixation point is to the edge of the image plane 200, the smaller the resolution difference between the areas becomes, so the boundary becomes less noticeable. Therefore, even if the fixation point exceeds the area that can be displayed by laser scanning, the boundary line becomes difficult to recognize.

[0021] In the reference state shown in (a), the reference value θ is set according to general visual characteristics, such as by setting the edge of the area 212a displayed by the laser scanning method to be at least outside the effective visual field. std If the difference in resolution is determined, the boundary between the two images is difficult to see even if there is a certain degree of difference in resolution. This increases the resolution of the area 212a, and allows a high-quality image to be displayed without discomfort. The above-mentioned effects are naturally obtained by controlling the amplitude of the angle of the mirror 222 in the laser scanning display method, and therefore the increase in the processing load can be suppressed.

[0022] Although the display device to which this embodiment can be applied is not particularly limited, hereinafter, a head mounted display will be described as an example. Fig. 3 shows an example of the appearance of the head mounted display of this embodiment. In this example, the head mounted display 100 is composed of an output mechanism unit 102 and a wearing mechanism unit 104. The wearing mechanism unit 104 includes a wearing band 106 that goes around the head when worn by the user and fixes the device. The output mechanism unit 102 includes a housing 108 shaped to cover the left and right eyes when the user wears the head mounted display 100, and includes inside a laser scanning type display mechanism as described above and a mechanism for displaying images of other areas.

[0023] The housing 108 further includes a mechanism for synthesizing the two displayed images, and an eyepiece for expanding the viewing angle. Stereoscopic vision may be achieved by displaying a stereo image having parallax to each of the left and right eyes. The housing 108 further includes a gaze point detector for detecting the user's gaze point on the displayed image.

[0024] The head mounted display 100 may further include speakers or earphones at positions corresponding to the user's ears when worn. In this example, the head mounted display 100 includes a stereo camera 110 on the front surface of the housing 108, and captures video of the surrounding real space in a field of view corresponding to the user's line of sight. Furthermore, the head mounted display 100 may include any of various sensors inside or outside the housing 108 for deriving the movement, attitude, position, etc. of the head mounted display 100, such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, and a GPS.

[0025] 4 shows an internal circuit configuration of the head mounted display 100. The head mounted display 100 includes a CPU (Central Processing Unit) 120, a GPU (Graphics Processing Unit) 122, and a main memory 124. These components are connected to each other via a bus 140. An input / output interface 138 is further connected to the bus 140. A communication unit 126, a motion sensor 128, a stereo camera 110, a gaze point detector 130, a first display unit 132, a second display unit 134, and an audio output unit 136 are connected to the input / output interface 138.

[0026] The CPU 120 executes an operating system stored in the main memory 124 to control the entire head mounted display 100. The CPU 120 also executes various programs downloaded via the communication unit 126 and plays electronic content. The GPU 122 has a geometry engine function and a rendering processor function, and draws a display image according to a drawing command from the CPU 120 and outputs it to the first display unit 132 and the second display unit 134.

[0027] The main memory 124 is composed of a RAM (Random Access Memory) and stores programs and data necessary for processing by the CPU 120 and the like. The communication unit 126 is a network interface such as a wired or wireless LAN or Bluetooth (registered trademark), and realizes communication with an external device. The motion sensor 128 is composed of at least one of sensors such as an acceleration sensor, a gyro sensor, a geomagnetic sensor, and a GPS, and measures the position, posture, and movement of the head mounted display 100 and, ultimately, the head of the user wearing it.

[0028] As shown in FIG. 3, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints in a field of view corresponding to the user's viewpoint. If the moving images captured by the stereo camera 110 are instantly displayed on the first display unit 132 and the second display unit 134, the state of the real space in the direction in which the user faces can be seen as it is, so-called video see-through can be realized. Furthermore, if a virtual object is drawn on the image of a real object captured in the captured image, augmented reality can be realized. Moreover, the position and posture of the head mounted display 100 and, ultimately, the user's head can be tracked by analyzing the images captured by the stereo camera 110 using a known technique such as Visual SLAM (Simultaneous Localization and Mapping).

[0029] The user's head movement may be obtained with higher accuracy by integrating the analysis result of the captured image and the measurement result of the motion sensor 128. This allows a display image in a field of view according to the head movement to be generated with high accuracy, enhancing the sense of immersion in the video world. In addition, the user's head movement can be accepted as a user operation on the content, and processing can be branched accordingly.

[0030] The gaze point detector 130 detects, at a predetermined rate, the position coordinates of the gaze point of the user who is viewing the images displayed by the first display unit 132 and the second display unit 134. The gaze point detector 130 is composed of, for example, a mechanism for irradiating the eyeball with infrared rays and a camera for capturing the reflected light, and tracks the point on the image at which the user is gazing by identifying the direction of the pupil from the captured image. In addition, various techniques have been put into practical use as a means for detecting the gaze point, and any of them may be adopted in this embodiment.

[0031] The first display unit 132 is configured with a laser scanning display mechanism described in FIG. 2, and projects and displays an image of an area including the center of the image plane. Hereinafter, the partial image displayed by the first display unit 132 is referred to as a "center image". The second display unit 134 displays an image of an area outside the center image. Hereinafter, the partial image displayed by the second display unit 134 is referred to as a "peripheral image". The display method of the second display unit 134 is not particularly limited, and may be a display panel consisting of a two-dimensional array of light-emitting elements such as a liquid crystal panel or an organic EL panel, or may be a laser scanning display mechanism similar to the first display unit 132. In any case, the second display unit 134 displays the peripheral image at a lower resolution than the center image displayed by the first display unit 132.

[0032] The first display unit 132 and the second display unit 134 each display the center image and the peripheral image generated by the GPU 122 at a predetermined rate. The images displayed by the first display unit 132 and the second display unit 134 are synthesized by a synthesis mechanism described later, and are visually recognized by the user as one display image. As described above, stereoscopic vision may be realized by displaying stereo images to the left and right eyes. In this case, the stereo image is a pair of images synthesized from the center image and the peripheral image. The audio output unit 136 is composed of speakers or earphones provided at positions corresponding to the user's ears when the head mounted display 100 is worn, and allows the user to hear audio.

[0033] Some of the functions of the head mounted display 100 shown in the figure may be provided in an external device that has established communication with the head mounted display 100. For example, at least some of the processes, such as determining an appropriate field of view and generating an overall image, controlling the boundary between the central image and the peripheral image according to the gaze point, and generating data of the central image and the peripheral image, may be performed by an external image generating device or an image providing server connected via a network.

[0034] Fig. 5 shows the configuration of functional blocks of the head mounted display 100. Each functional block shown in the figure can be realized in terms of hardware by the various circuits shown in Fig. 4, and in terms of software by a program that performs various functions such as an information processing function, an image processing function, a display function, and a communication function, which are loaded from a recording medium to the main memory 124. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by only hardware, only software, or a combination thereof, and are not limited to any one of them.

[0035] The head mounted display 100 comprises an image data acquisition unit 50 that acquires data of the image to be displayed, a gaze point acquisition unit 52 that acquires the user's gaze point for the displayed image, a center image size control unit 54 that controls the size of the center image, a center image generation unit 56 that generates the center image, a peripheral image generation unit 58 that generates the peripheral image, a center image output unit 60 that outputs the center image as the display target, a peripheral image output unit 62 that outputs the peripheral image as the display target, and an image synthesis unit 64 that causes the center image and peripheral image to reach the user's eyes in a synthesized state.

[0036] The image data acquisition unit 50 acquires data necessary for generating a moving image or still image to be displayed. The content of the image is not particularly limited, and may be any of game images, movies, live images, recorded images, animations, photos, environmental images, websites, documents, digital signage, and the like. It may also be an image captured by the stereo camera 110, or an image that has been processed or has a virtual object drawn thereon. Depending on the content of such an image, the image data acquisition unit 50 may acquire data from various sources.

[0037] For example, the image data acquisition unit 50 may acquire moving image data streamed by an external image generating device or server, or may draw or play an image using data stored in an internal storage device. The gaze point acquisition unit 52 includes the gaze point detector 130 shown in Fig. 4, and acquires the position coordinates of the user's gaze point relative to the displayed image at a predetermined rate. Note that in this embodiment, the displayed image may be an image that is within the user's field of vision, and the projection of laser light is also considered to be "display."

[0038] The center image size control unit 54 controls the size of the center image according to the position of the user's gaze point. Specifically, as described in FIG. 2, the gaze point is in the center of the image plane as a reference state, and the range of the center image is expanded so as to include the gaze point as the gaze point moves. As a result, the center image becomes wider the farther the gaze point is from the center, and narrower the closer the gaze point is to the center. The center image size control unit 54 determines the size of the center image at a predetermined rate or as needed, and supplies the information to the center image generation unit 56, the peripheral image generation unit 58, and the center image output unit 60.

[0039] The center image generating unit 56 includes the GPU 122 shown in Fig. 4, and generates a center image by acquiring necessary data from the image data acquiring unit 50. The peripheral image generating unit 58 also includes the GPU 122 shown in Fig. 4, and generates peripheral images by acquiring necessary data from the image data acquiring unit 50. Here, the peripheral image is an image in which the area of ​​the center image is blackened (invalidated) out of the entire displayed image. The boundary between the center image and the peripheral image is updated as appropriate according to information from the center image size control unit 54.

[0040] 4, and displays the center image generated by the center image generating unit 56 at a predetermined rate by laser scanning. In detail, the center image output unit 60 determines the amplitude of the mirror angle around each of the two axes according to the size of the center image notified by the center image size control unit 54. Then, the laser light representing the color of each pixel of the center image is two-dimensionally scanned at a scanning angle corresponding to the amplitude, thereby projecting the center image onto the user's retina or a screen in front of the user's eyes.

[0041] The peripheral image output unit 62 includes the second display unit 134 shown in FIG. 4, and displays the peripheral image using a display panel made of light-emitting elements or laser scanning. When using a display panel, the area of ​​the central image is not illuminated. When using a laser scanning method, laser light is not output to the area of ​​the central image. In either case, the density of pixels for displaying the peripheral image is constant regardless of the size of the central image.

[0042] The image synthesis unit 64 is an optical system that synthesizes the displayed central image and peripheral image so that they reach the eye as a single image. In other words, the image synthesis unit 64 is a hardware structure that synthesizes the central image and peripheral image without misalignment and can take various forms depending on the positional relationship between the first display unit 132 and the second display unit 134, the arrangement required for the gaze point detector 130, and the like. Specific examples will be described later.

[0043] FIG. 6 shows a schematic diagram of the change in the central image and peripheral image in response to the movement of the gaze point. The top part of the figure shows the entire displayed image, the middle part the central image, and the bottom part the peripheral image. Note that the central image and peripheral image show the range of the area relative to the entire image by making the outside of the range dark, and do not show the size of the image as data. When the gaze point 70a is at the center of the entire image as shown in (a), the central image is a predetermined size (X std ,Y std ) area, and the surrounding image is the area excluding the central image area from the entire image.

[0044] Here, the size of the central image (X std ,Y std ) is desirably determined based on the relationship between the angle when the line of sight is the central axis and visual acuity, as explained in Fig. 1. On the other hand, as shown in (b), when the gaze point 70b is displaced from the center on the image plane, the central image is expanded to include it. For example, as shown in the figure, when the displacement vector of the gaze point from the image center is (Δx, Δy), the size (X, Y) of the central image is determined as follows. X=2*(|Δx|+m x ) Y = 2*(|Δy|+m y )

[0045] where m x , m y is the margin given to the distance between the fixation point and the two sides of the edge of the central image that are closest to the fixation point. In other words, the edge of the central image is always at least (m x , m y ) to the outside. (m x ,m y) should also have a decision rule prepared in advance based on the relationship between the angle of the line of sight and visual acuity. For example, (m x ,m y )=(X std / 2,Y std / 2) or (m x , m y ) may be a function of the displacement vector (Δx, Δy). In this case, the peripheral image is also the area excluding the area of ​​the central image from the entire image.

[0046] The center image size control unit 54 may update the size of the center image as needed in response to a change in the displacement vector of the gaze point, or may update the size of the center image in stages, such as when the displacement vector changes by a threshold value or more. When an image is viewed stereoscopically, the center image generation unit 56 and the peripheral image generation unit 58 generate center images and peripheral images as shown in the figure for both the left eye image and the right eye image. The center image generation unit 56 and the peripheral image generation unit 58 generate center images and peripheral images with distortion in the opposite direction to the distortion aberration and chromatic aberration caused by the eyepiece lens of the head mounted display 100, so that an image without distortion or color shift is viewed when viewed through the eyepiece lens. Depending on the configuration of the first display unit 132, the shape of the center image is not limited to a rectangle, and the shape of the black area of ​​the peripheral image naturally depends on the shape of the center image.

[0047] 7 is a diagram for explaining the change in resolution with respect to the size of the central image. The upper part of the figure shows the entire displayed image, with (a) being the reference state with the gaze point 72a at the center, and (b) being the state in which the gaze point 72b is displaced from the center on the image plane. The lower part shows the distribution of resolution in the horizontal directions AA' and BB' passing through the gaze points 72a and 72b on the image plane. Compared to the central image 74a of the smallest size in the reference state shown in (a), when the central image 74b expands as shown in (b), the resolution of that part decreases.

[0048] Here, "resolution" does not refer to the fineness of the image data, but rather to the number of physical images depicted per unit area (or unit angle), in other words, pixel density. As mentioned above, laser scanning display mechanisms have the characteristic that the smaller the projection area, the higher the resolution. For example, a device that can display an image of 600 pixels within a horizontal viewing angle of 30° has an angular resolution of 20 ppd (pixels per degree). If the amplitude of the mirror angle is halved in this state, an image of 600 pixels will also be displayed within a viewing angle of 15°, so the angular resolution will be 40 ppd.

[0049] Due to this characteristic, the resolution of the smallest-sized central image 74a shown in (a) is maximized, and the larger the central image 74b is, the lower the resolution becomes as shown in (b). Because the user views the image with the fixation points 72a and 72b at the center, the resolution decreases as the user moves the line of sight toward the edge of the image, and the image appears to smoothly connect with the peripheral image. In addition, the peripheral image continues to be displayed at a constant resolution that is suitable for viewing, so the range of the field of view is maintained. As a result, no matter how the fixation point is displaced, the sense of incongruity caused by the area boundary is minimized, and the image with a wide field of view can be continuously viewed in high definition.

[0050] Next, the operation of the head mounted display 100 that can be realized by the above configuration will be described. FIG. 8 is a flowchart showing a processing procedure in which the head mounted display 100 of this embodiment displays an image. This flowchart is started when the user wears the head mounted display 100 and selects the content to be displayed via an input device (not shown). In response to this, the image data acquisition unit 50 starts acquiring image data of the content. Note that the head mounted display 100 may internally perform information processing such as a game, or establish communication with an external device to request image data, but the figure particularly shows the image display processing.

[0051] First, the head mounted display 100 displays an initial image of the content (S10). The initial image may also be an image obtained by combining the central image displayed by the central image output unit 60 and the peripheral image displayed by the peripheral image output unit 62, and in this case, the central image may have a size that is preset for the initial image. Next, the gaze point acquisition unit 52 acquires the user's gaze point for the initial image (S12). Then, the central image size control unit 54 first checks whether the gaze point is within the range that can be drawn by the central image output unit 60, i.e., the laser scanning for displaying the central image (S14).

[0052] If the gaze point is within the drawable range of the center image output unit 60 (Y in S14), the center image size control unit 54 determines the size of the center image according to the position of the gaze point as shown in Fig. 6 (S16). If the gaze point is outside the drawable range of the center image output unit 60 (N in S14), the center image size control unit 54 determines the center image to be the maximum size, i.e., the maximum drawable range (S18). This minimizes the resolution of the center image, reducing the possibility that the difference in resolution with the peripheral images will be perceived unnaturally. Note that the minimum value of the resolution of the center image and the resolution of the peripheral images may be configured to be uniform, so that the resolution is uniform when the gaze point is in the peripheral images.

[0053] The center image size control unit 54 notifies the center image output unit 60 of the determined center image size as needed, thereby causing the center image output unit 60 to set the amplitude of the angle of the MEMS mirror corresponding to the size (S20). Meanwhile, the center image generation unit 56 and the peripheral image generation unit 58 acquire necessary data from the image data acquisition unit 50 based on the center image size notified by the center image size control unit 54, and generate the center image and the peripheral image, respectively (S22). Then, the center image output unit 60 and the peripheral image output unit 62 display the center image and the peripheral image, respectively, and the display image synthesized by the image synthesis unit 64 reaches the user's eyes (S24).

[0054] When it is not necessary to end the display, such as when a user operation is performed to end the display of the content, the processes from S12 to S24 are repeated (N in S26). This allows the image display to continue while changing the range and resolution of the central image by moving the gaze point. When it is necessary to end the display, all processes are terminated (Y in S26).

[0055] Next, a specific structure for synthesizing and visually recognizing a central image and a peripheral image will be described. Fig. 9 shows a structural example of an image synthesizing unit 64 for synthesizing and visually recognizing a central image and a peripheral image in the head mounted display 100 of this embodiment. This figure shows a schematic vertical cross-sectional view of the positional relationship between the user's eyes 224 and the display unit including the first display unit 132 and the second display unit 134 when the head mounted display 100 is worn. The same applies to Figs. 10 to 13 described later.

[0056] 9, the first display unit 132 includes a laser light source 220, a mirror 222, and a central image screen 232 made of a member that diffuses and transmits reflected laser light. On the other hand, the second display unit 134 includes a peripheral image display panel 234 made of a two-dimensional array of light-emitting elements. The angle between the central image screen 232 and the peripheral image display panel 234 is set to 90°, and a half mirror 236 is placed in the middle at an angle of 45° to each of them to combine the central image and the peripheral image. The half mirror 236 may be a general type that transmits a predetermined percentage of the incident light and reflects the remainder.

[0057] In the illustrated example, the laser light reflected by the mirror 222 is diffused and transmitted by the central image screen 232, passes through the half mirror 236, and reaches the eye 224 via the eyepiece 238. That is, the first display unit 132 controls the operation of the mirror 222 so that the original image is displayed in a state in which it is diffused and transmitted by the central image screen 232. On the other hand, the light from the peripheral image display panel 234 is reflected by the half mirror 236, and reaches the eye 224 via the eyepiece 238. As a result, the central image and the peripheral image are viewed in a synthesized state.

[0058] However, the positional relationship between the first display unit 132 and the second display unit 134 may be reversed, and the laser light from the mirror 222 may be reflected by the half mirror 236, and the light from the peripheral image display panel 234 may be transmitted to reach the eye 224. Also, instead of the peripheral image display panel 234, the peripheral image may be displayed by a laser scanning method. In either case, in this configuration, the eye photographing camera 240 included in the gaze point detector 130 may be positioned beside the eyepiece lens 238 as shown in the figure.

[0059] Fig. 10 shows another example of the structure of the image synthesis unit 64 that synthesizes a central image and a peripheral image and makes them visible in the head mounted display 100 of this embodiment. This configuration differs from Fig. 9 in that no central image screen is provided as the first display unit 132, and an image made of laser light is directly projected onto the user's retina. According to the principle of Maxwell's vision, known techniques can be applied to project an image onto the retina, as described above. In other words, the first display unit 132 controls the operation of the mirror 222, etc., so that the original image is visible when the laser light is converged by the pupil and focused on the retina.

[0060] However, in this embodiment, the central image is projected through the half mirror 236, and is visually recognized by combining it with the peripheral image displayed on the peripheral image display panel 234 and reflected by the half mirror 236. In this case, since no central image screen is provided, the degree of freedom in arranging the eyeball photographing camera 240 included in the gaze point detector 130 is increased. For example, as shown in the figure, it is also possible to photograph the eyeball from near the front through the half mirror 236.

[0061] Fig. 11 shows another example of the structure of the image synthesis unit 64 that synthesizes and visually recognizes a central image and a peripheral image in the head mounted display 100 of this embodiment. In this configuration, the central image screen 242 that diffuses and transmits the laser light of the first display unit 132 is provided integrally with the peripheral image display 244, and is different from Fig. 9 in that no half mirror is provided. A light-transmitting display that can transmit light from the background in an area (non-display area) of a display panel that does not display an image is known (see, for example, International Publication No. 2014 / 010585). In this embodiment, this is applied, and the peripheral image display 244 is made of a light-transmitting display whose base material is made of a translucent material.

[0062] As a result, the area of ​​the peripheral image display 244 that does not display the peripheral image can be used as the central image screen 242 that diffuses and transmits the laser light reflected by the mirror 222. Naturally, the range of the peripheral image changes according to the change in size of the central image, so the range of the central image screen 242 changes appropriately according to the size of the central image. In this case, a part of the central image output unit 60 and the peripheral image output unit 62 also serve as the image synthesis unit 64. With this configuration, the optical system can be simplified compared to projecting two types of images from different directions. In this configuration, as in FIG. 9, it is considered that the eyeball photographing camera 240 included in the gaze point detector 130 is disposed beside the eyepiece 238.

[0063] FIG. 12 shows another example of the structure of the image synthesis unit 64 for synthesizing and visually recognizing a central image and a peripheral image in the head mounted display 100 of this embodiment. This configuration is similar to FIG. 11 in that a central image screen 242 and a peripheral image display 244 are integrally provided by applying a light transmission type display. On the other hand, it differs from FIG. 11 in that a half mirror 246 is provided between the central image screen 242 and the peripheral image display 244. That is, the light from the central image screen 242 and the peripheral image display 244 is visually recognized through the half mirror 246. In this way, the amount of light of the image is reduced, but if the half mirror 246 is arranged to form an angle of 45° with the surface of the eye 224, the image of the eye 224 can be captured by the eyeball capturing camera 240 by its reflection, so that the gaze point can be detected with the same quality as when the image is captured from the front.

[0064] Fig. 13 shows another example of the structure of the image synthesis unit 64 that synthesizes and visually recognizes a central image and a peripheral image in the head mounted display 100 of this embodiment. In this configuration, similar to Fig. 9, a central image screen 250 and a peripheral image display panel 252 are provided separately, but they are arranged on approximately the same plane, and image synthesis is achieved by guiding each image in an appropriate direction using a central image optical system 254 and a peripheral image optical system 256.

[0065] A method of introducing a free-form optical system into a head-mounted display and guiding images displayed on multiple displays to an appropriate position by reflection or refraction to be viewed as a single image is disclosed in, for example, International Publication No. 2019 / 147946. This configuration allows the head-mounted display 100 to be made smaller than when two types of images are projected from different directions. In addition, since the optical path can be designed relatively freely, the degree of freedom in the placement of the eye-photographing camera 240 can be increased.

[0066] For example, as shown in the figure, by arranging the central image screen 250, the peripheral image display panel 252, and their respective optical systems 254, 256 away from the front of the eye 224, the eye photographing optical system 258 and the eye photographing camera 240 can be arranged in front of the eye 224. This makes it easier to detect the gaze point. Note that the positions and orientations of the central image screen 250 and the peripheral image display panel 252 may vary depending on the design of the central image optical system 254 and the peripheral image optical system 256.

[0067] According to the present embodiment described above, the display image is divided into a central image and a peripheral image, and the central image is displayed by separate mechanisms so that the central image is displayed at a higher resolution, and then the images are synthesized and viewed. Here, the display mechanism that displays at least the central image is a laser scanning type that forms an image by two-dimensionally scanning a laser by reflection on a mirror. With the laser scanning type, the display range and the resolution as the density of pixels that display it can be changed by controlling the amplitude of the mirror angle. Therefore, it is easier to control the distribution of resolution in the display image compared to a display panel with a fixed density of light-emitting elements.

[0068] For example, as the user's gaze point moves away from the center of the image plane, the amplitude of the mirror angle is changed so that the central image is expanded to include the gaze point. This makes the boundary between the central image and the peripheral image farther away from the gaze point regardless of the position of the gaze point, making it difficult to recognize the resolution difference at the boundary. In addition, the resolution decreases as the central image expands, so the actual resolution difference is also small. This allows images with a distribution of resolution to be recognized without discomfort without high-load processing such as image data processing. As a result, resources can be concentrated in areas with high discrimination ability due to visual characteristics, and even wide-field images can be displayed with low latency and high resolution.

[0069] The present invention has been described above based on the embodiments. The above embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention.

[0070] For example, the implementation of the present invention is not limited to head mounted displays, but can also be applied to projectors and general television receivers. In these cases, the internal structure may be the same as any of Figs. 9, 11, 12, and 13. In the case of a projector, a projection lens that projects an image onto an external screen or the like is provided instead of the eyepiece lens 238. In the case of a television receiver, a screen that transmits the image serves as the display surface instead of the eyepiece lens 238. Alternatively, according to the configuration of Fig. 11, a light-transmitting display that realizes the peripheral image display 244 can be used as is.

[0071] In these cases, the gaze point detector is naturally provided with a camera for photographing the eyeball in the direction of the user's eyes looking at the screen or display surface. Note that in this embodiment, the size of the central image is controlled according to the movement of the gaze point, but instead of the gaze point, the size may be controlled according to the display position of a main object or the position of an important area in the display so that it is included in the central image. Since main objects and important areas are likely to be gazed upon, it is considered that doing so will also produce the same effect as this embodiment. [Explanation of symbols]

[0072] 50 image data acquisition unit, 52 gaze point acquisition unit, 54 central image size control unit, 56 central image generation unit, 58 peripheral image generation unit, 60 central image output unit, 62 peripheral image output unit, 64 image synthesis unit, 100 head mounted display, 110 stereo camera, 120 CPU, 122 GPU, 124 main memory, 130 gaze point detector, 132 first display unit, 134 second display unit.

Claims

1. a central image generating unit that generates a central image representing a central portion of the display image in an image plane; a peripheral image generating unit that generates a peripheral image representing an area outside the central image in the display image; a center image output unit that displays the center image by a laser scanning method in which a laser beam that displays pixels is two-dimensionally scanned by mirror reflection to project an image; a surrounding image output unit that displays the surrounding image; an image synthesis unit for synthesizing the central image and the peripheral image and visually recognizing the synthesized image; a gaze point acquisition unit that acquires a gaze point of a user with respect to a displayed image; a central image size control unit that controls a size of the central image so that the central image expands as the point of interest moves away from the center of the image plane, so that the point of interest is included in the central image; Equipped with The image display device according to claim 1, wherein the central image output unit changes the amplitude of the angle of the mirror in accordance with the size of the central image determined by the central image size control unit.

2. The image display device according to claim 1, characterized in that when the gaze point exceeds the upper limit of the range that the center image output unit can display, the center image size control unit sets the size of the center image to the upper limit.

3. 3. The image display device according to claim 1, wherein the image synthesis unit includes a half mirror that synthesizes the image represented by the central image output unit and the image represented by the peripheral image output unit by transmitting one of the images and reflecting the other.

4. the central image output unit includes a screen that diffuses and transmits the laser light reflected by the mirror, 4. The image display device according to claim 3, wherein the image synthesizing section synthesizes the image diffused and transmitted through the screen with the image displayed by the peripheral image output section by means of the half mirror.

5. the central image output unit projects laser light representing the central image viewed by Maxwellian vision in a direction toward an eye of a user; 4. The image display device according to claim 3, wherein the image synthesis unit transmits the laser light and reflects the image displayed by the peripheral image output unit using the half mirror, thereby synthesizing the two and allowing them to be viewed.

6. the peripheral image output unit displays the peripheral image using a light-transmitting display that transmits background light in a non-display area; 3. The image display device according to claim 1, wherein the central image output section projects the laser light reflected by the mirror onto a corresponding area of ​​the light-transmitting display and diffuses and transmits the laser light.

7. a half mirror that transmits light from the light-transmitting display and reflects an image of a user's eyeball; a camera for capturing an image of the eyeball reflected by the half mirror and acquiring a gaze point; 7. The image display device according to claim 6, further comprising:

8. the central image output unit includes a screen that diffuses and transmits the laser light reflected by the mirror, 3. The image display device according to claim 1, wherein the image synthesis unit synthesizes the image diffused through the screen and the image displayed by the peripheral image output unit using a free-form optical system that guides the two images to corresponding positions on an image plane.

9. 9. The image display device according to claim 8, further comprising a camera in a direction directly in front of the user's eyeball for photographing the eyeball and obtaining a gaze point, the screen and the display panel constituting the peripheral image output unit being arranged around the camera.

10. generating a central image representing a central portion of the displayed image in an image plane; generating a peripheral image of the displayed image that represents an area outside the central image; a step of displaying the central image by a central image output unit of a laser scanning type that projects an image by two-dimensionally scanning a laser light representing a pixel by reflection of a mirror; displaying the peripheral image by a peripheral image output unit; a step of visually recognizing the central image and the peripheral image by synthesizing the central image and the peripheral image; acquiring a user's gaze point with respect to a displayed image; controlling a size of the central image such that the central image expands as the point of regard moves away from the center of the image plane so that the point of regard is included in the central image; Including, 2. An image display method, comprising: a step of changing an amplitude of an angle of the mirror in accordance with a size of the central image determined in the step of controlling the central image;

Images