Image display device and image display method

Abstract

To cause a high-definition and wide-field image to be visually recognized easily without a sense of discomfort.SOLUTION: In an image plane 200, a region 212a including a central portion is represented at a higher resolution than that of a region 214a outside the region 212a. The region 212a is represented with a laser scanning system that projects an image by reflecting a beam from the laser beam source 220 using a mirror 222 and carrying out two-dimensional scanning. The region 212a uses when a gazing point 202a is at the center of the image as a reference and controls the amplitude of an angle of the mirror 222 so that a region 212b is extended according to movement from the center.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image display device and an image display method that allow a user to view an image. [Background technology]

[0002] Image display systems that allow users to view a target space from any viewpoint are becoming widespread. For example, a system has been developed that displays panoramic images on a head-mounted display, and when the user wearing the head-mounted display rotates their head, the panoramic image corresponding to the direction of their gaze is displayed. By using a head-mounted display, it is also possible to enhance immersion in the image and improve the usability of applications such as games. Furthermore, walk-through systems have been developed that allow users wearing a head-mounted display to virtually walk around the displayed space by physically moving.

[0003] To improve the quality of the video experience, it is necessary to represent images with a wide field of view in high definition. However, as resolution and field of view are expanded, the data size of the images that need to be processed increases, resulting in longer processing and transmission times and a higher likelihood of display delays. Therefore, a technology has been proposed that utilizes the visual characteristic of humans, where visual acuity decreases from the center to the edges of the field of view, to create a difference in image resolution between the central region and the outer regions, thereby reducing processing waste while maintaining visually apparent image quality (see, for example, Patent Document 1). [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] U.S. Patent No. 10140695 [Overview of the project] [Problems that the invention aims to solve]

[0005] In the field of image display technology, not just for head-mounted displays, the ability to display detailed images with a wide field of view and low latency is a common challenge. For example, in the technology described in Patent Document 1, separate displays are provided for the central area and other areas, and by giving a clear difference in display resolution, it becomes easier to appropriately allocate processing resources. On the other hand, when images with different resolutions are combined, the boundaries may appear unnatural and cause discomfort to the user. To resolve this, one could consider smoothly connecting the resolutions in the data, but this would require separate image processing. Especially when the high-resolution area is linked to the user's point of focus, it is possible that a significant amount of processing time will be required for cropping the fluctuating area and adjusting the resolution.

[0006] This invention was made in view of these problems, and its purpose is to provide a technology that allows high-definition, wide-field-of-view images to be easily viewed without any discomfort. [Means for solving the problem]

[0007] One aspect of the present invention relates to an image display device. This image display device is characterized by comprising: a central image generation unit that generates a central image representing the central portion of an image on an image plane; a peripheral image generation unit that generates peripheral images representing the region outside the central image of the displayed image; a central image output unit that displays the central image using a laser scanning method that projects an image by scanning laser light representing pixels in two dimensions by reflection from a mirror; a peripheral image output unit that displays peripheral images; and an image synthesis unit that combines the central image and peripheral images for viewing.

[0008] Another aspect of the present invention relates to an image display method. This image display method is characterized by including the steps of: generating a central image representing the central portion of the display image in the image plane; generating peripheral images representing the region outside the central image of the display image; representing the central image using a central image output unit of a laser scanning system that projects an image by scanning laser light representing pixels in two dimensions by reflection from a mirror; displaying the peripheral images using a peripheral image output unit; and combining the central image and the peripheral images for viewing.

[0009] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, computer programs, recording media containing computer programs, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0010] According to the present invention, high-definition, wide-field-of-view images can be easily viewed without any discomfort. [Brief explanation of the drawing]

[0011] [Figure 1] This figure illustrates the relationship between human visual characteristics and the display format of this embodiment. [Figure 2] This figure illustrates the relationship between the laser scanning display mechanism and the area on the image used in this embodiment. [Figure 3] This figure shows an example of the appearance of the head-mounted display according to this embodiment. [Figure 4] This figure shows the internal circuit configuration of the head-mounted display of this embodiment. [Figure 5] This diagram shows the configuration of the functional blocks of the head-mounted display in this embodiment. [Figure 6] This figure schematically illustrates the changes in the central and peripheral images in response to the movement of the point of fixation in this embodiment. [Figure 7]This is a diagram for explaining the change in resolution with respect to the size of the central image in the present embodiment. [Figure 8] This is a flowchart showing the processing procedure for the head-mounted display of the present embodiment to display an image. [Figure 9] This is a diagram showing an example of the structure of an image synthesis unit that synthesizes a central image and a peripheral image for visual recognition in the head-mounted display of the present embodiment. [Figure 10] This is a diagram showing another example of the structure of an image synthesis unit that synthesizes a central image and a peripheral image for visual recognition in the head-mounted display of the present embodiment. [Figure 11] This is a diagram showing another example of the structure of an image synthesis unit that synthesizes a central image and a peripheral image for visual recognition in the head-mounted display of the present embodiment. [Figure 12] This is a diagram showing another example of the structure of an image synthesis unit that synthesizes a central image and a peripheral image for visual recognition in the head-mounted display of the present embodiment. [Figure 13] This is a diagram showing another example of the structure of an image synthesis unit that synthesizes a central image and a peripheral image for visual recognition in the head-mounted display of the present embodiment. [[ID=2*]]

Mode for Carrying Out the Invention

[0012] FIG. 1 is a diagram for explaining the relationship between human visual characteristics and the display format of the present embodiment. First, in the image plane 200 shown in (a), it is assumed that the user's fixation point 202 indicated by a circle exists near the center. As general human visual characteristics, the region 204 corresponding to within 5° centered on the line of sight from the pupil to the fixation point is called the discrimination visual field, and visual functions such as visual acuity are excellent. Also, the region 206 corresponding to within approximately 30° in the horizontal direction and within approximately 20° in the vertical direction is called the effective visual field, and information can be instantaneously received only by eye movement.

[0013] It should be noted that there seems to be a small error in your original text where the "

発明を実施するための形態

[0014] In this embodiment, a laser scanning method is employed as the display mechanism for the region 212 including the point of focus 202. The laser scanning method is a technique that forms an image on a projection surface by scanning a laser beam corresponding to a pixel in two dimensions using a deflection mirror. For example, the technique of focusing a laser beam onto the user's pupil and projecting an image onto the retina is being increasingly applied to wearable displays (see, for example, International Publication No. 2009 / 066465). In addition, small projectors that project images onto external screens have also been put into practical use (see, for example, Japanese Patent Publication No. 2017-83657).

[0015] Figure 2 is a diagram illustrating the relationship between the laser scanning display mechanism and the area on the image used in this embodiment. The upper parts of (a) and (b) are schematic diagrams of the laser scanning display mechanism viewed from the side. The laser light source 220 outputs laser light containing red, blue, and green components. This laser light is reflected by the mirror 222 and projected onto the projection surface (image plane 200).

[0016] By oscillating the mirror 222 around two axes, the laser beam is scanned in two dimensions on the projection surface, forming an image with the laser beam output at each time point as pixels. In this example, the mirror 222 is oscillated symmetrically with respect to its orientation when the reflected laser light reaches the center of the image plane 200. As a result, the center of the region 212a represented by the laser scanning method coincides with the center of the image plane. However, this embodiment is not intended to be limited to this.

[0017] The diagram also assumes a head-mounted display, showing the user's eye 224 on the opposite side of the projection surface, such as a transparent screen. However, as mentioned above, direct projection onto the retina is also possible, and a screen is not essential. For the mirror 222, for example, a MEMS (Micro Electro Mechanical Systems) mirror can be introduced. A MEMS mirror is a small, low-power device that can precisely control angular changes around two axes by electromagnetic drive. However, the driving method of the mirror is not particularly limited.

[0018] In this embodiment, by changing the amplitude of the angle of the mirror 222 in accordance with the movement of the point of focus, the size of the area represented by the laser scanning method is expanded as the point of focus moves further away from the center of the image plane 200. First, in the case shown in (a), as shown in Figure 1, the point of focus 202a is at the center of the image plane 200. The amplitude of the angle of the mirror 222 at this time, and consequently the amplitude of the scanning angle of the laser beam, is set to a reference value θ std Let's assume the reference value is θ. std This can be any predetermined minimum value.

[0019] On the other hand, in the case shown in (b), the point of fixation 202b moves to the lower left of the image plane 200. At this time, the amplitude θ of the angle of the mirror 222 is θ std To make it larger. Specifically, the amplitude θ of the angle of the mirror 222 is increased so that the edge of the region 212b represented by the laser scanning method maintains a constant distance from the point of focus 202b, thereby expanding the region 212b. For example, when expanding the region 212a in the reference state shown in (a) by a factor of n in the vertical direction, the amplitude θ of the angle can be determined as follows. tanθ = n × tan(θ) std ) Naturally, the amplitude of the angle of mirror 222 can also be determined in the lateral direction.

[0020] In a laser scanning display method, if the frame rate is fixed, the scanning angle and resolution are inversely proportional. That is, when region 212a is expanded by n times, the number of pixels per unit angle in the expansion direction becomes 1 / n. On the other hand, the resolution of the outer regions 214a and 214b, which are represented by a different display mechanism, can remain constant. With this control, firstly, since the point of focus is always inside regions 212a and 212b represented by the laser scanning method, the boundary area where there is a difference in resolution is less likely to be noticed. Also, as the point of focus approaches the edge of the image plane 200, the difference in resolution between regions becomes smaller, making the boundary less noticeable. Therefore, even if the point of focus extends beyond the region that can be represented by laser scanning, the boundary line becomes less likely to be recognized.

[0021] In the reference state shown in (a), the reference value θ is set according to general visual characteristics, such as ensuring that the edge of the region 212a represented by the laser scanning method is at least outside the effective field of view. std By determining this, even if there is a certain difference in resolution, the boundary between them becomes less noticeable. This allows for a higher resolution in region 212a, enabling the display of high-quality images without any noticeable discrepancies. Since these effects are naturally obtained by controlling the amplitude of the angle of mirror 222 in a laser scanning display system, the increase in processing load can be suppressed.

[0022] The display device to which this embodiment can be applied is not particularly limited, but a head-mounted display will be used as an example below. Figure 3 shows an example of the appearance of the head-mounted display of this embodiment. In this example, the head-mounted display 100 consists of an output mechanism 102 and a mounting mechanism 104. The mounting mechanism 104 includes a mounting band 106 that wraps around the head when worn by the user to secure the device. The output mechanism 102 includes a housing 108 shaped to cover the left and right eyes when the user is wearing the head-mounted display 100, and includes a laser scanning display mechanism as described above and a mechanism for displaying images in other areas.

[0023] The housing 108 further includes a mechanism for combining images from two displayed regions, and an eyepiece for expanding the field of view. Stereoscopic vision may be achieved by displaying stereo images with parallax to each of the left and right eyes. The housing 108 further includes a gaze point detector for detecting the user's gaze point on the displayed image.

[0024] The head-mounted display 100 may also be equipped with speakers or earphones positioned to correspond to the user's ears when worn. In this example, the head-mounted display 100 is equipped with a stereo camera 110 on the front of the housing 108 to capture video of the surrounding real space with a field of view corresponding to the user's gaze. Furthermore, the head-mounted display 100 may be equipped with any of the following sensors inside or outside the housing 108 to determine the movement, posture, position, etc. of the head-mounted display 100, such as an accelerometer, gyroscope, geomagnetic sensor, or GPS.

[0025] Figure 4 shows the internal circuit configuration of the head-mounted display 100. The head-mounted display 100 includes a CPU (Central Processing Unit) 120, a GPU (Graphics Processing Unit) 122, and main memory 124. These components are interconnected via a bus 140. An input / output interface 138 is further connected to the bus 140. The input / output interface 138 is connected to a communication unit 126, a motion sensor 128, a stereo camera 110, a gaze point detector 130, a first display unit 132, a second display unit 134, and an audio output unit 136.

[0026] The CPU 120 controls the entire head-mounted display 100 by executing the operating system stored in the main memory 124. The CPU 120 also executes various programs downloaded via the communication unit 126 and plays electronic content. The GPU 122 has the functions of both a geometry engine and a rendering processor, and draws display images according to drawing commands from the CPU 120 and outputs them to the first display unit 132 and the second display unit 134.

[0027] The main memory 124 is composed of RAM (Random Access Memory) and stores programs and data necessary for processing by the CPU 120 and other components. The communication unit 126 is a network interface such as wired or wireless LAN or Bluetooth (registered trademark) and enables communication with external devices. The motion sensor 128 is composed of at least one of the following sensors: an accelerometer, a gyroscope, a geomagnetic sensor, or a GPS, and measures the position, posture, and movement of the head of the head-mounted display 100, and by extension, the head of the user wearing it.

[0028] As shown in Figure 3, the stereo camera 110 is a pair of video cameras that capture the surrounding real space from left and right viewpoints, with a field of view corresponding to the user's viewpoint. By immediately displaying the video images captured by the stereo camera 110 on the first display unit 132 and the second display unit 134, so-called video see-through can be achieved, allowing the user to see the real space in the direction they are facing. Furthermore, by drawing virtual objects on the images of real objects captured in the image, augmented reality can be realized. In addition, by analyzing the images captured by the stereo camera 110 using known technologies such as Visual SLAM (Simultaneous Localization and Mapping), the position and posture of the head-mounted display 100, and by extension the user's head, can be tracked.

[0029] By integrating the analysis results of the captured images with the measurement results of the motion sensor 128, the user's head movements can be acquired with higher accuracy. This allows for the generation of display images in the field of view that correspond to the head movements with high accuracy, thereby enhancing immersion in the visual world. Furthermore, the user's head movements can be accepted as user input to the content, and processing can be branched accordingly.

[0030] The gaze point detector 130 detects the position coordinates of the user's gaze point while viewing the images displayed by the first display unit 132 and the second display unit 134 at a predetermined rate. The gaze point detector 130 is composed of, for example, a mechanism that irradiates the eyeball with infrared light and a camera that captures the reflected light, and tracks the point on the image that the user is fixated on by determining the direction of the pupil from the captured image. Various other technologies have been put into practical use as means for detecting gaze points, and any of them may be adopted in this embodiment.

[0031] The first display unit 132 is composed of a laser scanning display mechanism as described in Figure 2, and projects and displays an image of the region including the center of the image plane. Hereafter, the partial image represented by the first display unit 132 will be called the "center image". The second display unit 134 displays an image of the region outside the center image. Hereafter, the partial image represented by the second display unit 134 will be called the "peripheral image". The display method of the second display unit 134 is not particularly limited and may be a display panel consisting of a two-dimensional array of light-emitting elements such as a liquid crystal panel or an organic EL panel, or it may be a laser scanning display mechanism similar to that of the first display unit 132. In any case, the second display unit 134 displays the peripheral image at a lower resolution than the center image displayed by the first display unit 132.

[0032] The first display unit 132 and the second display unit 134 each display the central image and peripheral image generated by the GPU 122 at a predetermined rate. The images displayed by the first display unit 132 and the second display unit 134 are combined by a synthesis mechanism described later and viewed by the user as a single display image. As mentioned above, stereoscopic vision may be achieved by displaying stereo images to the left and right eyes. In this case, the stereo image is a pair of images created by combining the central image and the peripheral image. The audio output unit 136 consists of speakers or earphones positioned to correspond to the user's ears when the head-mounted display 100 is worn, and provides the user with audio.

[0033] Some of the functions of the illustrated head-mounted display 100 may be provided by an external device that has established communication with the head-mounted display 100. For example, at least some of the processes such as determining an appropriate field of view and generating an overall image, controlling the boundary between the central image and peripheral images according to the point of gaze, and generating data for the central image and peripheral images may be performed by an external image generation device or an image provision server connected via a network.

[0034] Figure 5 shows the configuration of the functional blocks of the head-mounted display 100. Each functional block shown in the figure can be realized hardware-wise by the various circuits shown in Figure 4, and software-wise by programs that perform various functions such as information processing, image processing, display, and communication functions, which are loaded from the recording medium into the main memory 124. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination thereof, and are not limited to any one of these.

[0035] The head-mounted display 100 includes an image data acquisition unit 50 that acquires data of the image to be displayed, a gaze point acquisition unit 52 that acquires the user's gaze point on the displayed image, a central image size control unit 54 that controls the size of the central image, a central image generation unit 56 that generates the central image, a peripheral image generation unit 58 that generates peripheral images, a central image output unit 60 that outputs the central image as the display target, a peripheral image output unit 62 that outputs peripheral images as the display target, and an image synthesis unit 64 that delivers the central image and peripheral images to the user's eyes in a composite state.

[0036] The image data acquisition unit 50 acquires data necessary for generating moving or still images to be displayed. The content represented by the images is not particularly limited and may include game images, movies, live videos, recorded videos, animations, photographs, environmental videos, websites, documents, digital signage, etc. It may also include images captured by the stereo camera 110, or images that have been processed or into which virtual objects have been drawn. Depending on the content of such images, the data acquisition source by the image data acquisition unit 50 may vary.

[0037] For example, the image data acquisition unit 50 may acquire video data streamed by an external image generation device or server, or it may draw or play images using data stored in an internal storage device. The gaze point acquisition unit 52 includes the gaze point detector 130 shown in Figure 4, and acquires the position coordinates of the user's gaze point relative to the displayed image at a predetermined rate. In this embodiment, the displayed image is any image that is within the user's field of view, and projection of laser light is also considered "display".

[0038] The central image size control unit 54 controls the size of the central image according to the position of the user's gaze point. Specifically, as explained in Figure 2, the reference state is when the gaze point is at the center of the image plane, and the range is expanded so that the central image encompasses the gaze point as it moves. As a result, the central image widens as the gaze point moves away from the center and narrows as the gaze point approaches the center. The central image size control unit 54 determines the size of the central image at a predetermined rate or as needed, and supplies this information to the central image generation unit 56, the peripheral image generation unit 58, and the central image output unit 60.

[0039] The central image generation unit 56 includes the GPU 122 shown in Figure 4 and generates the central image by acquiring the necessary data from the image data acquisition unit 50. The peripheral image generation unit 58 also includes the GPU 122 shown in Figure 4 and generates the peripheral image by acquiring the necessary data from the image data acquisition unit 50. Here, the peripheral image is an image in which the area of ​​the central image has been blacked out (disabled) from the entire display image. The boundary between the central image and the peripheral image is updated as appropriate according to the information from the central image size control unit 54.

[0040] The central image output unit 60 includes the first display unit 132 shown in Figure 4, and displays the central image generated by the central image generation unit 56 at a predetermined rate by laser scanning. Specifically, the central image output unit 60 determines the amplitude of the mirror angle around each of the two axes according to the size of the central image notified by the central image size control unit 54. Then, by scanning the laser light representing the color of each pixel of the central image in two dimensions at the scanning angle corresponding to the amplitude, it projects the image onto the user's retina or a screen in front of their eyes.

[0041] The peripheral image output unit 62 includes the second display unit 134 shown in Figure 4, and displays the peripheral image using a display panel made of light-emitting elements or by laser scanning. When using a display panel, the central image area is not illuminated. When using a laser scanning method, laser light is not output to the central image area. In either case, the pixel density for displaying the peripheral image is constant regardless of the size of the central image.

[0042] The image synthesizing unit 64 is an optical system that synthesizes the displayed central image and the peripheral image so that they reach the eyes as one image. That is, the image synthesizing unit 64 is a hardware structure that synthesizes the central image and the peripheral image without misalignment and can take various forms according to the positional relationship between the first display unit 132 and the second display unit 134, the arrangement required for the fixation point detector 130, and the like. Specific examples will be described later.

[0043] FIG. 6 schematically shows changes in the central image and the peripheral image with respect to the movement of the fixation point. The upper part of the figure is the entire displayed image, the middle part is the central image, and the lower part is the peripheral image. Note that the central image and the peripheral image indicate the range of the region with respect to the entire image by making the outside dark, and do not indicate the size of the image as data. As shown in (a), when the fixation point 70a is at the center of the entire image, the central image is a region of a predetermined size (X std , Y std [[ID=, the peripheral image is the region obtained by excluding the region of the central image from the entire image.

[0044] Here, the size (X std , Y std ) of the central image is preferably determined based on the relationship between the angle and the visual acuity when the line of sight is the central axis, as described in FIG. 1. On the other hand, as shown in (b), when the fixation point 70b is displaced from the center on the image plane, the central image is expanded so as to enclose it. For example, when the displacement vector of the fixation point from the image center is (Δx, Δy) as shown in the figure, the size (X, Y) of the central image is determined as follows. X = 2 * (|Δx| + m x ) Y = 2 * (|Δy| + m [[ID=, the peripheral image is the region obtained by excluding the region of the central image from the entire image.

[0044] Here, the size (X std , Y std ) of the central image is preferably determined based on the relationship between the angle and the visual acuity when the line of sight is the central axis, as described in FIG. 1. On the other hand, as shown in (b), when the fixation point 70b is displaced from the center on the image plane, the central image is expanded so as to enclose it. For example, when the displacement vector of the fixation point from the image center is (Δx, Δy) as shown in the figure, the size (X, Y) of the central image is determined as follows. X = 2 * (|Δx| + m x ) Y = 2 * (|Δy| + m y )

[0045] Here, m x , m y are the margins given to the distances between the two sides of the central image closest to the fixation point and the fixation point. In other words, the edges of the central image are always controlled to be located at least (m x , m y ) outside the fixation point. (m x , m <000) It is also desirable to prepare determination rules in advance based on the relationship between the angle with respect to the line of sight as the central axis and visual acuity. For example (m x ,m y )=(X std / 2,Y std You can also use ( / 2). Or (m x , m y ) may also be a function of the displacement vector (Δx, Δy). In this case as well, the peripheral image is the region obtained by subtracting the central image region from the entire image.

[0046] The central image size control unit 54 may update the size of the central image as needed in response to changes in the displacement vector of the point of fixation, or it may update the size of the central image in stages when the displacement vector changes by more than a threshold. When viewing the image in stereoscopic form, the central image generation unit 56 and the peripheral image generation unit 58 generate the central image and peripheral image shown for both the left eye image and the right eye image. Furthermore, the central image generation unit 56 and the peripheral image generation unit 58 generate the central image and peripheral image with distortion in the opposite direction to the distortion and chromatic aberration caused by the eyepiece of the head-mounted display 100, so that when viewed through the eyepiece, an image without distortion or color shift can be seen. Depending on the configuration of the first display unit 132, the shape of the central image is not limited to a rectangle, and naturally, the shape of the blacked-out area of ​​the peripheral image also depends on the shape of the central image.

[0047] Figure 7 illustrates the change in resolution with respect to the size of the central image. The upper part of the figure shows the entire displayed image, where (a) is the reference state with the fixation point 72a at the center, and (b) is the state where the fixation point 72b is displaced from the center on the image plane. The lower part shows the resolution distribution in the horizontal directions AA' and BB' passing through fixation points 72a and 72b on the image plane. Compared to the minimum size central image 74a in the reference state shown in (a), as the central image 74b expands as shown in (b), the resolution in that area decreases.

[0048] Here, "resolution" refers not to the data-level detail of the image, but to the physical number of images depicted per unit area (or unit angle), i.e., pixel density. As mentioned above, laser scanning display mechanisms have the characteristic that resolution increases as the projection area decreases. For example, in a device that can display an image of 600 pixels within a horizontal field of view of 30°, the angular resolution is 20 ppd (pixels per degree). If the amplitude of the mirror angle is halved in that state, the same 600 pixels will be displayed within a field of view of 15°, so the angular resolution becomes 40 ppd.

[0049] Due to this characteristic, the resolution is highest in the smallest central image 74a shown in (a), and decreases as the size of the central image 74b increases, as shown in (b). Since the user views the image with the focus on points 72a and 72b, the resolution decreases as the gaze shifts towards the edges of the image, creating a smooth transition with the surrounding image. Furthermore, the surrounding image continues to be displayed at a constant resolution suitable for viewing, thus maintaining the field of view. As a result, regardless of how the focus point shifts, the discomfort caused by area boundaries is minimized, and a wide field of view image can be viewed with high resolution.

[0050] Next, the operation of the head-mounted display 100 that can be realized with the above configuration will be described. Figure 8 is a flowchart showing the processing procedure for displaying an image in the head-mounted display 100 of this embodiment. This flowchart is started when the user puts on the head-mounted display 100 and selects the content to be displayed via an input device (not shown). In response, the image data acquisition unit 50 starts acquiring image data of the content. The head-mounted display 100 may perform information processing such as games internally, or establish communication with external devices to request image data, but the figure specifically shows the image display process.

[0051] First, the head-mounted display 100 displays an initial image of the content (S10). This initial image may be a composite image of a central image displayed by the central image output unit 60 and a peripheral image displayed by the peripheral image output unit 62, and in this case, the central image may have a size predetermined relative to the initial image. Next, the gaze point acquisition unit 52 acquires the user's gaze point on the initial image (S12). Then, the central image size control unit 54 first checks whether the gaze point is within the range that can be drawn by the central image output unit 60, i.e., by laser scanning for displaying the central image (S14).

[0052] If the point of focus is within the reproducible range of the central image output unit 60 (Y in S14), the central image size control unit 54 determines the size of the central image according to the position of the point of focus, as shown in Figure 6 (S16). If the point of focus is outside the reproducible range of the central image output unit 60 (N in S14), the central image size control unit 54 determines the central image to its maximum size, i.e., the maximum reproducible range (S18). This results in the lowest resolution of the central image, reducing the possibility of an unnatural difference in resolution between the central image and the surrounding image being visible. Alternatively, the minimum resolution of the central image and the resolution of the surrounding image may be unified so that the resolution is uniform when the point of focus is in the surrounding image.

[0053] The central image size control unit 54 periodically notifies the central image output unit 60 of the determined central image size, causing it to set the amplitude of the MEMS mirror angle corresponding to that size (S20). Meanwhile, the central image generation unit 56 and the peripheral image generation unit 58 acquire the necessary data from the image data acquisition unit 50 based on the central image size notified by the central image size control unit 54, and generate the central image and peripheral image, respectively (S22). The central image output unit 60 and the peripheral image output unit 62 then display the central image and peripheral image, respectively, so that the display image synthesized by the image synthesis unit 64 reaches the user's eyes (S24).

[0054] During periods when there is no need to terminate the display, such as when a user action is taken to end the display of content, the processes from S12 to S24 are repeated (N in S26). This allows the image display to continue while changing the range and resolution of the central image as the point of focus moves. When it becomes necessary to terminate the display, all processes are terminated (Y in S26).

[0055] Next, we will describe a specific structure for combining the central image and the peripheral image for viewing. Figure 9 shows an example of the structure of the image synthesis unit 64 that combines the central image and the peripheral image for viewing in the head-mounted display 100 of this embodiment. This figure schematically shows the positional relationship between the user's eyes 224 and the display units, including the first display unit 132 and the second display unit 134, when the head-mounted display 100 is worn, in a vertical cross-sectional view. Figures 10 to 13, which will be described later, are similar.

[0056] In the embodiment shown in Figure 9, the first display unit 132 includes a laser light source 220, a mirror 222, and a central image screen 232 made of a material that diffuses and transmits reflected laser light. On the other hand, the second display unit 134 includes a peripheral image display panel 234 made of a two-dimensional array of light-emitting elements. The central image screen 232 and the peripheral image display panel 234 are set to a 90° angle, and a half-mirror 236 is placed in the middle of them at a 45° angle to each to combine the central image and the peripheral image. The half-mirror 236 can be a general type that transmits a predetermined percentage of the incident light and reflects the remainder.

[0057] In the example shown in the figure, the laser light reflected by the mirror 222 is diffusely transmitted through the central image screen 232, passes through the half mirror 236, and reaches the eye 224 via the eyepiece lens 238. In other words, the first display unit 132 controls the operation of the mirror 222, etc., so that the original image is displayed in a diffusely transmitted state through the central image screen 232. On the other hand, the light from the peripheral image display panel 234 is reflected by the half mirror 236 and reaches the eye 224 via the eyepiece lens 238. As a result, the central image and peripheral image are viewed in a combined state.

[0058] However, the relative positions of the first display unit 132 and the second display unit 134 may be reversed, and the laser light from the mirror 222 may be reflected by the half-mirror 236, allowing the light from the peripheral image display panel 234 to pass through and reach the eye 224. Alternatively, the peripheral image may be displayed using a laser scanning method instead of the peripheral image display panel 234. In any case, in this configuration, the eyeball imaging camera 240 included in the gaze point detector 130 may be positioned next to the eyepiece lens 238, as shown in the figure.

[0059] Figure 10 shows another example of the structure of the image synthesis unit 64 in the head-mounted display 100 of this embodiment, which combines a central image and a peripheral image for viewing. This configuration differs from Figure 9 in that the first display unit 132 does not have a screen for the central image, and instead projects an image consisting of laser light directly onto the user's retina. As described above, known techniques can be applied to project the image onto the retina using the principle of Maxwell's vision. In other words, the first display unit 132 controls the operation of the mirror 222 and the like so that the original image is visible when the laser light is focused at the pupil and formed on the retina.

[0060] However, in this embodiment, the central image is projected via the half-mirror 236, and is combined with the peripheral image displayed on the peripheral image display panel 234 and reflected by the half-mirror 236 for viewing. In this case, by not providing a screen for the central image, the degree of freedom in the placement of the eyeball imaging camera 240 included in the gaze point detector 130 is increased. For example, as shown in the figure, it becomes possible to photograph the eyeball from near the front via the half-mirror 236.

[0061] Figure 11 shows another example of the structure of the image synthesis unit 64 in the head-mounted display 100 of this embodiment, which combines a central image and a peripheral image for viewing. In this configuration, the central image screen 242 that diffusely transmits the laser light of the first display unit 132 is provided integrally with the peripheral image display 244, and a half mirror is not provided, which is different from Figure 9. Light-transmitting displays that can transmit light from the background in areas of the display panel where no image is displayed (non-display areas) are known (see, for example, International Publication No. 2014 / 010585). In this embodiment, this is applied, and the peripheral image display 244 is made of a light-transmitting display substrate made of a semi-transparent material.

[0062] This allows the area of ​​the peripheral image display 244 that is not displaying the peripheral image to be used as a central image screen 242 that diffusely transmits the laser light reflected by the mirror 222. Naturally, the range of the peripheral image changes according to the change in the size of the central image, so the range of the central image screen 242 changes appropriately according to the size of the central image. In this case, a part of the central image output unit 60 and the peripheral image output unit 62 also serves as the image synthesis unit 64. By adopting this configuration, the optical system can be simplified compared to projecting two types of images from different directions. In this configuration, as in Figure 9, the eyeball imaging camera 240 included in the gaze point detector 130 can be placed next to the eyepiece lens 238, for example.

[0063] Figure 12 shows another example of the structure of the image synthesis unit 64 in the head-mounted display 100 of this embodiment, which combines a central image and a peripheral image for viewing. Similar to Figure 11, this configuration uses a light-transmitting display to integrally provide a screen 242 for the central image and a display 244 for the peripheral image. However, it differs from Figure 11 in that a half mirror 246 is provided between it and the eyepiece 238. That is, the light from the central image screen 242 and the peripheral image display 244 is viewed through the half mirror 246. In this way, the amount of light in the image is reduced, but if the half mirror 246 is positioned at a 45° angle with the surface of the eye 224, the reflection allows the eye 224 image to be captured by the eyeball imaging camera 240, so that the point of fixation can be detected with the same quality as when photographed from the front.

[0064] Figure 13 shows another example of the structure of the image synthesis unit 64 in the head-mounted display 100 of this embodiment, which combines a central image and a peripheral image for viewing. In this configuration, as in Figure 9, a screen 250 for the central image and a display panel 252 for the peripheral image are provided separately, but they are arranged on substantially the same plane, and image synthesis is achieved by guiding each image in the appropriate direction using the optical system 254 for the central image and the optical system 256 for the peripheral image.

[0065] A method for introducing a free-form optical system into a head-mounted display and guiding images displayed on multiple displays to the appropriate position through reflection and refraction to be viewed as a single image is disclosed, for example, in International Publication No. 2019 / 147946. With this configuration, the head-mounted display 100 can be miniaturized compared to projecting two types of images from different directions. In addition, because the optical path can be designed relatively freely, the degree of freedom in the placement of the eye-capturing camera 240 can be increased.

[0066] For example, as shown in the figure, by arranging the central image screen 250, the peripheral image display panel 252, and their respective optical systems 254 and 256 so as not to be directly in front of the eye 224, the ophthalmophotography optical system 258 and the ophthalmophotography camera 240 can be positioned directly in front of the eye 224. This makes it easier to detect the point of fixation. The position and orientation of the central image screen 250 and the peripheral image display panel 252 may vary depending on the design of the central image optical system 254 and the peripheral image optical system 256.

[0067] According to the embodiment described above, the display image is divided into a central image and a peripheral image, and the central image is displayed using separate mechanisms so that it is represented at a higher resolution, and then combined for viewing. Here, at least the display mechanism representing the central image uses a laser scanning method that forms an image by scanning a laser in two dimensions through reflection from a mirror. With the laser scanning method, the display range and the resolution as the density of pixels representing it can be changed by controlling the amplitude of the mirror angle. Therefore, it is easier to control the resolution distribution in the display image compared to a display panel with a fixed density of light-emitting elements.

[0068] For example, the amplitude of the mirror angle is changed so that as the user's point of focus moves away from the center of the image plane, the central image expands to encompass the area around it. This makes the boundary between the central and peripheral images separate from the point of focus, regardless of the position of the point of focus, making it difficult to perceive the resolution difference at the boundary. Furthermore, as the resolution decreases with the expansion of the central image, the actual resolution difference also becomes smaller. This allows for the recognition of images with distributed resolutions without the need for high-load processing such as image data manipulation. As a result, resources can be concentrated on areas with high discriminatory ability in terms of visual characteristics, making it possible to display wide-field images with low latency and high resolution.

[0069] The present invention has been described above based on embodiments. The above embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of their respective components and processing processes, and that such modifications also fall within the scope of the present invention.

[0070] For example, the implementation of the present invention is not limited to head-mounted displays, but can also be applied to projectors and general television receivers. In these cases as well, the internal structure may be the same as in any of Figures 9, 11, 12, or 13. In the case of a projector, a projection lens that projects an image onto an external screen is provided instead of the eyepiece lens 238. In the case of a television receiver, a screen that transmits the image is used as the display surface instead of the eyepiece lens 238. Alternatively, according to the configuration in Figure 11, a light-transmitting display that realizes the peripheral image display 244 can be used as is.

[0071] In these cases, the gaze point detector is naturally positioned with a camera for capturing eye images in the direction of the user's eyes as they view the screen or display surface. In this embodiment, the size of the central image was controlled according to the movement of the gaze point, but instead of the gaze point, the size could also be controlled according to the display position of the main object or the position of an important area on the display, so that it is included in the central image. Since the main object and important areas are more likely to be gazed at, it is conceivable that the same effect as in this embodiment could be achieved in this way as well. [Explanation of symbols]

[0072] 50 Image data acquisition unit, 52 Point of focus acquisition unit, 54 Central image size control unit, 56 Central image generation unit, 58 Peripheral image generation unit, 60 Central image output unit, 62 Peripheral image output unit, 64 Image synthesis unit, 100 Head-mounted display, 110 Stereo camera, 120 CPU, 122 GPU, 124 Main memory, 130 Point of focus detector, 132 First display unit, 134 Second display unit.

Claims

1. A central image generation unit generates a central image representing the central portion of the displayed image in the image plane, A peripheral image generation unit generates a peripheral image representing the region outside the central image from the displayed image, A laser scanning method is used to project an image by scanning a laser beam representing a pixel in two dimensions using the reflection of a mirror, and a central image output unit representing the central image is provided. A peripheral image output unit that displays the aforementioned peripheral image, An image synthesis unit that combines the aforementioned central image and the aforementioned peripheral image to make it visible, Equipped with, The central image output unit includes a screen that diffusely transmits the laser light reflected by the mirror, and the image synthesis unit synthesizes the image obtained by diffusing the light through the screen and the image displayed by the peripheral image output unit using a free-form surface optical system that guides both to corresponding positions on the image plane. or The peripheral image output unit displays the peripheral image using a light-transmitting display that transmits background light in the non-display area, and the central image output unit projects the laser light reflected by the mirror onto the corresponding area of ​​the light-transmitting display and diffuses it through it. An image display device characterized by the following.

2. The system further comprises a central image size control unit that controls the size of the central image, The image display device according to claim 1, characterized in that the central image output unit changes the amplitude of the mirror angle according to the size of the central image determined by the central image size control unit.

3. It further includes a gaze point acquisition unit that acquires the user's gaze point on the displayed image, The image display device according to claim 2, characterized in that the central image size control unit expands the central image as the point of fixation moves away from the center of the image plane, so that the point of fixation is included in the central image.

4. The image display device according to claim 3, characterized in that the central image size control unit sets the size of the central image to the upper limit when the point of focus exceeds the upper limit of the range that the central image output unit can display.

5. In the aforementioned light-transmitting display, when the peripheral image is displayed and the laser light reflected by the mirror is diffusely transmitted, A half-mirror that transmits light from the light-transmitting display and reflects the image of the user's eyeball, A camera for capturing the image of the eyeball reflected by the half-mirror and acquiring the point of fixation, The image display device according to claim 1, further comprising the features described above.

6. When the image obtained by diffuse transmission through the aforementioned screen and the image represented by the peripheral image output unit are combined by a free-form surface optical system that guides them to corresponding positions on the image plane, The image display device according to claim 1, further comprising a camera positioned in front of the user's eyeball for photographing the eyeball and acquiring the point of fixation, wherein the screen and the display panel constituting the peripheral image output unit are arranged around the camera.

7. The steps include generating a central image representing the central portion of the displayed image in the image plane, The steps include generating a peripheral image representing the area outside the central image from the displayed image, The central image output unit of a laser scanning system projects an image by scanning a laser beam representing a pixel in two dimensions using mirror reflection, and the process involves representing the central image, The peripheral image output unit performs the step of displaying the peripheral image, The steps include combining the central image and the surrounding image to make them visible, Includes, The aforementioned step of combining and making visible involves combining the image formed by diffusing and transmitting the laser light reflected by the mirror through a screen, and the image displayed by the peripheral image output unit, using a free-form optical system that guides both to corresponding positions on the image plane. or The step of displaying the surrounding image involves displaying the surrounding image using a light-transmitting display that transmits background light in a non-display area, and the step of representing the central image involves projecting the laser light reflected by the mirror onto a corresponding area of ​​the light-transmitting display and causing it to diffusely transmit. An image display method characterized by the following.

Images