Image processing system, image processing method, and program

Abstract

An image processing system (S) for generating a display image that is displayed on a display panel (10) provided in a head-mounted display (1) and that is enlarged and viewed through a lens unit (20) which is provided so as to correspond to the display panel (10), the image processing system including: an inclination correction unit (62) for correcting image data input into the display panel (10) on the basis of an inclination correction value relating to the display image in the three-dimensional space; and an image generation unit (63) for generating the display image on the basis of the image data corrected on the basis of the inclination correction value.

Description

Image processing system, image processing method, and program

[0001] The present disclosure relates to an image processing system, an image processing method, and a program.

[0002] 2. Description of the Related Art Conventionally, a head-mounted display having a display panel for displaying an image is known, as disclosed in Patent Document 1.

[0003] International Publication No. 2016 / 136657

[0004] The inventors of the present invention have investigated a configuration in which a lens unit corresponding to the display panel is provided in addition to the eyepiece lens in order to improve the resolution of the displayed image. In such a configuration in which the display panel is enlarged and displayed using the lens unit, the display panel and the lens unit require positional accuracy so that the user can view the displayed image without feeling uncomfortable. While it is desirable to appropriately design the placement of the display panel and the lens unit, there are limits to this accuracy.

[0005] An object of the present disclosure is to provide an image processing system, an image processing method, and a program that allow a user to view a display image without feeling unnatural.

[0006] The image processing system proposed in the present disclosure is an image processing system that generates a display image that is displayed on a display panel provided in a head-mounted display worn on a user's head and that is enlarged and viewed through a lens unit provided corresponding to the display panel, and includes a tilt correction means that corrects image data input to the display panel based on a tilt correction value related to the display image in three-dimensional space, and an image generation means that generates the display image based on the image data corrected based on the tilt correction value.

[0007] FIG. 1 is a perspective view of an HMD according to this embodiment, as seen from the front left; FIG. 2 is a left side view of an HMD according to this embodiment, as seen from the left; FIG. 3 is a diagram schematically showing the configuration of a display panel and an optical system according to this embodiment; FIG. 4 is a plan view showing a view of the display panel according to this embodiment, as seen from the front; FIG. 5 is a diagram showing an example of a display image displayed on each display panel; FIG. 6 is a diagram showing an example of a hardware configuration of an image processing system according to this embodiment; FIG. 7 is a functional block diagram showing an example of a function realized by the image processing system according to this embodiment; FIG. 8 is a functional block diagram showing an example of a function realized by the image processing system according to this embodiment; FIG. 9 is a flowchart showing an example of acquisition control of correction values ​​in the image processing system according to this embodiment; FIG. 10 is a flowchart showing an example of display control in the image processing system according to this embodiment.

[0008] An embodiment of the present invention (hereinafter referred to as the present embodiment) will be described below with reference to the drawings. In the following description, a head-mounted display 1 will be referred to as an "HMD (Head Mounted Display) 1." In the following description, the directions indicated by X1 and X2 in the drawings will be referred to as right and left, respectively, the directions indicated by Y1 and Y2 in the drawings will be referred to as up and down, respectively, and the directions indicated by Z1 and Z2 in the drawings will be referred to as forward and backward, respectively. These directions indicate directions as seen by a user wearing the HMD 1. However, the left-right, up-down, and front-back directions do not indicate strict directions.

[0009] [Overview of HMD 1] Fig. 1A is a perspective view of an HMD according to this embodiment as seen from the front left. Fig. 1B is a left side view of the HMD according to this embodiment as seen from the left. In the following description, the state in which the HMD 1 is worn by a user is referred to as the "usage state." In the usage state, the HMD 1 covers the area in front of the user's eyes.

[0010] The HMD 1 preferably has a mounting band that surrounds the user's head and a main body housing that is supported on the front of the mounting band. The main body housing contains the display unit 100 and various components that constitute the optical system, such as an eyepiece 200.

[0011] The display unit 100 includes a display panel 10, which will be described later. The display panel 10 may display a three-dimensional image. The display panel 10 may be, for example, a liquid crystal display or an organic electroluminescence display device, but the type of the display panel is not particularly limited.

[0012] The HMD 1 shown in Figures 1A and 1B is an example and is not limited thereto. The HMD 1 is only required to be worn on the user's head and to include a display panel that displays images, and the shapes of the attachment band and main body housing are not limited to those shown. Also, Figure 1B shows the display unit 100 and eyepiece 200 schematically, and the shapes and sizes of these are not limited to those shown.

[0013] Next, the configuration of the display panel and optical system in this embodiment will be described in detail with reference to Figures 2 and 3. Figure 2 is a diagram schematically illustrating the configuration of the display panel and optical system in this embodiment. Figure 3 is a plan view showing a front view of the display panel in this embodiment. While Figure 2 illustrates a display panel and optical system on either the right eye side or the left eye side of the user U, the configuration shown in Figure 2 may be provided on both the right eye side and the left eye side. However, the configuration is not limited to a compound eye lens, and a single eye lens may also be used.

[0014] [Eyepiece 200] The eyepiece 200 constitutes part of the optical system of the HMD 1. When in use, the eyepiece 200 is located in front of the right or left eye of the user U, and is a convex lens that focuses the image light IL emitted from the display panel 10 toward the eyeball of the user U. The eyepiece 200 has the function of enlarging the image displayed on the display panel 10 so that the user can view it. Note that the eyepiece 200 may be composed of two or more lenses. The eyepiece 200 may also be a Fresnel lens or a liquid crystal lens.

[0015] 2, the optical axis 200C of the eyepiece 200 is indicated by a dashed line. The optical axis is an imaginary axis that passes through the center of the lens and is perpendicular to the surface of the lens.

[0016] [Display unit 100] The display unit 100 includes a plurality of display panels 10, a plurality of lens units 20 provided corresponding to each of the plurality of display panels 10, and a support housing (not shown) that supports the display panels 10 and the lens units 20.

[0017] In this embodiment, an example will be described in which the multiple display panels 10 include a front panel 10C, a right panel 10R, and a left panel 10L. The front panel 10C, the right panel 10R, and the left panel 10L may be, for example, micro OLEDs of 0.49 inches or less. By arranging three 0.49-inch (2K) micro OLEDs side by side, it is possible to create a micro OLED equivalent to a 1.5-inch (6K) micro OLED in appearance. The cost of using three 0.49-inch micro OLEDs is approximately one-tenth of the cost of using 1.5-inch micro OLEDs.

[0018] In this embodiment, an example will be described in which the multiple lens units 20 are a front lens unit 20C, a right lens unit 20R, and a left lens unit 20L. These lens units 20, together with an eyepiece lens 200, form an optical system. The lens units 20 are disposed in front of the display panel 10 and are not in contact with the display panel 10. The distance between the lens units 20 and the display panel 10 may be appropriately set to obtain a desired focal length. Note that the distance between the center of the right lens unit 20R and the center of the right panel 10R may be longer than the distance between the center of the front lens unit 20C and the center of the front panel 10C. Similarly, the distance between the center of the left lens unit 20L and the center of the left panel 10L may be longer than the distance between the center of the front lens unit 20C and the center of the front panel 10C. This can improve aberrations. The center of the right lens unit 20R is the portion that faces the right panel 10R and through which the optical axis OR passes, and the center of the right panel 10R is the portion that faces the right lens unit 20R and is located in the center in the vertical and horizontal directions. Similarly, the center of the left lens unit 20L is the portion that faces the left panel 10L and through which the optical axis OL passes, and the center of the left panel 10L is the portion that faces the left lens unit 20L and is located in the center in the vertical and horizontal directions.

[0019] In this embodiment, when there is no need to distinguish between the front panel 10C, the right panel 10R, and the left panel 10L, they will simply be referred to as the "display panel 10." Similarly, when there is no need to distinguish between the front lens unit 20C, the right lens unit 20R, and the left lens unit 20L, they will simply be referred to as the "lens unit 20."

[0020] 3, the front panel 10C includes a display area DC and a frame area FC surrounding the display area DC. The display area DC includes a plurality of unit pixels and is an area where an image is displayed. The frame area FC is an area where no image is displayed.

[0021] 2, the front panel 10C is provided in front of the eyepiece 200. Specifically, the front panel 10C is provided on an extension of the optical axis 200C of the eyepiece 200.

[0022] 3, the right panel 10R includes a display region DR and a frame region FR surrounding the display region DR. The display region DR includes a plurality of pixels and is an area where an image is displayed. The frame region FR is an area where no image is displayed.

[0023] The right panel 10R is provided adjacent to the right side of the front panel 10C. Therefore, the left portion of the frame region FR of the right panel 10R and the right portion of the frame region FC of the front panel 10C are provided adjacent to each other. The right panel 10R is not provided on an extension of the optical axis 200C of the eyepiece 200. The right panel 10R is provided to the right of the optical axis 200C and faces the eyepiece 200.

[0024] 2, the display surface of the display region DR of the right panel 10R is inclined relative to the display surface of the display region DC so that the right portion of the right panel 10R is closer to the eyepiece lens 200. Specifically, the display surface of the display region DR of the right panel 10R is inclined at 6.5 degrees relative to the display surface of the display region DC of the front panel 10C. However, the present invention is not limited to this, and the display surface of the display region DR of the right panel 10R may be inclined at 6 to 7 degrees relative to the display surface of the display region DC of the front panel 10C.

[0025] 3, the left panel 10L includes a display area DL and a frame area FL surrounding the display area DL. The display area DL includes a plurality of pixels and is an area where an image is displayed. The frame area FL is an area where no image is displayed.

[0026] The left panel 10L is provided adjacent to the left side of the front panel 10C. Therefore, the right portion of the frame region FL of the left panel 10L and the left portion of the frame region FC of the front panel 10C are provided adjacent to each other. The left panel 10L is not provided on an extension of the optical axis 200C of the eyepiece 200. The left panel 10L is provided to the left of the optical axis 200C and faces the eyepiece 200.

[0027] 2, the display surface of the display area DL of the left panel 10L is inclined relative to the display surface of the display area DC so that the left portion of the left panel 10L is closer to the eyepiece lens 200. Specifically, the display surface of the display area DL of the left panel 10L is inclined at 6.5 degrees relative to the display surface of the display area DC of the front panel 10C. However, the present invention is not limited to this, and the display surface of the display area DL of the left panel 10L may be inclined at 6 to 7 degrees relative to the display surface of the display area DC of the front panel 10C.

[0028] Each display panel 10 may be a flexible panel having flexibility. For example, the left panel 10L and the right panel 10R may be flexible panels that are curved.

[0029] [Display unit 100: lens section 20] Front lens section 20C is provided in front of eyepiece lens 200. Specifically, front lens section 20C is provided on an extension of optical axis 200C of eyepiece lens 200. Therefore, front lens section 20C is provided opposite front panel 10C.

[0030] The front lens unit 20C is a convex lens that guides the image light IL emitted from the display area DC to the eyepiece 200 and enlarges the display area DC so that it can be viewed by the user. The front lens unit 20C is disposed so that its optical axis OC is perpendicular to the display surface of the front panel 10C. The size of the front lens unit 20C is preferably approximately the same as that of the front panel 10C or large enough to cover at least half of the display surface of the front panel 10C.

[0031] The right lens unit 20R is provided adjacent to the right side of the front lens unit 20C. The right lens unit 20R is provided to the right of the optical axis 200C and faces the eyepiece 200. The right lens unit 20R is not provided on an extension of the optical axis 200C of the eyepiece 200.

[0032] The right lens unit 20R is a convex lens that guides the image light IL emitted from the display region DR to the eyepiece lens 200, enlarging the display region DR for viewing by the user. The right lens unit 20R is disposed so that its optical axis OR is perpendicular to the display surface of the right panel 10R. Therefore, the optical axis OR is inclined with respect to the optical axis OC. Specifically, the optical axis OR of the right lens unit 20R is inclined at 6.5 degrees with respect to the optical axis OC of the front lens unit 20C. However, this is not limited thereto, and the optical axis OR of the right lens unit 20R may be inclined at 6 to 7 degrees with respect to the optical axis OC of the front lens unit 20C. The size of the right lens unit 20R is preferably approximately the same as that of the right panel 10R, or large enough to cover at least half of the display surface of the right panel 10R.

[0033] The left lens unit 20L is provided adjacent to the left side of the front lens unit 20C. The left lens unit 20L is provided to the left of the optical axis 200C and faces the eyepiece 200. The left lens unit 20L is not provided on an extension of the optical axis 200C of the eyepiece 200.

[0034] The left lens unit 20L is a convex lens that guides the image light IL emitted from the display area DL to the eyepiece lens 200, enlarging the display area DL for viewing by the user. The left lens unit 20L is disposed so that its optical axis OL is perpendicular to the display surface of the left panel 10L. Therefore, the optical axis OL is inclined with respect to the optical axis OC. Specifically, the optical axis OL of the left lens unit 20L is inclined at 6.5 degrees with respect to the optical axis OC of the front lens unit 20C. However, this is not limited thereto, and the optical axis OL of the left lens unit 20L may be inclined at 6 to 7 degrees with respect to the optical axis OC of the front lens unit 20C. The size of the left lens unit 20L may be approximately the same as that of the left panel 10L, or may be large enough to cover at least half of the display surface of the left panel 10L.

[0035] In this embodiment, the lens unit 20 is an aspherical lens, but this is not limited thereto, and a spherical lens or a free-form lens may also be used. Furthermore, in this embodiment, as shown in FIG. 2, the front lens unit 20C, the right lens unit 20R, and the left lens unit 20L are integrally configured. However, this is not limited thereto, and these may be separate units. When the front lens unit 20C, the right lens unit 20R, and the left lens unit 20L are integrally configured, it is possible to easily improve the relative positional accuracy of each lens unit. On the other hand, when the front lens unit 20C, the right lens unit 20R, and the left lens unit 20L are separate units, as shown in FIG. 2, the front lens unit 20C, the right lens unit 20R, and the left lens unit 20L may be lenses with the same curved surface shape, thereby reducing mold costs and avoiding complex process management. However, the front lens unit 20C, the right lens unit 20R, and the left lens unit 20L may have different shapes.

[0036] 3, each display panel 10 may be provided with flexible wiring boards 80C, 80R, and 80L for supplying image signals (image data) to the display area thereof. The flexible wiring boards 80C, 80R, and 80L may also be wiring boards for supplying power.

[0037] The flexible wiring boards 80C, 80R, and 80L may be provided so as to extend in a direction perpendicular to the direction in which the display panels 10 are arranged (the left-right direction in this embodiment). In Fig. 3, each flexible wiring board is attached to the top of each display panel 10 and extends upward.

[0038] [Summary of the configuration of the head-mounted display] The HMD 1 according to this embodiment allows a user to view high-resolution images with a wide angle of view using an inexpensive, small panel. Furthermore, by adopting a configuration in which multiple lens units 20 are provided corresponding to each of the multiple display panels 10, the user can view a continuous image even when using a display panel 10 with a frame area.

[0039] Furthermore, using multiple small display panels can reduce costs compared to using one large display panel, because the purchase cost of commercially available display panels can be as much as 10 times or more if a display panel with a display area twice as large is used.

[0040] Generally, in an HMD requiring a wide viewing angle, increasing the resolution of the eyepiece requires the addition of a lens with a diameter and thickness comparable to that of the eyepiece. This results in the HMD becoming larger in the front-to-back direction. In this embodiment, by adopting a configuration in which multiple lens units 20 are arranged side by side in addition to the eyepiece 200, the focal length of the optical system can be shortened without increasing the thickness of the eyepiece 200 or increasing the curvature of the lens surface. Therefore, an HMD 1 can be provided that achieves high resolution of the displayed image without increasing the size in the front-to-back direction (depth direction). It is preferable that no other lenses are provided between the eyepiece 200 and the lens units 20 in the front-to-back direction.

[0041] Furthermore, the imaging performance of the eyepiece 200 is higher on the side closer to the optical axis 200C and lower on the outer edge of the eyepiece 200. That is, images tend to appear blurred on the outer edge of the eyepiece 200 compared to the vicinity of the optical axis 200C. In this embodiment, by providing a lens unit 20 corresponding to each display area, it is possible to improve visibility even in the area on the outer edge of the eyepiece 200. Furthermore, as described above, in this embodiment, the right panel 10R and the left panel 10L are tilted relative to the front panel 10C so as to approach the eyepiece 200, which makes it possible to improve visibility even in the area on the outer edge of the eyepiece 200.

[0042] [Display Image] Fig. 4 is a diagram showing an example of a display image displayed on each display panel. Fig. 4 shows a grid shape including a plurality of grid points displayed as a display image in the display area of ​​each display panel. Fig. 4 also shows the display panel 10 viewed without passing through an optical system including the lens unit 20. Furthermore, yaw, pitch, and roll shown in Fig. 4 represent the tilt direction of the display image in three-dimensional space.

[0043] In this embodiment, since a lens unit 20 is provided corresponding to each display panel 10, each displayed image will be distorted when viewed through the lens unit 20. Therefore, distortion correction must be performed using electronic means. Distortion correction is a correction that distorts the original image in the opposite direction according to the distortion aberration so that an undistorted image is viewed. By performing distortion correction to display an image that is the opposite of the distortion, taking into account the amount of distortion according to the optical performance of the optical system, each displayed image will be viewed as having a rectangular outer shape. This makes it possible to allow a user to view a continuous displayed image without any discomfort in a configuration using multiple display panels 10.

[0044] Here, the HMD 1 of this embodiment requires high positional accuracy and high orientation accuracy of the display panel 10 and the lens unit 20. For example, if the display panel 10 is tilted in the rotation direction (yaw shown in FIG. 4 ) around a virtual axis extending perpendicular to the display surface of the display panel 10, the user will view a tilted display image. This will cause problems at the joints of the images displayed on each display panel.

[0045] Furthermore, in a configuration in which a user views a continuous display image using multiple display panels 10, particularly high precision is required for the position and orientation of the display panels 10 and the lens unit 20. For example, if the optical axis OC of the front lens unit 20C is tilted with respect to the direction perpendicular to the display surface of the front panel 10C, the display image viewed through the front lens unit 20C will not be distorted as intended, and the display image viewed after distortion correction will also not have the intended shape. As a result, problems will occur at the joints between the display image on the front panel 10C and the display images on the right panel 10R and left panel 10L.

[0046] Furthermore, for example, if the relative positions of the right panel 10R and the left panel 10L with respect to the front panel 10C are not appropriate, problems will occur at the joints between the image displayed by the front panel 10C and the image displayed by the right panel 10R and the left panel 10L.

[0047] It is advisable to appropriately design the layout of the display panel 10 and the lens unit 20 so that no problems occur at the joints between the displayed images, but there is a limit to how accurate this can be.

[0048] Furthermore, according to the research of the inventors of the present invention, it was found that in order to make multiple displayed images visible as a continuous image without any sense of discomfort, it is important to correct the tilt of the displayed images three-dimensionally.

[0049] Therefore, in this embodiment, an image processing system S is provided that can correct the tilt of the displayed image so that the user can view an image that does not look strange, even if there is a design error in the position or attitude of the display panel 10 and the lens unit 20.

[0050] [Hardware Configuration of Image Processing System] The image processing system S according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the hardware configuration of the image processing system according to this embodiment.

[0051] The image processing system S is a computer mounted on the HMD 1 or a computer mounted on a device separate from the HMD 1. Furthermore, part of the hardware configuration of the image processing system S may be mounted on the HMD 1, and the other part may be mounted on a separate device. In other words, the image processing system S may be configured from multiple computers. The multiple computers may be connected to each other wirelessly or via a wire.

[0052] As shown in FIG. 5, the image processing system S includes, in addition to the display panel 10, a control unit 101, a storage unit 102, a communication unit 103, an operation unit 104, an audio output unit 105, and an imaging device 106.

[0053] The control unit 101 includes at least one processor. The control unit 101 may be, for example, a program-controlled device such as a CPU that operates according to a program installed in the image processing system S. The control unit 101 also includes a GPU (Graphics Processing Unit) that draws images in a frame buffer based on graphics commands and data supplied from the CPU.

[0054] The storage unit 102 includes, for example, a main storage device such as a ROM or a RAM, and an auxiliary storage device such as a HDD or an SSD. The storage unit 102 stores programs executed by the control unit 101. The storage unit 102 stores, for example, a game program (game software) in addition to programs for realizing various functions of the image processing system S, which will be described later. The storage unit 102 also reserves an area for a frame buffer in which images are drawn by the GPU.

[0055] The communication unit 103 is a communication interface such as an Ethernet (registered trademark) module or a wireless LAN module.

[0056] The operation unit 104 is a user interface such as a controller, and receives operation input from the user and outputs a signal indicating the content of the input to the control unit 101 .

[0057] The audio output unit 105 is, for example, a speaker, and outputs audio represented by audio data generated by the image processing system S.

[0058] The image capturing device 106 may be, for example, a camera, etc. The image capturing device 106 may be provided so as to be able to capture an image of the display area of ​​the display panel 10 of the HMD 1.

[0059] [Functions Realized by the Image Processing System] Fig. 6 is a functional block diagram showing an example of functions realized by the image processing system according to this embodiment. As shown in Fig. 6, the image processing system S realizes a reference position acquisition unit 51, a plane correction value acquisition unit 52, a tilt correction value acquisition unit 53, a magnification correction value acquisition unit 54, a distortion correction value acquisition unit 55, a correction information storage unit 56, and a distortion correction unit 57.

[0060] The reference position acquisition unit 51, the plane correction value acquisition unit 52, the tilt correction value acquisition unit 53, the magnification correction value acquisition unit 54, the distortion correction value acquisition unit 55, and the distortion correction unit 57 are mainly realized by the control unit 101. The correction information storage unit 56 is mainly realized by the storage unit 102.

[0061] The reference position acquisition unit 51 acquires a reference position P0 indicating the center of the display area DL based on, for example, an image acquired by capturing an image of the display area DL of the front panel 10C using the imaging device 106. The reference position acquisition unit 51 may also acquire reference positions P1 and P2 in addition to the reference position P0, which is the optical origin. In this embodiment, the grid points indicated by P0, P1, and P2 in FIG. 4 are described as reference positions. However, this is merely an example, and multiple points, including three points that are not aligned in a straight line, may be used as reference positions.

[0062] The plane correction value acquisition unit 52 acquires a plane correction value. The plane correction value is information related to position correction of the displayed image in the vertical and horizontal directions. For example, the plane correction value acquisition unit 52 may acquire, as the plane correction value, the amount of deviation of the reference position acquired by the reference position acquisition unit 51 from an ideal reference position determined in advance from design values.

[0063] Here, the "ideal reference position determined in advance from the design values" is a position (hereinafter referred to as the "given position") that is determined in advance based on at least the arrangement of the display panel 10, the lens unit 20, and the imaging device 106, and the optical performance of the lens unit 20.

[0064] The tilt correction value acquisition unit 53 acquires a tilt correction value. The tilt correction value is information related to tilt correction of a displayed image in three-dimensional space according to a tilt error of at least one of the display panel 10 and the lens unit 20. Specifically, the tilt correction value is a correction value related to a rotation direction (yaw shown in FIG. 4 ) about a virtual axis extending in a direction perpendicular to the display surface of the display panel 10, a rotation direction (pitch shown in FIG. 4 ) about a virtual axis extending in the up-down direction of the display panel 10, and a rotation direction (roll shown in FIG. 4 ) about a virtual axis extending in the left-right direction of the display panel 10.

[0065] The tilt correction value acquisition unit 53 acquires a tilt correction value adjusted to minimize the error of the reference position acquired by the reference position acquisition unit 51 with respect to a given position determined in advance from a design value, for example.

[0066] The magnification correction value acquisition unit 54 acquires a magnification correction value. The magnification correction value is information related to the correction of the display magnification. In other words, the magnification correction value is information related to the enlargement or reduction of the display image. The magnification correction value acquisition unit 54 acquires, for example, a magnification correction value adjusted so as to minimize the error of the reference position acquired by the reference position acquisition unit 51 with respect to a given position determined in advance from a design value.

[0067] The distortion correction value acquisition unit 55 acquires a distortion correction value that has been adjusted to minimize an error between a given position determined in advance from a design value and the reference position acquired by the reference position acquisition unit 51, for example.

[0068] It should be noted that each correction value described here is preferably a value that is set so as to minimize the error in each reference position by using an optimization method such as the least squares method.

[0069] The correction information storage unit 56 stores the plane correction value acquired by the plane correction value acquisition unit 52, the tilt correction value acquired by the tilt correction value acquisition unit 53, the magnification correction value acquired by the magnification correction value acquisition unit 54, and the distortion correction value acquired by the distortion correction value acquisition unit 55.

[0070] In this embodiment, it is preferable to obtain a plane correction value, a tilt correction value, a magnification correction value, and a distortion correction value for each of the front panel 10C, the right panel 10R, and the left panel 10L.

[0071] Furthermore, in this embodiment, a common optical system is used for each display panel 10. That is, the front lens unit 20C, the right lens unit 20R, and the left lens unit 20L share common optical performance. Therefore, it is preferable to reuse the correction values ​​acquired for the front panel 10C for the right panel 10R and the left panel 10L, taking into consideration the design relative positions and relative inclinations of the right panel 10R and the left panel 10L relative to the front panel 10C. Specifically, for example, when acquiring distortion correction values ​​and magnification correction values ​​for the right panel 10R, it is preferable to use the distortion correction values ​​and magnification correction values ​​for the front panel 10C stored in the correction information storage unit 56 as initial values ​​(reference values) and search for optimal distortion correction values ​​and magnification correction values ​​for the right panel 10R. This makes it possible to efficiently acquire distortion correction values ​​and magnification correction values ​​for the right panel 10R. The same applies to the left panel 10L.

[0072] Fig. 7 is a functional block diagram showing an example of functions realized by the image processing system according to this embodiment. As shown in Fig. 7, the image processing system S realizes, in addition to the functions described with reference to Fig. 6, a viewpoint information acquisition unit 61, a tilt correction unit 62, an image generation unit 63, a display control unit 64, a plane position correction unit 71, a magnification correction unit 72, and a distortion correction unit 73. Each of these functions is realized mainly by the control unit 101.

[0073] The correction information storage unit 56 stores the plane correction value, tilt correction value, magnification correction value, and distortion correction value for each display panel 10, as described with reference to FIG.

[0074] Image data relating to video such as a game screen is input to the image processing system S. The image data includes viewpoint information corresponding to the orientation of the HMD 1. The viewpoint information acquisition unit 61 acquires the viewpoint information corresponding to the orientation of the HMD 1 based on the input image data.

[0075] The tilt correction unit 62 corrects the image data based on the tilt correction value stored in the correction information storage unit 56 so that the image displayed on the display panel 10 is tilted in the yaw, pichi, and roll directions.

[0076] The image generating unit 63 generates an image seen from a viewpoint based on the viewpoint information by rendering the image corrected by the tilt correcting unit 62 .

[0077] The plane position correction unit 71 corrects the plane position of the image generated by the image generation unit 63 based on the plane correction value stored in the correction information storage unit 56. In other words, the plane position correction unit 71 corrects the image data so that the image generated by the image generation unit 63 is displayed displaced in the X direction and the Y direction.

[0078] The magnification correction unit 72 corrects the display magnification of the image generated by the image generation unit 63 based on the magnification correction value stored in the correction information storage unit 56. That is, the magnification correction unit 72 corrects the image data based on the magnification correction value stored in the correction information storage unit 56 so that the image generated by the image generation unit 63 is displayed enlarged or reduced.

[0079] The distortion correction unit 73 corrects the image data based on the distortion correction value stored in the correction information storage unit 56 so as to distort the image generated by the image generation unit 63 .

[0080] The display control unit 64 causes the display panel 10 to display the image corrected based on the position correction value, the magnification correction value, and the distortion correction value as a display image.

[0081] [Correction Value Acquisition Control] Next, correction value acquisition control in the image processing system S will be described with reference to Fig. 8. Fig. 8 is a flowchart showing an example of correction value acquisition control in the image processing system according to this embodiment.

[0082] First, the reference position acquisition unit 51 acquires each reference position based on a captured image generated by capturing an image displayed on the front panel 10C (ST11). Next, the plane correction value acquisition unit 52 acquires, as a plane correction value, an error of the reference position acquired by the reference position acquisition unit 51 with respect to a given position determined in advance from a design value (ST12).

[0083] Next, distortion correction unit 57 performs distortion correction using a predetermined distortion correction value calculated in advance from the design value (ST13). As a result, the display image displayed on front panel 10C becomes approximately rectangular. However, since tilt correction and the like have not been performed at this stage, distortion and the like due to design errors in front panel 10C and front lens unit 20C remain.

[0084] Next, the tilt of the displayed image is corrected using a predetermined tilt correction value set by the user.The tilt correction value is then repeatedly changed within a predetermined range, and the tilt correction value that minimizes the error for a given position determined in advance from the design value is acquired by the tilt correction value acquisition unit 53 (ST14).

[0085] Next, the display image is enlarged or reduced using a predetermined magnification correction value set by the user.The magnification correction value is then repeatedly changed within a predetermined range, and the magnification correction value that minimizes the error for a given position determined in advance from the design value is acquired by the magnification correction value acquisition unit 54 (ST15).

[0086] Next, the distortion correction value set in ST13 is changed, and the distortion correction value that minimizes the error with respect to a given reference position determined in advance from the design value is acquired by the distortion correction value acquisition unit 55 (ST16).

[0087] Note that steps ST14 and ST15 may be repeated until optimal correction values ​​are obtained. For example, after obtaining a tilt correction value in ST14, a magnification correction value may be obtained in ST15, and then the process may return to ST14 to search for a more optimal tilt correction value.

[0088] Furthermore, correction values ​​for the image displayed on the right panel 10R are acquired. Steps ST21 to ST26 correspond to steps ST11 to ST16 described above. The following mainly describes the differences from steps ST11 to ST16.

[0089] In ST23, distortion correction can be performed using the distortion correction value for front panel 10C acquired in ST16. By using the distortion correction value for front panel 10C, which has the same optical performance as right panel 10R, it is possible to perform distortion correction that is close to ideal on right panel 10R as well.

[0090] In ST24, similar to ST14, the tilt correction value is repeatedly changed within a predetermined range, and the tilt correction value that minimizes the error for a given position determined in advance from the design value is acquired by the tilt correction value acquisition unit 53. When changing the tilt correction value within the predetermined range in ST24, the tilt correction value for the front panel 10C acquired in ST14 can be used as the initial value. However, the tilt correction value used as the initial value must be changed taking into account the relative position and relative tilt of the right panel 10R relative to the front panel 10C in terms of design. This makes it possible to efficiently derive the optimal tilt correction value for the right panel 10R.

[0091] In ST25, similar to ST15, the magnification correction value is repeatedly changed within a predetermined range, and the magnification correction value that minimizes the error for a given position determined in advance from the design value is acquired by the magnification correction value acquisition unit 54. When changing the magnification correction value within the predetermined range in ST25, the magnification correction value for the front panel 10C acquired in ST15 can be used as the initial value. However, the magnification correction value used as the initial value must be changed taking into account the design relative position and relative inclination of the right panel 10R with respect to the front panel 10C. This makes it possible to efficiently derive the optimal magnification correction value for the right panel 10R.

[0092] In ST26, the distortion correction value set in ST23 is changed, and the distortion correction value acquisition unit 55 acquires a distortion correction value that minimizes the error for a given position that is previously determined from the design value.

[0093] Although not shown in the drawings or described in detail, it is preferable to perform the same processes as in ST21 to ST26 on the left panel 10L to obtain each correction value. At this time, it is necessary to take into consideration the design relative position and relative inclination of the left panel 10L with respect to the front panel 10C.

[0094] Furthermore, the correction values ​​used to efficiently derive the correction values ​​for the left panel 10L may be the correction values ​​for the front panel 10C, as with the right panel 10R, or may be the correction values ​​for the right panel 10R. In other words, the correction values ​​to be used are not limited to those for the front panel 10C, and may be correction values ​​that have already been acquired for another display panel 10.

[0095] In this embodiment, an example in which there are three display panels 10 will be described, but even if there are four or more display panels 10, each correction value can be obtained by performing the same processing as ST21 to ST26.

[0096] [Display Control Flow] Next, display control in the image processing system S will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of display control in the image processing system according to this embodiment.

[0097] First, the viewpoint information acquisition unit 61 acquires viewpoint information corresponding to the orientation of the HMD 1 based on image data input to the image processing system S (ST31). Then, the tilt correction unit 62 corrects the image data based on the tilt correction value (ST32). Furthermore, the image generation unit 63 performs rendering based on the tilt-corrected image data to generate an image viewed from a viewpoint based on the viewpoint information (ST33). By performing tilt correction on the image data based on the tilt correction value before rendering in this way, it is possible to form an image in which objects to be displayed based on the viewpoint information are appropriately displayed.

[0098] Furthermore, the image generated by the image generation unit 63 is corrected based on the plane correction value, magnification correction value, and distortion correction value for the front panel 10C (ST34).Then, the display control unit 64 displays the image corrected in ST34 as a display image on the front panel 10C (ST35).

[0099] 9, the display control for the front panel 10C has been described, but similar display control may be performed for the right panel 10R and the left panel 10L. That is, the display control unit 64 may cause an image corrected based on the correction values ​​for the right panel 10R to be displayed as a display image on the right panel 10R, and an image corrected based on the correction values ​​for the left panel 10L to be displayed as a display image on the left panel 10L.

[0100] [Summary] In the embodiment described above, by performing display based on image data corrected based on the tilt correction value, it is possible to allow the user to view a display image that does not look strange even when there are design errors in the display panel 10 and the lens unit 20. In particular, in an HMD 1 that uses multiple display panels 10, it is possible to allow the user to view a seamless display image that does not look strange. Furthermore, in a configuration in which the display surfaces of multiple display panels 10 are arranged at an angle to each other, particularly high positional accuracy is required for the display panels 10 and the lens unit 20 in order to view a seamless display image, and the image processing system S according to this embodiment is useful.

[0101] [Others] In the embodiment, an example has been described in which correction values ​​for the front panel 10C are reused for other display panels 10, thereby enabling efficient acquisition of correction values ​​for the other display panels 10. However, this is not limiting. That is, correction values ​​for the other display panels 10 may be acquired independently without reusing the correction values ​​for the front panel 10C. Furthermore, acquisition of correction values ​​is not limited to the method described with reference to FIG. 6. That is, it is sufficient if correction values ​​including tilt correction values ​​are acquired by some method and stored in the correction information storage unit 56, and an image is generated using the stored tilt correction values.

[0102] In the above embodiment, an example has been described in which the display unit 100 includes a plurality of display panels 10, but the number of display panels 10 may be one. Also, the number of display panels 10 may be four or more. When a large number of display panels 10 are used, the number of seams between displayed images increases, making the image processing system S described in this embodiment more useful.

[0103] [Note] For example, the image processing system S may also be configured as follows: (1) An image processing system that generates a display image that is displayed on a display panel provided in a head-mounted display worn on a user's head and that is enlarged and viewed through a lens unit provided corresponding to the display panel, the image processing system including: a tilt correction unit that corrects image data input to the display panel based on a tilt correction value related to the display image in three-dimensional space; and an image generation unit that generates the display image based on the image data corrected based on the tilt correction value. (2) The head mounted display is provided with a plurality of the display panels and a plurality of the lens units, and the plurality of display panels include at least a first display panel that displays a first display image and a second display panel that is adjacent to the first display panel and displays a second display image, and the plurality of lens units include at least a first lens unit that is provided corresponding to the first display panel and a second lens unit that is provided corresponding to the second display panel, and the tilt correction means corrects first image data input to the first display panel based on a first tilt correction value related to the first display image in three-dimensional space, and corrects second image data that is input to the second display panel based on a second tilt correction value related to the second display image in three-dimensional space, and the image generation means generates the first display image to be displayed on the first display panel based on the first image data corrected based on the first tilt correction value, The image processing system according to (1), wherein the second display image is displayed on the second display panel based on the second image data corrected based on the second tilt correction value, and the second display image is visually recognized by the user as a continuation of the first display image. (3) The image processing system according to (1) or (2), wherein the image data includes viewpoint information according to an attitude of the head-mounted display, and the image generation means generates the display image viewed from a viewpoint based on the viewpoint information by rendering based on the image data corrected by the tilt correction means.(4) The image processing system according to any one of (1) to (3), wherein the tilt correction value is a correction value for a rotation direction around a virtual axis extending in a direction perpendicular to the display surface of the display panel, a rotation direction around a virtual axis extending in the up-down direction of the display panel, and a rotation direction around a virtual axis extending in the left-right direction of the display panel. (5) The image processing system according to (2), wherein the display surface of the second display panel is disposed at an angle with respect to the display surface of the first display panel. (6) The image processing system according to (2) or (5), comprising: a reference position acquisition means for acquiring a reference position based on a captured image generated by capturing an image of the display panel by an imaging means; and a tilt correction value acquisition means for acquiring the tilt correction value based on an error of the reference position with respect to a given position. (7) The image processing system according to (6), wherein the first lens unit and the second lens unit have common optical performance, and the tilt correction value acquisition means acquires the second tilt correction value calculated based on the first tilt correction value. (8) An image processing system according to (6) or (7), comprising: a magnification correction means for correcting the display magnification of the display image using a magnification correction value obtained based on an error of the reference position relative to a given position; a distortion correction means for correcting the distortion of the display image using a distortion correction value obtained based on an error of the reference position relative to the given position; and a display control means for displaying the display image corrected based on the magnification correction value and the distortion correction value on the display panel.

[0104] 1 head-mounted display, 10 display panel, 20 lens unit, 62 tilt correction unit, 63 image generation unit, S image processing system.

Claims

1. An image processing system that generates a display image that is displayed on a display panel provided on a head mounted display worn on a user's head and that is enlarged and viewed through a lens unit provided corresponding to the display panel, comprising: a tilt correction means that corrects image data input to the display panel based on a tilt correction value related to the display image in three-dimensional space; and an image generation means that generates the display image based on the image data corrected based on the tilt correction value.

2. The head mounted display is provided with a plurality of the display panels and a plurality of the lens units, the plurality of display panels including at least a first display panel that displays a first display image, and a second display panel that is provided adjacent to the first display panel and displays a second display image, the plurality of lens units including at least a first lens unit provided corresponding to the first display panel, and a second lens unit provided corresponding to the second display panel, the tilt correction means corrects first image data input to the first display panel based on a first tilt correction value related to the first display image in three-dimensional space, and corrects second image data input to the second display panel based on a second tilt correction value related to the second display image in three-dimensional space, and the image generation means generates the first display image to be displayed on the first display panel based on the first image data corrected based on the first tilt correction value, The image processing system according to claim 1 , further comprising: a display unit configured to display a second display image on the second display panel based on the second image data corrected based on the second tilt correction value, the second display image being displayed on the second display panel based on the second image data corrected based on the second tilt correction value, and the second display image being viewed by the user in succession with the first display image.

3. The image processing system according to claim 1 or 2, wherein the image data includes viewpoint information corresponding to the attitude of the head-mounted display, and the image generation means generates the display image viewed from a viewpoint based on the viewpoint information by rendering based on the image data corrected by the tilt correction means.

4. The image processing system according to claim 1 or 2, wherein the tilt correction value is a correction value for a rotation direction about a virtual axis extending in a direction perpendicular to the display surface of the display panel, a rotation direction about a virtual axis extending in the vertical direction of the display panel, and a rotation direction about a virtual axis extending in the horizontal direction of the display panel.

5. The image processing system according to claim 2, wherein the display surface of the second display panel is disposed at an angle with respect to the display surface of the first display panel.

6. The image processing system according to claim 2, comprising: a reference position acquisition means for acquiring a reference position based on an image generated by imaging the display panel with an imaging means; and a tilt correction value acquisition means for acquiring the tilt correction value based on an error of the reference position with respect to a given position.

7. The image processing system according to claim 6, wherein the first lens unit and the second lens unit have common optical performance, and the tilt correction value acquisition means acquires the second tilt correction value calculated based on the first tilt correction value.

8. The image processing system of claim 6, comprising: a magnification correction means for correcting the display magnification of the display image using a magnification correction value obtained based on an error of the reference position relative to a given position; a distortion correction means for correcting the distortion of the display image using a distortion correction value obtained based on an error of the reference position relative to a given position; and a display control means for causing the display panel to display the display image corrected based on the magnification correction value and the distortion correction value.

9. An image processing method for generating a display image that is displayed on a display panel provided in a head mounted display worn on a user's head and that is enlarged and viewed through a lens section provided corresponding to the display panel, the image processing method comprising: a step of correcting image data input to the display panel based on a tilt correction value related to the display image in three-dimensional space; and a step of generating the display image based on the image data corrected based on the tilt correction value.

10. A program that causes a computer to generate a display image that is displayed on a display panel provided on a head-mounted display worn on a user's head and that is enlarged and viewed through a lens portion provided corresponding to the display panel, the program causing the computer to execute the steps of: correcting image data input to the display panel based on a tilt correction value related to the display image in three-dimensional space; and generating the display image based on the image data corrected based on the tilt correction value.

Images