Image processing device, head-mounted display, image processing method, and program

Abstract

To allow users using an HMD to easily locate the desired object. [Solution] The image processing apparatus includes an acquisition means for acquiring a wide-angle image, which is an image with a field of view wider than a predetermined field of view and in which objects are distorted; a generation means for generating a combined image by combining cropped images with reduced distortion, corresponding to each region included in the wide-angle image; and a display control means for displaying the combined image on the head-mounted display so that it is included in the field of view displayed by the head-mounted display.

Description

Technical Field

[0001] The present disclosure relates to processing of images to be displayed on a head-mounted display apparatus.

Background Art

[0002] There is a head-mounted display apparatus (also referred to as a head mounted display or HMD) that allows a user to enjoy highly immersive images. Due to its structure, the HMD has a small viewing angle that indicates the display range. For this reason, in order to search for an object existing outside the field of view, the user needs to perform an action such as turning their head while wearing the HMD to move the display range. Accordingly, Patent Document 1 describes displaying an image with a wider angle of view than normal within the viewing angle of the HMD.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problem to be Solved by the Invention

[0004] In Patent Document 1, an equirectangular image is displayed on the HMD as an image with a wider angle of view than a normal angle of view. Unlike an image close to human vision such as a perspective projection image, an equirectangular image displays objects in a distorted manner. For this reason, when a wide-angle-of-view image with distortion such as an equirectangular image is displayed on the HMD, it may be difficult for the user to accurately recognize the shape of the object. Therefore, even if a wide-angle-of-view image with distortion such as an equirectangular image is displayed within the viewing angle of the HMD, there is a possibility that the user cannot confirm the position of a desired object.

Means for Solving the Problem

[0005] The image processing apparatus of the present disclosure is characterized by comprising: acquisition means for acquiring a wide-angle image which is an image with a field of view wider than a predetermined field of view and in which objects are distorted; generation means for generating a combined image by combining cropped images with reduced distortion corresponding to each region included in the wide-angle image; and display control means for displaying the combined image on the head-mounted display so that it is included in the field of view displayed by the head-mounted display. [Effects of the Invention]

[0006] According to this disclosure, users using an HMD can easily determine the location of a desired object. [Brief explanation of the drawing]

[0007] [Figure 1] A diagram to explain the overview of HMDs. [Figure 2] A diagram illustrating an example of the hardware configuration of an image processing device. [Figure 3] A diagram illustrating equirectangular images and cropped images. [Figure 4] A diagram to explain rendering. [Figure 5] A diagram illustrating the position and orientation of a virtual camera and the image captured by the virtual camera. [Figure 6] A diagram to explain combined images. [Figure 7] A block diagram illustrating the functional configuration included in an image processing device. [Figure 8] A flowchart illustrating the process of displaying combined images. [Figure 9] A diagram representing a cropped and combined image. [Figure 10] A block diagram illustrating the functional configuration included in an image processing device. [Figure 11] A flowchart illustrating the process of displaying combined images. [Figure 12] A diagram showing the central position and the combined image from which the image was cut. [Figure 13] A flowchart for explaining the flow of display processing of a combined image. [Figure 14] A block diagram for explaining the functional configuration included in an image processing apparatus. [Figure 15] A flowchart for explaining the flow of display processing of a combined image. [Figure 16] A diagram showing a clipped combined image. [Figure 17] A block diagram for explaining the functional configuration included in an image processing apparatus. [Figure 18] A flowchart for explaining the flow of display processing of a combined image. [Figure 19] A diagram showing an example of a screen for specifying an enlarged region. [Figure 20] A diagram showing an example of an enlarged region that has been enlarged. [Mode for Carrying Out the Invention]

[0008] Hereinafter, embodiments of the technology of the present disclosure will be described with reference to the drawings. The following embodiments do not limit the technology of the present disclosure, and not all combinations of features described in the following embodiments are necessarily essential for solving the problems of the technology of the present disclosure. The same configurations are denoted by the same reference numerals in the description.

[0009] <Embodiment 1> In the present embodiment, a method for displaying a distortion-free, wide viewing angle image on an HMD from a distorted wide viewing angle image will be described.

[0010] In the following embodiments, the combined image refers to an image with a wide viewing angle and reduced distortion, generated from a distorted wide viewing angle image. Images such as wide viewing angle images, equirectangular images, clipped images, and combined images refer to both moving images and still images, unless otherwise specified.

[0011] [Hardware Configuration] Figure 1 is a diagram for describing the HMD of the present embodiment. The HMD 101 includes a strap 102 to be worn on the head of a user who is a viewer of images (videos), a strap length adjustment unit 103, an eyepiece lens 104 for the left eye, an eyepiece lens 105 for the right eye, a display 106 for the left eye, and a display 107 for the right eye. When using the HMD 101, the user adjusts the length of the strap 102 via the strap length adjustment unit 103 to mount the HMD 101 on the user's head.

[0012] The images input to the HMD 101 are a left image to be displayed on the display 106 for the left eye, and a right image to be displayed on the display 107 for the right eye. The user of the HMD 101 views the left image with the left eye through the eyepiece lens 104 for the left eye, and views the right image with the right eye through the eyepiece lens 105 for the right eye. Note that the display 106 for the left eye and the display 107 for the right eye may be separate independent displays on the left and right sides, or may be a single display that displays the left image and the right image in a divided manner. The images displayed on the display 106 for the left eye and the display 107 for the right eye are included in the user's field of view.

[0013] Figure 2 is a diagram illustrating an example of the hardware configuration of the image processing apparatus 201 included in the HMD 101. The image processing apparatus 201 includes a CPU 202, a RAM 203, a ROM 204, a VC (video card) 205, a SATA (Serial ATA) I / F 206, a general-purpose I / F 207, a NIC (network interface card) 208, and a system bus 209.

[0014] The CPU 202 uses the RAM 203 as a work memory to execute an OS (Operating System) or various programs stored in the ROM 204, an HDD (Hard Disk Drive) 211, and the like. Further, the CPU 202 controls each component via the system bus 209.

[0015] A display device 210 is connected to the VC205. The display device 210 is, for example, a monitor for mirroring the images displayed on the left-eye display 106 or the right-eye display 107 of the HMD 101.

[0016] The general-purpose I / F 207 is connected to input devices 213 such as a mouse and keyboard, and a camera 214 via the system bus 209. The SATA I / F 206 is connected to a general-purpose drive 212 that reads and writes to the HDD 211 and various recording media via the system bus 209. The NIC 208 handles the input and output of information to and from external devices. The CPU 202 uses the various recording media mounted on the HDD 211 and the general-purpose drive 212 as storage locations for various data. The CPU 202 displays a GUI (Graphical User Interface) provided by a program on the display device 210 and receives input such as user instructions via the input device 213.

[0017] [Regarding distorted wide-angle images] Figure 3 illustrates the difference between a distorted wide-angle image (hereinafter simply referred to as a wide-angle image) and a cropped image that is normally displayed on the left-eye display 106 or the right-eye display 107 of the HMD 101.

[0018] Figure 3(a) shows an example of a shooting environment for obtaining a wide-angle image to be input to the HMD101. As shown in Figure 3(a), a shooting device such as a 360° spherical camera with a horizontal field of view of 360° and a vertical field of view of 360° is used to take a wide-angle image to be input to the HMD101, near the center of a rectangular room.

[0019] Figure 3(b) shows an equirectangular image, which is an example of a wide-angle image input to the HMD101. An equirectangular image is an image (called an equirectangular image) that represents a wide-angle image, such as one with a horizontal field of view of 360° and a vertical field of view of 360°, using equirectangular projection. Because an equirectangular image is represented so that the number of pixels per angle is uniform, objects included in the equirectangular image are distorted when compared to actual objects because their angles and proportions change. For example, the edges of the desk 301 are actually straight lines, but as shown in Figure 3(b), the edges of the desk 301 included in the equirectangular image are distorted. Thus, in this embodiment, distortion refers to distortion that can be confirmed when compared to an object that a person actually sees, and refers to a state where there is distortion compared to when it is represented by perspective projection, for example, a state in which something that is originally a straight line is represented as curved. Wide field of view refers to a field of view wider than a predetermined field of view. The predetermined field of view is, for example, the field of view represented by the cropped image normally displayed on the HMD101 (see Figure 3(c)).

[0020] Figure 3(c) shows an example of a cropped image. The image processing device 201 performs rendering to extract a region of the equirectangular image corresponding to the display field of view area corresponding to the orientation of the HMD 101 and convert it to a perspective projection. As a result of this rendering, an image like Figure 3(c) is obtained, which has a smaller field of view than the equirectangular image but corrects the distortion of objects. In this embodiment, this image is called a "cropped image". Figure 3(c) is an example of a cropped image that is displayed on the left eye display 106 or the right eye display 107 when the wide-angle image of Figure 3(b) is input to the HMD 101. The cropped image normally displayed on the HMD 101 is a portion of the wide-angle image corresponding to the orientation of the HMD 101. In other words, the cropped image is a portion of the wide-angle image corresponding to the orientation of the user wearing the HMD 101. Therefore, the user wearing the HMD 101 can change the displayed area of ​​the wide-angle image by changing the orientation of their own face. However, users wearing the HMD101 will view cropped images with a small field of view, as shown in Figure 3(c). Therefore, for example, the window 302 that was included in the wide-angle image in Figure 3(b) will be located outside the field of view in the cropped image in Figure 3(c). As a result, users will not be able to recognize the presence of the window 302 unless they change the direction of their face to change their field of view.

[0021] [rendering] Figure 4 is a diagram illustrating rendering. As shown in Figure 4(a), the distorted wide-angle image from Figure 3(b) is mapped and pasted onto the mesh inside the roughly spherical shape. Figures 4(b) and 4(c) illustrate the position and orientation of a virtual camera placed in a three-dimensional CG space representing the inside of the roughly spherical shape shown in Figure 4(a) onto which the wide-angle image is pasted. Since the orientation in which the virtual camera captures images is along the y-axis, it is possible to observe the vertical field of view of the virtual camera in Figure 4(b) and the horizontal field of view of the virtual camera in Figure 4(c). Also, in Figures 4(b) and 4(c), scales are marked every 30 degrees from the center of the roughly spherical shape. Figure 4(b) is called the horizontal view, and Figure 4(c) is called the vertical view. The image obtained when this virtual camera captures the inside of the roughly spherical shape onto which the wide-angle image is pasted is a partial image in which the distortion of the wide-angle image has been corrected, and becomes the cropped image in Figure 3(c). This is a conceptual explanation of the rendering in this embodiment. Although rendering has been explained using a virtual camera, in reality, rendering is performed by computation.

[0022] The virtual camera L for the left eye, shown in Figures 4(b) and 4(c), is a virtual camera that captures the cropped image displayed on the left-eye display 106 of the HMD 101. The virtual camera R for the right eye is a virtual camera that captures the cropped image displayed on the right-eye display 107 of the HMD 101. The virtual camera L for the left eye and the virtual camera R for the right eye need to be at a certain distance from each other in order to create parallax in the image viewed by the user. Since the parallax is determined by the distance between the left-eye eyepiece 104 and the right-eye eyepiece 105, the distance between the virtual camera L for the left eye and the virtual camera R for the right eye is set to a distance corresponding to the distance between the left-eye eyepiece 104 and the right-eye eyepiece 105. Also, in Figures 4(b) and 4(c), the center position of the distance between the virtual camera L for the left eye and the virtual camera R for the right eye coincides with the center of a roughly spherical shape. This is just one example, and the positions of the virtual camera L for the left eye and the virtual camera R for the right eye are not limited to those shown in Figures 4(b) and 4(c), provided that subsequent processing is possible. For the sake of explanation, the processing of the virtual camera L for the left eye will be described below.

[0023] [About combined images] Figure 5(a) is a horizontal and vertical view showing the field of view of a virtual left-eye camera L in a three-dimensional CG space within a roughly spherical shape, with the equirectangular image of Figure 3(b) pasted inside. The field of view of the virtual left-eye camera L is assumed to be 30° horizontally and 30° vertically. Furthermore, the cropped image A in Figure 5(a) is a figure showing an image taken at the position and orientation of the virtual left-eye camera L in Figure 5(a).

[0024] Figure 5(b) shows the horizontal and vertical views representing the field of view of the left-eye virtual camera L in the same three-dimensional CG space, and its orientation is different from that of the left-eye virtual camera L in Figure 5(a). The cropped image B in Figure 5(a) is a diagram showing an image taken at the position and orientation of the left-eye virtual camera L in Figure 5(b).

[0025] In the horizontal views of Figures 5(a) and 5(b), region Z is a region common to both cropped image A and cropped image B. In this way, the orientation of the virtual camera may be set so that a cropped image has a region common to other cropped images. In Figure 5(a), the orientation of the virtual camera L for the left eye is positioned perpendicular to the x-axis and parallel to the y-axis.

[0026] In Figure 5(b), the orientation of the virtual camera L for the left eye is the same as that of the virtual camera L for the left eye in Figure 5(a), rotated 20° clockwise horizontally. As shown in cropped images A and B of Figure 5, the cropped images, which are images captured by the virtual camera, change depending on the orientation of the virtual camera L for the left eye. In this embodiment, cropped images that appear to have been captured as if the virtual camera had changed orientation are obtained by rendering, and these cropped images are then combined.

[0027] Figure 6 illustrates a combined image obtained by combining cropped images that were rendered and generated as if captured by a virtual camera. By combining cropped images with reduced distortion, albeit with a small field of view, a combined image with a wide field of view and no distortion can be obtained. In this embodiment, it is explained that the combined image is generated by combining cropped images so that the horizontal field of view is 360°. In this case, the setting of the vertical field of view in the combined image may be a predetermined value held by the HMD101, or a value set by the user wearing the HMD101. The combined image may also be an image with a vertical field of view of 360°. In that case, the determination of the horizontal field of view may be a predetermined value held by the HMD101, or a value set by the user wearing the HMD101.

[0028] The method for combining the cropped images may involve identifying and combining regions based on the field of view and position of the virtual camera, or it may involve known image combining techniques such as feature point matching. For example, when using known image combining techniques, a common region captured by multiple virtual cameras, such as region Z, may be required. Therefore, the size of the common region should be determined according to the load of the combining process and the combining technique applied. In this embodiment, the image processing device 201 generates a combined image that has a wider field of view than a single cropped image normally viewed by the user, and in which distortion of objects within the image is reduced.

[0029] [Functional Configuration] Figure 7 is a block diagram illustrating the functional configuration included in the image processing device 201. The image processing device 201 includes a wide-angle image acquisition unit 701, an image generation unit 702, a field of view determination unit 703, a cropping unit 704, and a display control unit 705.

[0030] The wide-angle image acquisition unit 701 acquires a distorted wide-angle image and outputs the acquired wide-angle image to the image generation unit 702. In this embodiment, the distorted wide-angle image is described as an equirectangular image, but the wide-angle image is not limited to an equirectangular image. Any wide-angle image that can be displayed on the left-eye display 106 or the right-eye display 107 is acceptable, such as an image captured from a camera mounted on the HMD 101 or an image stored on the HDD 211 of the HMD 101. For example, another example of a distorted wide-angle image acquired by the wide-angle image acquisition unit 701 is a dome master type wide-angle image. Furthermore, if the wide-angle image is a moving image, the wide-angle image acquisition unit 701 may acquire each frame that makes up the moving image, or it may acquire a specific frame from the moving image.

[0031] The image generation unit 702 generates cropped images with reduced distortion from the wide-angle image acquired from the wide-angle image acquisition unit 701, and combines the cropped images to generate a combined image that is both wide-angle and has reduced distortion. The image generation unit 702 outputs the combined image to the cropping unit 704. The image generation unit 702 defines a virtual camera and generates a combined image by combining images taken by that virtual camera.

[0032] The field of view determination unit 703 determines the field of view of the combined image displayed on the left eye display 106 and the right eye display 107 of the HMD 101. The field of view determination unit 703 outputs the determined field of view to the cropping unit 704. If the combined image generated by the image generation unit 702 has a horizontal field of view of 360°, the field of view determination unit 703 determines the horizontal field of view. The field of view determination unit 703 determines the field of view from a value that is greater than or equal to the horizontal display field of view of the HMD 101 and less than or equal to 360 degrees. The field of view determination unit 703 may determine the field of view from a value predetermined within the HMD 101, or it may determine the field of view from a value set by the user wearing the HMD 101 from a field of view greater than or equal to the display field of view. The field of view determination unit 703 may also determine the vertical field of view.

[0033] The cropping unit 704 crops the combined image generated by the image generation unit 702 so that it matches the field of view determined by the field of view determination unit 703.

[0034] The display control unit 705 displays the cropped and combined image so that it is included in the display field of view of the HMD 101. The field of view in an HMD refers to the range that the user wearing the HMD can see without moving their head. In this embodiment, the display control unit 705 displays the cropped and combined image on the left eye display 106 and the right eye display 107.

[0035] Each functional unit shown in Figure 7 is realized by the CPU of the image processing device 201 executing a predetermined program, but is not limited to this. Other hardware, such as a GPU (Graphics Processing Unit) or FPGA (Field Programmable Gate Array), may also be used to speed up calculations. Each functional unit may be realized through the cooperation of software and hardware such as a dedicated IC, or some or all of the functions may be realized by hardware alone.

[0036] [flowchart] Figure 8 is a flowchart illustrating the flow of the combined image display process in this embodiment. Each step shown in Figure 8 is realized by loading a program stored in the ROM 212 of the image processing device 201 into the RAM 213 and executing it by the CPU 211. Alternatively, some or all of the functions of the steps in Figure 8 may be realized by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in the flowchart.

[0037] In normal mode, the HMD101 displays a cropped portion of a wide-angle image corresponding to the orientation of the HMD101. When the user wearing the HMD101 performs an operation to widen the field of view of the image, the flowchart shown in Figure 8 is initiated.

[0038] In S801, the wide-angle image acquisition unit 701 acquires an equirectangular image as a wide-angle image to be displayed on the left-eye display 106 or the right-eye display 107 of the HMD 101. An equirectangular image is an example of a distorted wide-angle image, and the wide-angle image acquisition unit 701 may acquire other wide-angle images.

[0039] In step S802, the image generation unit 702 generates distorted cropped images corresponding to each region of the equirectangular image acquired from the wide-angle image acquisition unit 701, and combines the cropped images to generate a combined image. The image generation unit 702 outputs the combined image to the display control unit 705.

[0040] In S803, the field of view determination unit 703 determines the field of view of the combined image to be displayed by the display control unit 705 based on the combined image generated by the image generation unit 702.

[0041] In S804, the cropping unit 704 crops the combined image generated in S802 so that it becomes an image with the field of view determined by the field of view determination unit 703 in S803. In this embodiment, the generated combined image is cropped so that the position of the center of the equirectangular image, which is a wide-angle image acquired by the wide-angle image acquisition unit 701 in S801, becomes the center of the cropped image.

[0042] In S805, the display control unit 705 displays the combined image cropped in S804 on the left-eye display 106 and the right-eye display 107 so that they are included in the display field of view of the HMD 101.

[0043] Figure 9 shows an example of a combined image obtained by cropping a combined image from Figure 6, which has a horizontal field of view of 360°, so that the horizontal field of view becomes 270°. In S805, the display control unit 705 displays the cropped combined image on the left-eye display 106 and the right-eye display 107 so that it fits within the display field of view of the HMD 101. At that time, the display control unit 705 displays the cropped combined image so that the aspect ratio of the combined image does not change. For example, if the cropped combined image is a widescreen image, regions 901 and 902 are added to the top and bottom of the image to increase the vertical length of the image, as shown in Figure 9. In this way, the aspect ratio of the cropped combined image itself is maintained so that it fits within the display field of view of the HMD 101.

[0044] As a result of the flowchart in Figure 8, the screen displayed on the left eye display 106 or the right eye display 107 of the HMD 101 switches from the cropped image in Figure 3(c) to the combined image in Figure 9. The combined image in Figure 9 has a wider field of view than Figure 3(c) and is less distorted than the equirectangular image shown in Figure 3(b), which has the same horizontal field of view of 360°. Since the cropped combined image shown in Figure 9 is displayed within the display field of view of the HMD 101, the combined image is displayed so that it is included in the field of view of the user wearing the HMD 101. Therefore, the user can view a wide-angle image with reduced distortion without moving the direction of their head.

[0045] If the wide-angle image input to the HMD101 is a moving image, in S801 the wide-angle image acquisition unit 701 acquires the frame to be processed from among the frames constituting the moving image as a wide-angle image. Then, in S805, the display control unit 705 displays the combined image corresponding to the frame to be processed within the display field of view of the HMD101. If there is a next frame, the process returns to S801, and the flowchart in Figure 8 is repeated with the next frame as the frame to be processed. As a result, the user wearing the HMD101 can view a wide-angle combined image with reduced distortion as a moving image within their field of view.

[0046] Furthermore, if the combined image generated in S802 has a horizontal field of view of 360°, as shown in Figure 6, it will become a widescreen image, and even if it is displayed to fit within the user's field of view, it may be difficult for the user to view. On the other hand, if the field of view of the combined image generated in S802 is as shown in Figure 9, the cropping process in S804 may not be necessary. Therefore, depending on the field of view of the combined image generated in S802, it may be determined whether or not to execute the cropping process in S804, and the cropping process in S804 may be executed only if it is determined that the cropping process should be executed. For example, before S804, the cropping unit 704 may determine whether the combined image generated in S802 has a field of view greater than or equal to a threshold, and only proceed to S804 if it is determined that the field of view of the combined image is greater than or equal to a threshold. In this case, in S804, the cropping unit 704 should crop the combined image so that the horizontal field of view is less than the threshold. If the field of view of the combined image is determined to be less than a threshold, the display control unit 705 may display the combined image generated in S802 directly within the field of view of the HMD 101.

[0047] In this embodiment, an image with a wider field of view than the cropped image normally viewed by the user, and with reduced object distortion, can be generated from a wide-angle image and displayed within the display field of view of the HMD101. Therefore, the user wearing the HMD101 can confirm the location of the object they are looking for without changing the orientation of their own face. As a result, the object they are looking for can be found more easily.

[0048] Furthermore, if the user performs an operation to disable the display of the wide-angle image, the screen displayed on the left-eye display 106 or the right-eye display 107 of the HMD 101 switches from the combined image in Figure 9 to a cropped image corresponding to the user's orientation, as shown in Figure 3(c).

[0049] Furthermore, it is conceivable that users may also want to view wide-angle images with a vertical field of view of 360° and a horizontal field of view of 360°, such as the equirectangular image in Figure 3(b). For this reason, the CPU 202 may control the display so that the user can switch between displaying a distorted wide-angle image, such as the equirectangular image in Figure 3(b), and a distorted combined image, such as the one in Figure 9. When displaying a distorted wide-angle image, such as the equirectangular image, the display control unit 705 should display the equirectangular image in Figure 3(b) on the left-eye display 106 and the right-eye display 107 so that it fits within the display field of view of the HMD 101.

[0050] <Embodiment 2> Embodiment 1 described a method for displaying a wide-angle, distortion-free combined image from a distorted wide-angle image within the display field of view of the HMD 101. However, the display field of view of the HMD 101 may be larger than the effective field of view in which humans can identify information, and even if the combined image is displayed to fit within the display field of view of the HMD 101, the edges of the combined image may be difficult for the user to see. Therefore, this embodiment describes a method in which the user specifies the position of the center of the combined image so that the area the user wants to see is included in the center of the user's field of view. This embodiment will be described mainly in terms of the differences from Embodiment 1. Unless otherwise specified, the configuration and processing are the same as in Embodiment 1.

[0051] Figure 10 is a block diagram showing the functional configuration of the image processing apparatus 201 in this embodiment. In Figure 10, the wide-angle image acquisition unit 701, the image generation unit 702, the field of view determination unit 703, and the display control unit 705 perform the same processing as in Embodiment 1, so their descriptions are omitted.

[0052] The center designation unit 1001 acquires the position to be the center of the image, as specified by the viewer, who is the user wearing the HMD 101, and outputs the acquired position information to the cropping unit 704.

[0053] Figure 11 is a flowchart illustrating the flow of the combined image display process in this embodiment. The processes S801, S802, and S803 are the same as in Embodiment 1, so their explanation is omitted. In this embodiment, after the process in S803 is completed, the process proceeds to S1101.

[0054] In S1101, the center designation unit 1001 obtains the position to be the center of the combined image to be displayed by the display control unit 705 from the information of the user wearing the HMD 101 and the HMD 101.

[0055] Figure 12(a) shows an example of a cropped image for the user to specify a position. The center selection unit 1001 may display cropped images corresponding to the orientation of the HMD 101 on the left eye display 106 or the right eye display 107 of the HMD 101, and allow the user to specify the center position from among the displayed cropped images. The star mark 1201 in Figure 12(a) indicates the position specified by the user. Alternatively, the center selection unit 1001 may display a wide-angle image like the one in Figure 3(b) on the left eye display 106 or the right eye display 107 of the HMD 101, and allow the user to specify the center position from among the wide-angle images.

[0056] The user can, for example, operate a controller included in the HMD101 to specify a central position from either a cropped image or a wide-angle image. Alternatively, if the HMD101 is equipped with a camera for identifying the user's hand gestures, the user may be asked to specify the central position by making a hand gesture. For example, if the CPU202 identifies the user's first hand gesture, it moves the star marker 1201, and if the user's second hand gesture is identified, it determines the current position of the star marker 1201 as the central position. Alternatively, if the HMD101 is equipped with an eye camera for detecting the user's line of sight, the user may be asked to specify a position according to their line of sight. For example, the CPU202 identifies the position the user is looking at and moves the star marker 1201 to the identified position. If the CPU202 determines that the user has been gazing at the star marker for a predetermined time, it determines the current position of the star marker 1201 as the central position. Alternatively, if the confirm button on the controller included in the HMD101 is pressed, the CPU202 may determine the current position of the star marker 1201 as the center position.

[0057] In S1102, the cropping unit 704 crops the combined image generated by the image generation unit 702 in S802 in Figure 11 so that the position acquired by the center designation unit 1001 becomes the center, and the field of view is determined by the field of view determination unit 703.

[0058] In S1103, the display control unit 705 displays the combined image cropped in S1102 on the left-eye display 106 and the right-eye display 107 so that they are included in the display field of view of the HMD 101. Similar to Embodiment 1, the display control unit 705 displays the cropped combined image so that the aspect ratio of the combined image does not change. Figure 12(b) is a diagram showing an example of a combined image cropped around the position of the star mark 1201 shown in Figure 12(a).

[0059] In addition, in S1102, it may be necessary to crop the image so that the edge of the combined image generated in S802 in Figure 11 becomes the center of the cropped combined image. For this reason, for example, in S1102, the cropping unit 704 duplicates the combined image and combines the two combined images, as shown in Figure 12(c). The cropping unit 704 may then crop the combined image to a field of view determined by the field of view determination unit 703, with the position of the star mark 1202 specified by the user as the center. As a result, the combined image shown in Figure 12(d) can be displayed within the display field of view of the HMD 101.

[0060] Figure 13 illustrates another example of how a user-specified center position can be used. In Figure 13, in S1301, the center specification unit 1001 specifies the center position before the combined image is generated in S802. Then, in the combined image generation in S802, the combined image is generated so that the user-specified position becomes the center. For example, if a user wants to display a combined image with the floor as the center to find an object on the floor, the user wearing the HMD 101 may turn their face downwards, display the floor on the HMD 101's display, and then specify the floor position. In this case, in S1301 of Figure 13, the center specification unit 1001 specifies the floor position as the center position. Then, in S802, a wide-angle image with reduced distortion may be generated with the floor as the center.

[0061] As described above, according to this embodiment, the user can specify the position to be the center of the combined image. Therefore, it is possible to display the combined image to the user so that the area the user wants to focus on is at the center.

[0062] <Embodiment 3> When the combined image displayed on the HMD101 is a video, the object the user wants to see may move. Therefore, even if a wide-angle combined image is displayed, the user may have difficulty appreciating the object because they have to follow it with their eyes to keep it in view. In this embodiment, a method for displaying the combined image so that a specified object is at the center in each frame is described. This embodiment will be described mainly for the differences from Embodiment 1. Unless otherwise specified, the configuration and processing are the same as in Embodiment 1.

[0063] Figure 14 is a block diagram showing the functional configuration of the image processing apparatus 201 in this embodiment. The wide-angle image acquisition unit 701, image generation unit 702, field of view determination unit 703, cropping unit 704, and display control unit 705 perform the same processing as in Embodiment 1, so their descriptions are omitted.

[0064] The object detection unit 1401 detects the object to be displayed in the center from the combined image. For example, the object detection unit 1401 detects the object to be displayed in the center that is determined to be the most prominent object in the splendor detection.

[0065] The method by which the object detection unit 1401 detects the object to be displayed in the center is not limited. For example, the object detection unit 1401 may detect an object specified by the creator of the wide-angle image input to the HMD 101 as the object to be displayed in the center. Alternatively, the object detection unit 1401 may detect a moving object as the object to be displayed in the center. Or, it may detect a specific object specified by the user wearing the HMD 101 as the object to be displayed in the center. For example, if the user specifies a mode to detect people, the object detection unit 1401 will detect people from within the image and detect them as the object to be displayed in the center. Alternatively, if the user specifies a specific object from within the screen, the object detection unit 1401 will track the specified object and detect the specified object for each frame as the object to be displayed in the center. The object detection unit 1401 outputs the object detection results to the cropping unit 704.

[0066] Figure 15 is a flowchart illustrating the flow of the combined image display process in this embodiment. The processes S801, S802, and S803 are the same as in Embodiment 1, so their explanation is omitted. In this embodiment, after the process in S803 is completed, the process proceeds to S1501. The flowchart in Figure 15 is a diagram illustrating the display process of a combined image for one frame. When the wide-angle image is a video, as in this embodiment, once the display process of one frame of the combined image is completed, the flowchart in Figure 15 is performed again to perform the display process for the next frame, and the flowchart in Figure 15 is executed repeatedly.

[0067] In S1501, the object detection unit 1401 detects the object to be displayed in the center from the combined image generated in S802 in Figure 15.

[0068] In S1502, the cropping unit 704 crops the combined image generated in S802 so that the field of view is determined by the field of view determination unit 703, with the position of the object detected in S1501 as the center.

[0069] In S1503, the display control unit 705 displays the combined image cropped in S1502 on the left-eye display 106 and the right-eye display 107 so that they are included in the display field of view of the HMD 101. Similar to Embodiment 1, the display control unit 705 displays the cropped combined image so that the aspect ratio of the combined image does not change. Since the flowchart in Figure 15 is executed for each frame, the combined image generated in S1503 is displayed in frame order, so that an image centered on the object is displayed as a video on the HMD 101.

[0070] Figure 16 shows an example of an image that the display control unit 705 displays on the left eye display 106 or the right eye display 107 of the HMD 101. For example, in S801 in Figure 15, the wide-angle image acquisition unit 701 acquires each frame of the wide-angle video containing the running person 1601 as the wide-angle image to be processed. Then, in S1501, the object detection unit 1401 detects the person 1601 from the combined image generated from the frames to be processed, and in S1502, the combined image is cropped around the position of the person 1601 in each frame.

[0071] Figures 16(a) to (c) are examples of combined images of three frames within a certain period. In S1503, the combined image corresponding to the frame to be processed is displayed, so the combined image is displayed as a video. Figures 16(a) and 16(b) are combined images cropped so that the running person 1601 is at the center. Figure 16(c) is a combined image cropped so that the person 1601 is about to enter a building is at the center. Even when each frame of a wide-angle video in which person 1601 is moving is acquired in this way, the left eye display 106 and the right eye display 107 of the HMD 101 will display a combined image, which is a video with person 1601 at the center.

[0072] As described above, according to this embodiment, the HMD101 can display a wide-angle image with reduced object distortion, in which the desired object's position is centered. Therefore, the user wearing the HMD101 can capture changes in the desired object.

[0073] <Embodiment 4> This embodiment describes a method for easily searching for even small objects by enlarging the area that the viewer (user) wants to focus on in the combined image. This embodiment will be described mainly in terms of the differences from Embodiment 1. Unless otherwise specified, the configuration and processing are the same as in Embodiment 1.

[0074] Figure 17 is a block diagram showing the functional configuration of the image processing apparatus 201 in this embodiment. The wide-angle image acquisition unit 701, image generation unit 702, field of view determination unit 703, cropping unit 704, and display control unit 705 perform the same processing as in Embodiment 1, so their descriptions are omitted.

[0075] The enlarged area identification unit 1701 identifies the area that the user has specified as the area they want to enlarge and display in the combined image displayed by the display control unit 705. The enlarged area identification unit 1701 outputs information indicating the identified area to the display control unit 705.

[0076] Figure 18 is a flowchart illustrating the processing flow in this embodiment. The processing from S801 to S805 is the same as in Embodiment 1, so its explanation is omitted. In this embodiment, once the processing in S805 is completed, the process proceeds to S1801.

[0077] In S1801, the enlarged area identification unit 1701 identifies the enlarged area, which is the area to be enlarged, from the combined image displayed by the display control unit 705 in S805 in Figure 18.

[0078] Figure 19 shows an example of a user interface (hereinafter referred to as "UI") screen for the user to specify the magnified area. The UI screen 1900 in Figure 19 is the screen displayed on the left eye display 106 or the right eye display 107 of the HMD 101. The UI screen 1900 in Figure 19 is displayed when the user performs a predetermined operation after S805 in Figure 18.

[0079] The combined image 1910 included in the UI screen 1900 may be the combined image before cropping generated in S802 in Figure 18, or the combined image after cropping in S804 in Figure 18. The user specifies the position of the enlarged area via the UI screen 1900. For example, as shown in Figure 19(a), the user moves the position of the rectangular frame 1901 corresponding to the enlarged area to specify the enlarged area from among the combined images 1910 included in the UI screen 1900. Alternatively, as shown in Figure 19(b), the user moves the circular pointer 1902 to specify the position of the center of the enlarged area from among the combined images 1910 included in the UI screen 1900.

[0080] The user can specify the position of the magnified area, for example, by operating a controller included in the HMD101. Alternatively, if the HMD101 is equipped with a camera for identifying the user's hand gestures, the position of the magnified area may be specified according to the user's hand gestures. For example, if the CPU202 identifies the user's first hand gesture, it moves the frame 1901 or the pointer 1902.

[0081] Alternatively, if a second hand gesture of the user is identified, the CPU 202 determines the current position of frame 1901 or pointer 1902 as the position of the magnified area. Alternatively, if the HMD 101 is equipped with an eye camera that detects the user's gaze position, the position of the magnified area may be specified according to the user's gaze. For example, the CPU 202 identifies the position of the UI screen 1900 that the user is looking at and moves frame 1901 or pointer 1902 to the identified position. If the CPU 202 determines that the user has been gazing at it for a predetermined time, it determines the current position of frame 1901 or pointer 1902 as the position of the magnified area. Alternatively, if a button determined from the controller included in the HMD 101 is pressed, the CPU 202 may determine the current position of frame 1901 or pointer 1902 as the position of the magnified area. Note that the icon used by the user to specify the position of the magnified area is not limited to frame 1901 or pointer 1902.

[0082] The size of the magnified area may be specified by the user, or it may be a predetermined size available in the HMD101.

[0083] In S1802, the display control unit 705 processes the image of the magnified area specified by the magnified area identification unit 1701 in S1801 to be displayed on the left eye display 106 or the right eye display 107 of the HMD 101.

[0084] For example, suppose a user wants to enlarge and display a mountain 1911 included in the combined image 1910. In this case, the user specifies the location of the mountain 1911 as the enlargement area, as shown in Figure 19. Then, in S1801, the enlargement area identification unit 1701 identifies the area in the combined image 1910 that includes the location of the mountain 1911 as the enlargement area.

[0085] Figure 20 shows an example of the screens displayed on the left-eye display 106 and the right-eye display 107 as a result of processing in S1802, showing a screen with the user-specified mountain 1911 area magnified. As shown in Figure 20, the display control unit 705 displays the screen so that it is included in the display field of view of the HMD 101. The display control unit 705 maintains the aspect ratio when displaying the magnified area image within the display field of view.

[0086] Furthermore, the display control unit 705 can, by having the user perform a predetermined operation, switch back from the image in Figure 20 to the image in Figure 19(a), which is the image to be displayed within the viewing angle of the HMD 101.

[0087] As described above, according to this embodiment, a user-specified area from the combined image can be enlarged and displayed to the user. Therefore, even if the object is small, the user can easily find the object.

[0088] <Other Embodiments> In the embodiment described above, the image processing device 201 was described as being included in the HMD 101, but this configuration is just one example and other configurations are possible. For example, the image processing device 201 may be an external device connected to the HMD 101 by a network or the like. In this case, the process by the display control unit 705 to display an image on the display of the HMD 101 refers to transmitting the image data to be displayed to the HMD 101 and instructing the HMD 101 to display the image.

[0089] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0090] The above-described embodiments include the following configurations.

[0091] (Composition 1) An acquisition means for acquiring a wide-angle image, which is an image with a field of view wider than a predetermined field of view and in which objects are distorted; A generation means for generating a combined image by combining the distortion-reduced cropped images corresponding to each region included in the wide-angle image, Display control means for displaying the combined image on the head-mounted display so that it is included in the field of view displayed by the head-mounted display, An image processing apparatus characterized by having (Configuration 2) The wide-angle image is an equirectangular image. The image processing apparatus according to configuration 1, characterized in that... (Composition 3) The aforementioned wide-angle image is an image with a horizontal field of view of 360 degrees. An image processing apparatus according to configuration 1 or 2, characterized by the above. (Composition 4) The generation means further includes a cropping means for cropping a predetermined region from the combined image generated by the generation means, The display control means is The combined image cut out by the cutting means is displayed. An image processing apparatus according to any one of configurations 1 to 3. (Composition 5) The cropping means crops the combined image so that the horizontal field of view becomes a predetermined field of view. The image processing apparatus according to configuration 4, characterized in that... (Composition 6) The cutting means is, If the combined image generated by the generation means has a horizontal field of view greater than or equal to a threshold, the combined image is cropped so that the horizontal field of view is less than the threshold. The image processing apparatus according to configuration 4, characterized in that... (Composition 7) It further includes a second acquisition means for obtaining a location specified by the user, The cropping means crops the combined image generated by the generating means such that the center of the cropped combined image is at the designated position. An image processing apparatus according to any one of configurations 4 to 6, characterized by the above. (Composition 8) The system further includes a third acquisition means for acquiring a position specified by the user among the combined images, The generation means generates the combined image such that the specified position is at the center. An image processing apparatus according to any one of configurations 1 to 7. (Composition 9) The second acquisition means described above is: Operation by a controller connected to the head-mounted display, The user's hand gestures detected by the camera included in the head-mounted display, or Based on the gaze of the user wearing the head-mounted display, the position specified by the user is acquired. The image processing apparatus according to configuration 7, characterized by the features described above. (Composition 10) The system further includes detection means for detecting a predetermined object included in the combined image, The display control means displays the combined image so that the predetermined object is at the center. An image processing apparatus according to any one of configurations 1 to 9, characterized in that (Composition 11) The system further includes a means for identifying a region specified by the user from among the displayed combined images, The display control means is The identified area is enlarged and displayed on the head-mounted display. An image processing apparatus according to any one of configurations 1 to 10, characterized by the above. (Composition 12) The generating means is The wide-angle image is pasted onto the inside of the spherical shape, and the cropped image corresponding to the region is generated so that the image is taken by a virtual camera from within the spherical shape. An image processing apparatus according to any one of configurations 1 to 11, characterized by the features described herein. (Composition 13) The display control means has a first mode in which it displays, among the cropped images corresponding to each region included in the wide-angle image, a cropped image of a predetermined field of view corresponding to the orientation of the head-mounted display, within the field of view of the head-mounted display, The display control means further includes a switching means for switching between a second mode in which the combined image is displayed within the field of view of the head-mounted display. An image processing apparatus according to any one of configurations 1 to 12, characterized by the above. (Composition 14) The aforementioned switching means is The display control means can further switch to a third mode in which the wide-angle image is displayed within the field of view of the head-mounted display. The image processing apparatus according to configuration 13, characterized by the above. (Composition 15) A head-mounted display including an image processing device as described in any one of Configurations 1 to 14. (Composition 16) An acquisition step to obtain a wide-angle image, which is an image with a field of view wider than a predetermined field of view and in which the object is distorted; A generation step of generating a combined image by combining the distortion-reduced cropped images corresponding to each region included in the wide-angle image, A display control step of displaying the combined image on the head-mounted display so that it is included in the field of view displayed by the head-mounted display, An image processing method characterized by having the following features. (Composition 17) A program for causing a computer to execute each of the means of the image processing apparatus described in any one of configurations 1 to 14. [Explanation of Symbols]

[0092] 101 HMD 201 Image Processing Device 701 Wide-angle image acquisition unit 702 Image Generation Unit 705 Display Control Unit

Claims

1. An acquisition means for acquiring a wide-angle image, which is an image with a field of view wider than a predetermined field of view and in which objects are distorted; A generation means for generating a combined image by combining the distortion-reduced cropped images corresponding to each region included in the wide-angle image, Display control means for displaying the combined image on the head-mounted display so that it is included in the field of view displayed by the head-mounted display, An image processing apparatus characterized by having

2. The wide-angle image is an equirectangular image. The image processing apparatus according to feature 1.

3. The aforementioned wide-angle image is an image with a horizontal field of view of 360 degrees. The image processing apparatus according to feature 1.

4. The generation means further includes a cropping means for cropping a predetermined region from the combined image generated by the generation means, The display control means is The combined image cut out by the cutting means is displayed. The image processing apparatus according to feature 1.

5. The cropping means crops the combined image so that the horizontal field of view becomes a predetermined field of view. The image processing apparatus according to feature 4.

6. The cutting means is, If the combined image generated by the generation means has a horizontal field of view greater than or equal to a threshold, the combined image is cropped so that the horizontal field of view is less than the threshold. The image processing apparatus according to feature 4.

7. It further includes a second acquisition means for obtaining a location specified by the user, The cropping means crops the combined image generated by the generating means such that the center of the cropped combined image is at the designated position. The image processing apparatus according to feature 4.

8. The system further includes a third acquisition means for acquiring a position specified by the user among the combined images, The generation means generates the combined image such that the specified position is at the center. The image processing apparatus according to feature 1.

9. The second acquisition means described above is: Operation by a controller connected to the head-mounted display, The user's hand gestures detected by the camera included in the head-mounted display, or Based on the gaze of the user wearing the head-mounted display, the position specified by the user is acquired. The image processing apparatus according to feature 7.

10. The system further includes detection means for detecting a predetermined object included in the combined image, The display control means displays the combined image so that the predetermined object is at the center. The image processing apparatus according to feature 1.

11. The system further includes a means for identifying a region specified by the user from among the displayed combined images, The display control means is The identified area is enlarged and displayed on the head-mounted display. The image processing apparatus according to feature 1.

12. The generating means is The wide-angle image is pasted onto the inside of the spherical shape, and the cropped image corresponding to the region is generated so that the image is taken by a virtual camera from within the spherical shape. The image processing apparatus according to feature 1.

13. The display control means has a first mode in which it displays, within the field of view of the head-mounted display, a cropped image of a predetermined field of view corresponding to the orientation of the head-mounted display, from among the cropped images corresponding to each region included in the wide-angle image, The display control means further includes a switching means for switching between a second mode in which the combined image is displayed within the field of view of the head-mounted display. The image processing apparatus according to feature 1.

14. The aforementioned switching means is The display control means can further switch to a third mode in which the wide-angle image is displayed within the field of view of the head-mounted display. The image processing apparatus according to feature 13.

15. A head-mounted display including the image processing apparatus described in claim 1.

16. An acquisition step to obtain a wide-angle image, which is an image with a field of view wider than a predetermined field of view and in which the object is distorted; A generation step of generating a combined image by combining the distortion-reduced cropped images corresponding to each region included in the wide-angle image, A display control step of displaying the combined image on the head-mounted display so that it is included in the field of view displayed by the head-mounted display, An image processing method characterized by having the following features.

17. A program for causing a computer to execute each means of the image processing apparatus described in any one of claims 1 to 14.

Images