Image processing device, image processing method, and program

Abstract

To enable easy observation of stereo images with a natural sense of depth. [Solution] The stereo image data acquisition unit acquires a sub-stereo image to be displayed within the main stereo image, and the imaging information of that sub-stereo image. The HMD information acquisition unit acquires the HMD information of the HMD to be used. The screen information determination unit determines screen information, including the size of the planar screen for displaying the entire sub-stereo image and the distance from the observation viewpoint, based on the sub-stereo image and imaging information. The display image generation unit generates a main stereo image to be displayed on the HMD based on the acquired sub-stereo image, the acquired HMD information, and the acquired screen information. The display control unit performs the necessary conversion processing on the acquired main stereo image for display on the HMD and outputs it to the HMD.

Description

Technical Field

[0001] The present disclosure relates to an image processing technique for generating a stereo image.

Background Art

[0002] In recent years, opportunities to observe video content using a head-mounted display (HMD) have increased. When observing a stereo image with a parallax captured from different left and right positions in video content using an HMD, an observer observes an image in which the stereo image is displayed on a virtual flat screen installed in a virtual space or a virtual reality space. At this time, the observer can observe a three-dimensional image in which an object included in the stereo image pops out from or retracts from the virtual flat screen. The way of feeling the three-dimensional effect when observing a stereo image varies depending on imaging conditions such as the distance between the left and right lenses (baseline length) and the angle of view when capturing the stereo image, and observation conditions such as the distance and size to the virtual flat screen.

[0003] Patent Document 1 discloses a lens system that enables calculation of an appropriate baseline length for feeling a specified degree of three-dimensional effect and capturing a stereo image with the calculated baseline length. The observer can observe the stereo image obtained by capturing in this way with a specified degree of three-dimensional effect by observing it under appropriate observation conditions. When displaying a sub-stereo image within a main stereo image, the observation conditions of the sub-stereo image can be arbitrarily set by setting the position and size of the sub-stereo image arranged in the three-dimensional space reproduced by the main stereo image.

Prior Art Documents

Patent Documents

[0004] International Publication No. 2012 / 128178

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technology described in Patent Document 1 requires the observer to identify and set appropriate observation conditions. If the observer is unable to identify appropriate observation conditions, the three-dimensional effect of the stereo image will differ from what was expected, resulting in a loss of natural three-dimensionality. [Means for solving the problem]

[0006] This disclosure comprises a first acquisition means for acquiring a first stereo image having parallax for realizing stereoscopic vision, and parameters including information corresponding to the field of view of the first stereo image when the first stereo image is generated, and a generation means for generating a second stereo image based on the first stereo image and the parameters, wherein the generation means generates the second stereo image in which the field of view of a display area for displaying the entire first stereo image from an observation viewpoint in a three-dimensional space reproduced by the second stereo image is set based on the field of view of the first stereo image. [Effects of the Invention]

[0007] According to this disclosure, it becomes possible to easily observe stereo images with a natural sense of depth. [Brief explanation of the drawing]

[0008] [Figure 1] A diagram illustrating the principle of perceiving depth from stereo images. [Figure 2] (a) is a diagram showing an example configuration of an image display system using an HMD, and (b) is a diagram showing an example of the hardware configuration of an image processing device. [Figure 3] A block diagram showing an example of the software configuration of the image processing device of Embodiment 1. [Figure 4] A flowchart illustrating the processing of the image processing apparatus of Embodiment 1. [Figure 5] A diagram illustrating the relationship between the field of view of a stereo image and the size and position of the screen on which that stereo image is displayed. [Figure 6]A diagram illustrating the relationship between a virtual planar screen in a three-dimensional space reproduced by stereo images and the observer. [Figure 7] A block diagram showing an example of the software configuration of the image processing device of Embodiment 2. [Figure 8] A flowchart illustrating the processing of the image processing apparatus of Embodiment 2. [Figure 9] A block diagram showing the configuration of the image processing apparatus of Embodiment 3. [Figure 10] A flowchart showing the processing flow of the image processing apparatus of Embodiment 3. [Modes for carrying out the invention]

[0009] The embodiments of this disclosure will be described below with reference to the drawings. The following embodiments are not limiting to this disclosure, and not all combinations of features described in these embodiments are essential to the solutions of this disclosure. The same components will be denoted by the same reference numerals.

[0010] [Embodiment 1] In Embodiment 1, when a sub-stereo image is displayed on the main stereo image shown on the HMD, the display area of ​​the entire sub-stereo image in the main stereo image is set based on a parameter that includes information corresponding to the field of view of the imaging device that captured the sub-stereo image. More specifically, a virtual planar screen is set up in the three-dimensional space reproduced by the main stereo image to display the entire sub-stereo image. The field of view of the virtual planar screen from the observation viewpoint in the reproduced three-dimensional space is then set to be equal to the field of view of the sub-stereo image at the time of acquisition. Note that even if the field of view of the virtual planar screen from the observation viewpoint and the field of view of the sub-stereo image are not exactly equal, it will have the effect of making the sense of depth more natural. Furthermore, although an HMD is used as the display device for displaying the main stereo image in this embodiment, the display device is not limited to an HMD, and any display device capable of individually displaying each image of the stereo image to the left and right eyes is acceptable.

[0011] First, using Figure 1, we will explain the principle by which an observer perceives depth when viewing a stereo image with parallax, which enables stereoscopic vision using an HMD.

[0012] As shown in Figure 1(a), the HMD has lenses 101L / 101R and panels 102L / 102R positioned in front of each of the left and right eyes. When an observer wearing the HMD looks at panels 102L / 102R through lenses 101L / 101R with each eye, they can perceive an image as a virtual image. At this time, by displaying different images with parallax on each of the panels 102L / 102R, the observer obtains a sense of depth from the virtual image due to binocular parallax. The position of this virtual image changes depending on the amount of parallax between the images displayed on panels 102L / 102R. For example, as shown in Figure 1(a), if the position of the same object 104 is significantly different between the left-eye image 103L and the right-eye image 103R, and the parallax 105 is large, the observer will perceive the object 104 as being in a relatively close position. In contrast, as shown in Figure 1(b), if the position of the same object 107 in the left-eye image 106L and the right-eye image 106R is not significantly different and the parallax 108 is small, the observer perceives the object 107 as being located at a relatively distant position.

[0013] When the wearer of the HMD observes the stereo images displayed on the HMD panel, they perceive a three-dimensional virtual space or virtual reality space reproduced by the stereo images. When a sub-stereo image is displayed within the main stereo image displayed on the HMD, the stereo image is displayed on a flat screen 110 placed in the three-dimensional space reproduced by the main stereo image, as shown in Figure 1(c). The flat screen 110 is placed at a finite distance from the observation viewpoint in the reproduced three-dimensional space. Therefore, in the display area of ​​the flat screen 110 for images 109L / 109R displayed on panels 102L / 102R, there is a parallax 111 corresponding to the distance of the flat screen 110 from the observation viewpoint. At this time, the image displayed in the display area of ​​the flat screen 110 shown on panel 102L for the left eye is the left-eye image of the sub-stereo image. Conversely, the image displayed in the display area of ​​the flat screen 110 shown on panel 102R for the right eye is the right-eye image of the sub-stereo image. Therefore, the observer can perceive the image three-dimensionally, with objects appearing to pop out at a distance corresponding to the parallax in the sub-stereo image, using the flat screen 110 as a reference. For example, consider the case where the main stereo image, obtained by imaging a space as shown in Figure 1(d) and displaying the stereo images 112L / 112R as sub-stereo images on the flat screen 110 as shown in Figure 1(c), is observed with an HMD. In this case, the observer perceives object 113, which is relatively close to the imaging device and has parallax, as being closer than the flat screen 110. On the other hand, for object 114, which is very far from the imaging device and has almost no parallax, the observer perceives it as being at approximately the same distance as the flat screen 110.

[0014] The above explains the principle by which, when the main stereo image 109L / 109R is observed while the sub-stereo images 112L / 112R are displayed within the main stereo image 109L / 109R, a sense of depth is perceived in the sub-stereo images 112L / 112R. In this embodiment, based on this principle of perceiving the sense of depth of the sub-stereo images, the conditions for the display area for displaying the entire sub-stereo image within the main stereo image are determined.

[0015] The specific configuration of this embodiment will be described below. Figure 2(a) is a diagram showing an example configuration of an image display system using an HMD20. The image display system shown in Figure 2(a) consists of an image processing device 10 that controls the HMD20 and an HMD20, which is a head-mounted display device. In this embodiment, the image processing device 10 is described as a system configuration independent of the HMD20, but an integrated HMD system configuration in which the image processing device 10 is included inside the HMD20 is also possible.

[0016] Fig. 2(b) shows an example of the hardware configuration of the image processing apparatus 10. In Fig. 2(b), the CPU 201 uses the RAM 202 as a work memory, executes programs stored in the ROM 203 and the hard disk drive (HDD) 205 which is a secondary storage device, and controls the operations of each block described later via the system bus 210. The HDD interface (hereinafter, the interface is denoted as "I / F") 204 connects secondary storage devices such as the HDD 205 and an optical disk drive. The HDD I / F 204 is an I / F such as Serial ATA (SATA) for example. The CPU 201 can read data from the HDD 205 and write data to the HDD 205 via the HDD I / F 204. Further, the CPU 201 can expand the data stored in the HDD 205 to the RAM 202, and conversely, can also save the data expanded in the RAM 202 to the HDD 205. And the CPU 201 can execute the data expanded in the RAM 202 as a program. The input I / F 206 can connect input devices such as a keyboard, a mouse, and an HMD controller. The input I / F 206 is a serial bus I / F such as USB or IEEE1394 for example. The CPU 201 reads data from the input device 207 via the input I / F 206. The output I / F 208 connects the image processing apparatus 10 and the HMD 20 which is an output device 209. The output I / F 208 is an image output I / F such as DVI or HDMI (registered trademark), and / or a serial bus I / F such as USB or IEEE1394. The CPU 201 can send data to the output device 209 such as the HMD 20 via the output I / F 208 to display a predetermined image. Also, information such as the position and orientation of the HMD 20 when the user is experiencing an image (hereinafter referred to as "HMD information") is received from the HMD 20. The HMD information may be input via a mouse, a keyboard, a camera, etc. Although there are also components of the image processing apparatus 10 other than those described above, since they are not the main focus of the present disclosure, the description thereof is omitted.

[0017] Fig. 3 shows an example of the software configuration of the image processing apparatus 10 in the present embodiment. The image processing apparatus 10 according to the present embodiment includes a stereo image data acquisition unit 301, an HMD information acquisition unit 302, a screen information determination unit 303, a display image generation unit 304, and a display control unit 305. Hereinafter, each component will be described.

[0018] The stereo image data acquisition unit 301 acquires a sub-stereo image to be displayed in the main stereo image and imaging information, which is a parameter including information corresponding to the viewing angle of the imaging device that captured the sub-stereo image. The stereo image acquired by the stereo image data acquisition unit 301 may be stored in the HDD 205 or may be output directly from the imaging device immediately after imaging. The stereo image in the present embodiment is, for example, a stereo image with parallax captured by a standard lens with a viewing angle of 46 degrees. The data format of the stereo image may be any format that can represent two images with parallax. For example, the data format of a moving image may be MV-HEVC or the like. The viewing angle as the imaging information may be information indicating the viewing angle itself or information that can be calculated from other imaging information. In this example, these are regarded as information corresponding to the viewing angle. Also, the viewing angle is not limited to the above viewing angle, and may be a narrower viewing angle such as a telephoto lens or a wider viewing angle such as a wide-angle lens. Note that in the present embodiment, the stereo image is an imaging image obtained by imaging with an imaging device, but it may also be a CG image. When the sub-stereo image is a CG image, the imaging information is a parameter regarding the viewpoint using the CG image for rendering.

[0019] The stereo image data acquisition unit 301 also acquires imaging information, which is information regarding the imaging conditions at the time of imaging corresponding to the stereo image to be acquired. This imaging information includes at least the baseline length, which represents the distance between the lenses of the left and right imaging devices, and the horizontal field of view (horizontal field of view) in the horizontal direction (baseline length direction). The horizontal field of view may also be the diagonal field of view. In this embodiment, the imaging information is stored as metadata of the stereo image, and the stereo image data acquisition unit 301 acquires the imaging information by reading the metadata of the stereo image. If the horizontal field of view is stored in the metadata, it is used as is as imaging information, but if the sensor size and focal length are stored in the metadata, the horizontal field of view is calculated from that information. The horizontal field of view based on the sensor size and focal length can be calculated using 2 × atan(cs / 2 / f), where the horizontal sensor size is cs and the focal length is f. The stereo image and imaging information acquired by the stereo image data acquisition unit 301 are output to the screen information determination unit 303 and the display image generation unit 304.

[0020] The HMD information acquisition unit 302 acquires information as HMD information indicating the position and orientation of the HMD20 when the wearer of the HMD20 is experiencing a virtual space or virtual reality space. The position of the HMD20 in the HMD information is represented in a three-dimensional coordinate system that represents the three-dimensional space reproduced by the stereo image displayed on the HMD20. In this embodiment, the three-dimensional coordinate system uses the x-axis to represent the horizontal (baseline direction) spread, the y-axis to represent the height spread, and the z-axis to represent the depth spread, based on the position and orientation of the HMD20 at the start of operation. The reference position and orientation may be the position and orientation of the HMD20 at the start of operation, or the position and orientation of the HMD20 when the position / display reset function provided in the HMD20 is performed. The HMD20 also has multiple RGB cameras and an inertial measurement unit (IMU) to realize position tracking using an inside-out method. The IMU is a device that detects three-dimensional inertial motion (translational and rotational motion in three orthogonal axes), and consists of a gyro sensor that captures rotational motion and an accelerometer that captures translational motion. The IMU represents the attitude of the HMD20 using a 3x3 rotation matrix in three-dimensional space, along with roll, pitch, and yaw. However, the method of representing attitude is not limited to this, and other methods such as quaternions may be used. The HMD information acquired by the HMD information acquisition unit 302 is output to the display image generation unit 304.

[0021] The screen information determination unit 303 determines screen information for a virtual planar screen that displays the entire sub-stereo image, based on the stereo image and imaging information input from the stereo image data acquisition unit 301. The screen information includes the size of the planar screen and the distance from the observation viewpoint. The size of the planar screen is a scalar value representing the horizontal and vertical lengths, respectively. The distance of the planar screen from the observation viewpoint is the z-coordinate value in the three-dimensional coordinate system representing the three-dimensional space reproduced by the stereo image described above, and is expressed as the distance in the z-axis direction from the origin of the three-dimensional coordinate system. The screen information determined by the screen information determination unit 303 is output to the display image generation unit 304.

[0022] The display image generation unit 304 generates the main stereo image to be displayed on the HMD 20 based on the sub-stereo image, HMD information, and screen information. As described above, this main stereo image is a rendering image of a three-dimensional space in which a display area for displaying the entire sub-stereo image is arranged, and consists of two images to be displayed on the left and right panels of the HMD 20, respectively. The main stereo image generated by the display image generation unit 304 is output to the display control unit 305.

[0023] The display control unit 305 converts the main stereo image input from the display image generation unit 304 into an image suitable for observation with the HMD20 and outputs it to the HMD20. This conversion includes color conversion processing suitable for the HMD20's built-in panel and distortion correction processing to correct the distortion of the HMD20's eyepiece lens.

[0024] Figure 4 shows a flowchart illustrating the main stereo image data generation process in this embodiment. The image processing device 10 executes a series of processes shown in the flowchart of Figure 3 by having the CPU 201 execute a program stored in the ROM 203 using the RAM 202 as work memory. Note that not all of the processes shown below need to be executed by the CPU 201; the image processing device 10 may be configured so that some or all of the processes are performed by one or more processing circuits other than the CPU 201. In this embodiment, a video image that has been captured and stored in advance is read from the HDD 205, and processing is started in response to a display start instruction in the HMD 20, and is executed frame by frame. In the following flowchart, each process (step) will be denoted as "S".

[0025] In S401, the stereo image data acquisition unit 301 acquires a sub-stereo image displayed within the main stereo image, along with the imaging information for that sub-stereo image. The acquired stereo image and imaging information are output to the screen information determination unit 303 and the display image generation unit 304.

[0026] In S402, the HMD information acquisition unit 302 acquires the HMD information of the HMD 20 to be used. The acquired HMD information is output to the display image generation unit 304.

[0027] In S403, the screen information determination unit 303 determines screen information, including the size of the planar screen for displaying the entire sub-stereo image and the distance from the observation viewpoint, based on the sub-stereo image and the imaging information. Specifically, it determines the size of the planar screen and the distance from the observation viewpoint in the screen information so that the horizontal field of view of the sub-stereo image in the imaging information is equal to the horizontal field of view of the planar screen observed by the observer in the main stereo image.

[0028] Figure 5 shows a diagram illustrating the relationship between imaging conditions and observation conditions. Figure 5 is an overhead view of the imaging device 501 and the observer 503 from the vertical direction (y-axis direction). Figure 5(a) shows the horizontal field of view 502 when the imaging device 501 captures a stereo image. Figures 5(b) and (c) show the horizontal field of view 502 of a virtual planar screen at the observation viewpoint observed by the observer 503 in this embodiment. Thus, in this embodiment, the horizontal field of view 502 of the planar screen observed by the observer 503 is equal to the horizontal field of view 502 of the imaging device 501 when imaging. Here, the horizontal field of view of the planar screen observed by the observer changes depending on the relationship between the size of the planar screen and the distance from the observation viewpoint. For example, in Figure 5(c), a planar screen 507 larger in size than the planar screen 505 shown in Figure 5(b) is placed at a distance 506, which is farther than the distance 504 from the observation viewpoint of the planar screen in Figure 5(b). In both Figures 5(b) and (c), the horizontal field of view 502 during imaging and the horizontal field of view 502 of the planar screens 505 and 507 observed from the observation viewpoint are the same. In this embodiment, first, one of the values ​​of the horizontal length of the observed virtual planar screen and the distance from the observation viewpoint is determined. Then, the other value is determined so that the horizontal field of view during imaging and the horizontal field of view of the observed virtual planar screen are equal. For example, if the horizontal length of the planar screen is scw, then the distance of the planar screen from the observation viewpoint is scw / 2 / tan(θ / 2). Here, θ is the horizontal field of view in the imaging information. Conversely, if the distance of the planar screen from the observation viewpoint is d, then the horizontal length of the planar screen is 2×d×tan(θ / 2). The horizontal length of the fixed planar screen and the distance from the observation viewpoint can be arbitrary values ​​such as predetermined values ​​or values ​​entered by the observer. The vertical length of the flat screen is determined so that the horizontal length of the flat screen and the aspect ratio of the sub-stereo image displayed on it match the aspect ratio of the sub-stereo image at the time of capture. Specifically, it is determined by scw / imw × imh, using the horizontal pixel count imw and vertical pixel count imh of the sub-stereo image displayed on the flat screen.The screen information, consisting of the size of the planar screen determined as described above and the distance from the observation viewpoint, is output to the display image generation unit 304.

[0029] In the above description, the horizontal field of view of the observation screen was determined to perfectly match the horizontal field of view θ at the time of imaging. However, this is not limited to this method; both fields of view may be determined to be approximately equivalent. In that case, the interval between possible values ​​for the horizontal field of view of the observation screen is defined, the acquired horizontal field of view θ at the time of imaging is converted to the closest possible field of view θ' within the defined interval, and the screen information is determined based on the field of view θ' using the formula described above. For example, if the interval between possible values ​​for the horizontal field of view θ' of the observation screen is defined as 5 degrees, and the horizontal field of view θ at the time of imaging is 46 degrees, the converted field of view θ' will be 45 degrees. Note that this is not the only method for determining the field of view at observation to be approximately equivalent to the field of view at imaging. For example, the interval between possible values ​​for the size of the observation screen and the distance from the observation viewpoint may be defined, and the screen information value determined based on the stereo image and imaging information may be converted to the closest possible value.

[0030] In S404, the display image generation unit 304 generates a main stereo image to be displayed on the HMD20 based on the sub-stereo image acquired in S401, the HMD information acquired in S402, and the screen information acquired in S403. Specifically, first, from the posture information of the HMD20 included in the HMD information, the line of sight direction of the observer's viewpoint (observation viewpoint) wearing the HMD20 in the three-dimensional space reproduced by the main stereo image is determined as a three-dimensional unit vector. Then, based on the position information of the HMD20 included in the HMD information and the direction of the determined three-dimensional unit vector (line of sight direction of the observation viewpoint), the main stereo image representing the view from the observation viewpoint is rendered in accordance with the display field of view of the HMD20. The display field of view at this time is a fixed value that depends on the HMD20 and is defined by the display's viewing angle and panel resolution. Furthermore, rendering is a process that generates a perspective projection image from three-dimensional space, and a general three-dimensional rendering method can be used.

[0031] Figure 6 shows a virtual planar screen positioned in three-dimensional space, reproduced by the main stereo image observed by the observer through the HMD 20. This virtual planar screen 601 has a size and distance 602 from the observation viewpoint set based on the screen information, and displays the sub-stereo image acquired in S401. In this embodiment, the direction in which the virtual planar screen 601 is positioned is the z-axis direction in the three-dimensional coordinate system described above, and when expressed as a unit direction vector of the three-dimensional coordinate system, v = (0, 0, 1). That is, the virtual planar screen 601 is positioned in the direction in front of the observation viewpoint in the reference state when the three-dimensional coordinate system was defined. In addition, the height of the virtual planar screen 601 in this embodiment is a predetermined height h. When the position of the center 603 of the planar screen is expressed as a three-dimensional coordinate value in the three-dimensional coordinate system, the coordinate value is (0, h, d + pz). Here, d is the distance 602 from the observation viewpoint of the planar screen in the screen information, and pz is the position of the observation viewpoint in the z-axis direction in the HMD information. Furthermore, the orientation and position of the virtual planar screen 601 are not limited to the above example, as long as the size of the virtual planar screen 601 and its distance from the observation viewpoint match the screen information. For example, even if the line of sight direction of the observation viewpoint changes, the virtual planar screen 601 may always be positioned directly in front of the line of sight direction of the observation viewpoint. In that case, the line of sight direction of the observation viewpoint can be calculated as a three-dimensional unit vector from the posture information included in the HMD information, and the virtual planar screen 601 can be positioned at a distance from the observation viewpoint specified in the screen information in the direction of the three-dimensional unit vector. In addition, the height of the center of the virtual planar screen 601 may also be the same as the height of the observation viewpoint, in which case the position coordinates of the center of the virtual planar screen 601 will be (0, py, d+pz). Here, py is the position of the observation viewpoint in the y-axis direction in the HMD information. As mentioned above, the images displayed on the virtual planar screen 601 are different for the left eye image and the right eye image of the main stereo image. When generating the left-eye image of the main stereo image to be displayed on the left-eye panel, the left-eye image of the sub-stereo image is displayed on a virtual flat screen.When generating the right-eye image of the main stereo image to be displayed on the right-eye panel, the right-eye image of the sub-stereo image is displayed on a virtual planar screen 601.

[0032] In S405, the display control unit 305 performs the necessary conversion processing on the main stereo image acquired in S404 for display on the HMD20 and outputs it to the HMD20. The HMD20 displays the converted main stereo image received from the image processing device 10 on the panels 102L / 103R. Once the processing in S405 is complete, the series of processes ends.

[0033] The above describes the processing performed by the image processing device 10 in this embodiment. The above description assumes that the processing is started in response to the operation start instruction of the HMD 20 and is executed on a frame-by-frame basis, but is not limited to this. For example, this processing may be performed only when the observer gives an instruction to start playback of the stereo image. In that case, the size of the flat screen 601 and the distance from the observation viewpoint will remain fixed and will not change until the playback of the acquired sub-stereo image is finished, or until another sub-stereo image is acquired and an instruction to start playback of the acquired other sub-stereo image is given.

[0034] As described above, in this embodiment, when displaying a sub-stereo image within the main stereo image observed using the HMD20, the horizontal field of view of the planar screen displaying the entire sub-stereo image is made equal to the horizontal field of view of the sub-stereo image at the time of acquisition. This allows the observer to observe objects contained within the sub-stereo image with a natural sense of depth.

[0035] [Embodiment 2] In Embodiment 1, based on the imaging information of the sub-stereo image displayed within the main stereo image, information for a planar screen that displays the entire sub-stereo image, which is placed in the three-dimensional space reproduced by the main stereo image, was determined. Specifically, the size of the virtual planar screen and its distance from the observation viewpoint were determined so that the horizontal field of view of the sub-stereo image at the time of imaging and the horizontal field of view of the virtual planar screen were approximately equal.

[0036] In contrast, Embodiment 2 defines the maximum and minimum horizontal field of view of the virtual planar screen used for observation. Furthermore, it adds a process to determine screen information so that the horizontal field of view of this virtual planar screen can be observed with the most natural sense of depth between the maximum and minimum fields of view.

[0037] In Embodiment 1, if the horizontal field of view of the sub-stereo image is a narrow 20 degrees, the horizontal field of view of the flat screen displaying the entire sub-stereo image will also be a narrow 20 degrees. In this case, depending on the content of the sub-stereo image displayed on the flat screen, the image may become too small and difficult to see. Conversely, if the horizontal field of view of the sub-stereo image is a wide 60 degrees, the horizontal field of view of the flat screen will also be a wide 60 degrees. In this case as well, depending on the HMD used for observation, it may exceed the field of view of the HMD, forcing observation of a portion of the sub-stereo image being cut off. Therefore, in Embodiment 2, minimum and maximum fields of view are defined for the horizontal field of view of the flat screen displaying the entire sub-stereo image.

[0038] In Embodiment 2, if the horizontal field of view of the sub-stereo image displayed within the main stereo image is within a predetermined range of minimum and maximum field of view, the screen information is determined in the same manner as in Embodiment 1.

[0039] On the other hand, if the horizontal field of view of the sub-stereo image is smaller than the minimum field of view, the screen information is determined so that the horizontal field of view of the flat screen displaying the entire sub-stereo image becomes the minimum field of view. At this time, the wider the difference between the horizontal field of view of the sub-stereo image and the minimum field of view of the flat screen, the more unnatural the sense of depth obtained during observation may become, such as objects appearing thinner than they actually are. Therefore, in Embodiment 2, the distance of the flat screen from the observation viewpoint is fixed to a predetermined minimum field of view distance, and the size of the flat screen is determined so that the horizontal field of view of the flat screen becomes the minimum field of view at that position. By fixing the distance of the flat screen from the observation viewpoint to the minimum field of view distance, the amount of objects included in the sub-stereo image protruding from the flat screen is suppressed, making it less likely for objects to appear unnaturally thin during observation.

[0040] Furthermore, if the horizontal field of view of the captured sub-stereo image is greater than the maximum field of view of the displaying flat screen, the screen information is determined so that the horizontal field of view of the flat screen becomes the maximum field of view. At this time, the greater the difference between the horizontal field of view of the sub-stereo image and the maximum field of view, the more unnatural the sense of depth obtained during observation may become, such as making objects appear thicker than they actually are. Therefore, in Embodiment 2, the distance of the flat screen from the observation viewpoint is fixed to a predetermined distance for the maximum field of view, and the size of the virtual flat screen is determined so that the horizontal field of view of the virtual flat screen during observation becomes the maximum field of view. By fixing the distance of the flat screen from the observation viewpoint to the distance for the maximum field of view, the amount of protrusion of objects included in the sub-stereo image from the flat screen is increased, making it less likely for objects to appear unnaturally thick during observation.

[0041] Figure 7 is a block diagram showing the configuration of the image processing apparatus 10 in this embodiment. The image processing apparatus 10 according to this embodiment includes a stereo image data acquisition unit 301, a screen condition acquisition unit 701, a screen information determination unit 702, an HMD information acquisition unit 302, a display image generation unit 304, and a display control unit 305. Each component will be described below. Components similar to those in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1 and their descriptions are omitted.

[0042] The screen condition acquisition unit 701 acquires predetermined screen condition information from the HDD 205 according to the HMD 20. In this embodiment, the screen condition information includes the minimum and maximum horizontal field of view of the planar screen that displays the entire sub-stereo image, and the distance from the observation viewpoint of the planar screen corresponding to those field of view. The distances from the two observation viewpoints corresponding to the minimum and maximum field of view are used when the horizontal field of view during stereo image acquisition is smaller than the minimum field of view or larger than the maximum field of view.

[0043] The minimum and maximum fields of view can be set arbitrarily. For example, the minimum field of view could be set to 30 degrees, which is said to be the central field of view that humans are most sensitive to, and the maximum field of view could be set to 50 degrees, which is the average maximum field of view for glasses-type HMDs. Alternatively, the field of view could be set to the field of view of the HMD20 that utilizes the maximum field of view, or the observer could be allowed to arbitrarily select the minimum and maximum fields of view. The shorter the distance from the observation viewpoint corresponding to the minimum field of view of the flat screen, the better. However, if it is too short, objects that appear to pop out from the flat screen may appear too close to the observer. If objects that appear to pop out are too close to the observer, it may be difficult for the observer to fuse the objects. Therefore, a suitable distance for the observation viewpoint corresponding to the minimum field of view of the flat screen is, for example, 1m, which is the distance at which the observer can fuse the objects without difficulty. The longer the distance from the observation viewpoint corresponding to the maximum field of view of the flat screen, the better. For example, a suitable distance for the observation viewpoint corresponding to the maximum field of view of the virtual flat screen is, for example, 7m, which is considered to be the distance at which depth perception is easiest to achieve. The screen condition information determined in this way is stored in the HDD 205 beforehand, and the screen condition acquisition unit 701 outputs the screen condition information acquired from the HDD 205 to the screen information determination unit 702.

[0044] The screen information determination unit 702 determines screen information for a virtual planar screen based on the stereo image and imaging information input from the stereo image data acquisition unit 301 and the screen condition information input from the screen condition acquisition unit 701. The screen information determined by the screen information determination unit 702 is the size of the virtual planar screen and the distance from the observation viewpoint, similar to Embodiment 1. The determined screen information is output to the display image generation unit 304.

[0045] Figure 8 shows a flowchart illustrating the main stereo image data generation process in Embodiment 2. Processes S401 to S405 are the same as in Embodiment 1, so their explanation is omitted here. Only processes S801 to S803, which were added in Embodiment 2, will be explained.

[0046] In S801, the screen condition acquisition unit 701 acquires screen condition information. The acquired screen condition information is output to the screen information determination unit 702.

[0047] In S802, the screen information determination unit 702 compares the horizontal field of view of the imaging information acquired from the stereo image data acquisition unit 301 with the minimum and maximum horizontal field of view of the screen condition information acquired from the screen condition acquisition unit 701. If the horizontal field of view of the imaging information is within the range of the minimum and maximum field of view, the process proceeds to S403. If the horizontal field of view of the imaging information is smaller than the minimum field of view or larger than the maximum field of view, the process proceeds to S803.

[0048] In S803, the screen information determination unit 803 determines screen information from screen condition information. Specifically, it determines the size of the planar screen in the screen information so that the minimum or maximum horizontal field of view in the screen condition information is approximately equal to the horizontal field of view of the planar screen that displays the entire sub-stereo image. In addition, the distance of the planar screen from the observation viewpoint in the screen information is determined to be the distance corresponding to the minimum or maximum field of view included in the screen condition information. If the horizontal field of view in the imaging information is smaller than the minimum field of view, the minimum field of view is used as the horizontal field of view of the planar screen, and the distance for the minimum field of view is used as the distance of the planar screen from the observation viewpoint. Conversely, if the horizontal field of view in the imaging information is larger than the maximum field of view, the maximum field of view is used as the horizontal field of view of the planar screen, and the distance for the maximum field of view is used as the distance of the planar screen from the observation viewpoint. The method for calculating the size of the planar screen is the same as the method described in Embodiment 1, so the explanation is omitted. The screen information determination unit 803 outputs screen information consisting of the determined size of the planar screen and the distance from the observation viewpoint to the display image generation unit 304.

[0049] The above describes the processing performed by the image processing device 10 in Embodiment 2. In Embodiment 2, the horizontal field of view of the planar screen for displaying the sub-stereo image to be displayed within the main stereo image is determined based on the horizontal field of view of the sub-stereo image during imaging, as well as predetermined maximum and minimum field of view. This allows the observer to fuse the stereo images seamlessly and obtain a natural sense of depth with high sensitivity to the perception of three-dimensionality.

[0050] In this embodiment, both the minimum and maximum field of view are set, but it is also possible to set only one of them.

[0051] [Embodiment 3] In Embodiment 3, the imaging information for the sub-stereo image includes the baseline length, which is the distance between the left and right lenses of the imaging device. Furthermore, information on the interpupillary distance, which is the distance between the observer's left and right eyes, is newly acquired, and the screen information for the planar screen that displays the entire sub-stereo image is determined based on the imaging information and the interpupillary distance information. Specifically, the screen information is determined so that the corrected field of view, obtained by scaling the horizontal field of view of the sub-stereo image at the time of imaging using the ratio of baseline length to interpupillary distance, is approximately equal to the horizontal field of view of the planar screen.

[0052] Even if the horizontal field of view of the sub-stereo image during imaging is the same as that of the virtual planar screen, as in Embodiment 1, if the baseline length of the imaging device differs significantly from the observer's interpupillary distance, the three-dimensional effect of objects included in the sub-stereo image may appear unnatural. For example, if the baseline length is shorter than the interpupillary distance, objects may appear excessively thin, impairing a natural sense of depth. Conversely, if the baseline length is longer than the interpupillary distance, objects may appear excessively thick, impairing a natural sense of depth. Therefore, in Embodiment 3, screen information is determined such that the corrected field of view, obtained by scaling the horizontal field of view of the sub-stereo image during imaging by the ratio of baseline length to interpupillary distance, is approximately equivalent to the horizontal field of view of the planar screen. This allows for observation of the image with a more natural sense of depth, even if the baseline length of the sub-stereo image differs from the observer's interpupillary distance.

[0053] Figure 9 shows an example of the software configuration of the image processing device 10 in this embodiment. The image processing device 10 in this embodiment includes a stereo image data acquisition unit 301, an observer information acquisition unit 901, a screen information determination unit 902, an HMD information acquisition unit 302, a display image generation unit 304, and a display control unit 305. Each component will be described below. Components similar to those in Embodiment 1 are denoted by the same reference numerals as in Embodiment 1 and their descriptions are omitted.

[0054] The observer information acquisition unit 901 acquires information on the observer's interpupillary distance from the HDD 205. For example, the interpupillary distance information is a scalar value representing the observer's interpupillary distance, such as 65 mm. Generally, the HMD 20 is equipped with a mechanism that allows the distance between the eyepieces to be adjusted according to the observer's interpupillary distance, so in this embodiment, the distance between the eyepieces of the HMD 20 is acquired as the observer's interpupillary distance information. However, this is not the only way to acquire the observer's interpupillary distance; various methods are available, such as using a value input from the observer or estimating it from information from the eye-tracking camera installed in the HMD 20. The interpupillary distance information acquired by the observer information acquisition unit 901 is output to the screen information determination unit 902.

[0055] The screen information determination unit 902 determines screen information for a virtual planar screen based on the sub-stereo image and imaging information input from the stereo image data acquisition unit 301, and the interpupillary distance information input from the observer information acquisition unit 901. The screen information is the size of the planar screen that displays the entire sub-stereo image and the distance from the observation viewpoint, similar to Embodiment 1. The screen information determination unit 902 outputs the determined screen information to the display image generation unit 3.

[0056] Figure 10 shows a flowchart illustrating the main stereo image data generation process in the embodiment. Processes S401, S402, S404, and S405 are the same as in Embodiment 1, so their explanation is omitted here. Processes S1001 and S1002, which were added in Embodiment 3, will be explained. Note that the imaging information acquired in S401 includes the baseline length of the imaging device used for imaging.

[0057] In S1001, the observer information acquisition unit 901 acquires the observer's interpupillary distance information. The acquired interpupillary distance information is output to the screen information determination unit 902.

[0058] In S1002, the screen information determination unit 902 determines screen information from imaging information acquired from the stereo image data acquisition unit 301 and interpupillary distance information acquired from the observer information acquisition unit 901. Specifically, the screen information is determined so that the horizontal field of view of the sub-stereo image included in the imaging information, scaled by the ratio of baseline length to interpupillary distance, is approximately equal to the horizontal field of view of the planar screen. If the horizontal field of view of the imaging information is θ, the baseline length is T, and the interpupillary distance is e, then the horizontal field of view φ of the virtual planar screen, scaled by the ratio of baseline length to interpupillary distance, is φ = θ × T ÷ e. Furthermore, the size of the planar screen and the distance from the observation viewpoint are determined using the method described in Embodiment 1 so that the horizontal field of view of the planar screen is φ. The screen information determination unit 902 outputs the screen information consisting of the determined size of the planar screen and the distance from the observation viewpoint to the display image generation unit 304.

[0059] The above describes the additional processing performed by the image processing device 10 of Embodiment 3 compared to the processing of Embodiment 1. In Embodiment 3, information on the baseline length, which is the distance between the left and right lenses when capturing a sub-stereo image, is acquired as imaging information, and information on the interpupillary distance, which is the distance between the left and right eyes of the observer, is acquired. Then, by determining a planar screen on which to display the entire sub-stereo image based on the baseline length and interpupillary distance information, the sub-stereo image can be observed with a more natural sense of depth even if the observer's interpupillary distance differs from the baseline length of the imaging device.

[0060] The embodiments of this disclosure are not limited to the examples described above, and various embodiments are possible. For example, Embodiment 2 and Embodiment 3 may be used in combination.

[0061] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0062] This disclosure includes the following configurations and methods: [Configuration 1] A first acquisition means for acquiring a first stereo image having parallax to realize stereoscopic vision, and parameters including information corresponding to the field of view of the first stereo image when the first stereo image was generated, A generation means for generating a second stereo image based on the first stereo image and the parameters, Equipped with, The generation means generates the second stereo image in which the field of view of the display area that displays the entire first stereo image from an observation viewpoint in a three-dimensional space reproduced by the second stereo image is set based on the field of view of the first stereo image. An image processing apparatus characterized by the following: [Configuration 2] The field of view of the first stereo image and the field of view of the display area are horizontal fields of view. The image processing apparatus according to configuration 1, characterized in that... [Configuration 3] The generation means generates a second stereo image in which the difference between the field of view of the display area and the field of view of the first stereo image is within a predetermined range. An image processing apparatus according to configuration 1 or 2, characterized by the above. [Structure 4] The generation means generates the second stereo image such that the field of view of the display area is substantially the same as the field of view of the first stereo image. An image processing apparatus according to any one of configurations 1 to 3. [Composition 5] The generation means generates a second stereo image in which the size of the display area and the distance of the display area from the observation viewpoint are set based on the field of view of the display area. An image processing apparatus according to any one of configurations 1 to 4, characterized in that [Composition 6] The system further comprises a second acquisition means for acquiring conditional information including at least one of the minimum and maximum fields of view of the display area, The generation means sets the field of view of the display area to the minimum field of view if the condition information includes the minimum field of view and the field of view of the first stereo image is smaller than the minimum field of view, and sets the field of view of the display area to the maximum field of view if the condition information includes the maximum field of view and the field of view of the first stereo image is larger than the maximum field of view. The image processing apparatus according to configuration 5, characterized by the features described herein. [Composition 7] The condition information includes at least one of a first distance corresponding to the minimum field of view and a second distance corresponding to the maximum field of view. The generation means includes the minimum field of view and the first distance in the condition information, and if the field of view of the first stereo image is smaller than the minimum field of view, the distance of the display area to the observation viewpoint is set as the first distance; and if the condition information includes the maximum field of view and the second distance, and the field of view of the first stereo image is larger than the maximum field of view, the distance of the display area to the observation viewpoint is set as the second distance. The image processing apparatus according to configuration 6, characterized by the features described therein. [Structure 8] The image processing apparatus according to configuration 7, characterized in that the first distance is a distance at which an observer observing the second stereo image displayed on the display means can fuse the second stereo image. [Composition 9] The image processing apparatus according to configuration 7 or 8, characterized in that the second distance is the distance at which an observer viewing the second stereo image displayed on the display means can most easily perceive a sense of depth. [Configuration 10] The image processing apparatus according to any one of configurations 6 to 9, characterized in that the minimum field of view is the central field of view of an observer observing the second stereo image displayed on the display means. [Composition 11] The image processing apparatus according to any one of configurations 8 to 10, characterized in that the maximum field of view is the maximum field of view of the display means for displaying the second stereo image. [Composition 12] The display means further comprises a third acquisition means for acquiring observer information, including the interpupillary distance of the observer observing the second stereo image, The parameter includes information corresponding to the baseline length of the first stereo image, The generating means generates a second stereo image in which the field of view of the display area is set based on the interpupillary distance, the baseline length, and the field of view of the first stereo image. An image processing apparatus according to any one of configurations 1 to 11, characterized by the above. [Composition 13] The generation means generates a second stereo image in which the field of view of the display area is set based on a field of view obtained by scaling the field of view of the first stereo image by the ratio of the baseline length to the interpupillary distance. The image processing apparatus according to configuration 12, characterized in that... [Composition 14] Display means and A display control means for displaying the second stereo image on the display means, An image processing apparatus according to any one of configurations 1 to 13, further comprising: [Composition 15] Steps include obtaining a first stereo image having parallax for realizing stereoscopic vision, and parameters including information corresponding to the field of view of the first stereo image when it was generated, A step of generating a second stereo image based on the first stereo image and the parameters, The steps include displaying the second stereo image on the display means, Equipped with, The generation step involves generating a second stereo image in which, in the three-dimensional space reproduced by the second stereo image, the field of view of the display area for displaying the entire first stereo image from the observation viewpoint is set based on the field of view of the first stereo image. An image processing method characterized by the following: [Composition 16] A program for causing a computer to function as an image processing device as described in any one of items 1 to 14.

Claims

1. A first acquisition means for acquiring a first stereo image having parallax to realize stereoscopic vision, and parameters including information corresponding to the field of view of the first stereo image, A generation means for generating a second stereo image based on the first stereo image and the parameters, Equipped with, The generation means generates the second stereo image in which, in the three-dimensional space reproduced by the second stereo image, the field of view of the display area for displaying the entire first stereo image from the observation viewpoint is set based on the field of view of the first stereo image. An image processing apparatus characterized by the following:

2. The field of view of the first stereo image and the field of view of the display area are horizontal fields of view. The image processing apparatus according to feature 1.

3. The generation means generates a second stereo image in which the difference between the field of view of the display area and the field of view of the first stereo image is within a predetermined range. The image processing apparatus according to feature 1.

4. The generation means generates the second stereo image such that the field of view of the display area is substantially the same as the field of view of the first stereo image. The image processing apparatus according to feature 1.

5. The generation means generates a second stereo image in which the size of the display area and the distance of the display area from the observation viewpoint are set based on the field of view of the display area. The image processing apparatus according to feature 1.

6. The system further comprises a second acquisition means for acquiring conditional information including at least one of the minimum and maximum fields of view of the display area, The generation means sets the field of view of the display area to the minimum field of view if the condition information includes the minimum field of view and the field of view of the first stereo image is smaller than the minimum field of view, and sets the field of view of the display area to the maximum field of view if the condition information includes the maximum field of view and the field of view of the first stereo image is larger than the maximum field of view. The image processing apparatus according to feature 5.

7. The condition information includes at least one of a first distance corresponding to the minimum field of view and a second distance corresponding to the maximum field of view. The generation means includes the minimum field of view and the first distance in the condition information, and if the field of view of the first stereo image is smaller than the minimum field of view, the distance of the display area to the observation viewpoint is set as the first distance; and if the condition information includes the maximum field of view and the second distance, and the field of view of the first stereo image is larger than the maximum field of view, the distance of the display area to the observation viewpoint is set as the second distance. The image processing apparatus according to claim 6.

8. The image processing apparatus according to claim 7, characterized in that the first distance is a distance at which an observer observing the second stereo image displayed on the display means can fuse the second stereo image.

9. The image processing apparatus according to claim 7, characterized in that the second distance is the distance at which an observer viewing the second stereo image displayed on the display means can most easily perceive a sense of depth.

10. The image processing apparatus according to claim 6, characterized in that the minimum field of view is the central field of view of an observer observing the second stereo image displayed on the display means.

11. The image processing apparatus according to claim 6, characterized in that the maximum field of view is the maximum field of view of the display means for displaying the second stereo image.

12. The display means further comprises a third acquisition means for acquiring observer information, including the interpupillary distance of the observer observing the second stereo image, The parameter includes information corresponding to the baseline length of the first stereo image, The generation means generates a second stereo image in which the field of view of the display area is set based on the interpupillary distance, the baseline length, and the field of view of the first stereo image. The image processing apparatus according to feature 1.

13. The generation means generates a second stereo image in which the field of view of the display area is set based on a field of view obtained by scaling the field of view of the first stereo image by the ratio of the baseline length to the interpupillary distance. The image processing apparatus according to feature 12.

14. Display means and A display control means for displaying the second stereo image on the display means, The image processing apparatus according to claim 1, further comprising:

15. Steps include obtaining a first stereo image having parallax for realizing stereoscopic vision, and parameters including information corresponding to the field of view of the first stereo image, A step of generating a second stereo image based on the first stereo image and the parameters, Equipped with, The generation step involves generating a second stereo image in which, in the three-dimensional space reproduced by the second stereo image, the field of view of the display area for displaying the entire first stereo image from the observation viewpoint is set based on the field of view of the first stereo image. An image processing method characterized by the following:

16. A program for causing a computer to function as an image processing device according to any one of claims 1 to 14.

Images